New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 693533 link

Starred by 1 user
Status: WontFix
Owner:
sullivan@chromium.org
Closed: Aug 2017
Cc:
simonhatch@chromium.org
█ alexclarke@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

16.9%-37.5% regression in smoothness.key_silk_cases at 450361:450438

Project Member Reported by alexclarke@chromium.org, Feb 17 2017 
See the link to graphs below.
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Feb 17 2017 


=== BISECT JOB RESULTS ===
Bisect failed unexpectedly

Bisect was aborted with the following:
  Step('Compare samples') failed with return_code 1


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : mean_frame_time/mean_frame_time


To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8987407007507915280

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5834481396613120


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 5 by 42576172...@developer.gserviceaccount.com, Feb 23 2017 


=== BISECT JOB RESULTS ===
Bisect failed unexpectedly

Bisect was aborted with the following:
  Step('Re-testing reference range.Compare samples (4)') failed with return_code 1


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : mean_frame_time/mean_frame_time

Revision             Result                   N
chromium@450360      26.5306 +- 0.648744      14      good
chromium@450438      26.8808 +- 2.05926       9       bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8986921512969110592

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5874277246566400


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 9 by 42576172...@developer.gserviceaccount.com, Mar 3 2017 


=== BISECT JOB RESULTS ===
Bisect failed unexpectedly

Bisect was aborted with the following:
  Step('Re-testing reference range.Compare samples (2)') failed with return_code 1


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : mean_frame_time/mean_frame_time

Revision             Result                   N
chromium@450360      26.3589 +- 0.983312      9      good
chromium@450438      26.6295 +- 0.768363      6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8986185644075178400

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5898351276457984


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 10 by 42576172...@developer.gserviceaccount.com, Mar 3 2017 


=== BISECT JOB RESULTS ===
Perf regression found but unable to narrow commit range

Build failures prevented the bisect from narrowing the range further.


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : avg_surface_fps/avg_surface_fps
  Change       : 0.55% | 46.4948979592 -> 46.2380952381

Suspected Commit Range
  5 commits in range
  https://chromium.googlesource.com/chromium/src/+log/90790ebcd0632f1860a89179643ef89fcf34c270..d892f9592860691ae9a782c12260c94ed6bd1a63


Revision             Result                   N
chromium@450360      46.4949 +- 0.512241      14       good
chromium@450365      46.3393 +- 0.256298      6        good
chromium@450366      ---                      ---      build failure
chromium@450367      ---                      ---      build failure
chromium@450368      ---                      ---      build failure
chromium@450369      ---                      ---      build failure
chromium@450370      46.1607 +- 0.140607      6        bad
chromium@450380      46.2802 +- 0.654354      13       bad
chromium@450399      46.2738 +- 0.381324      9        bad
chromium@450438      46.2381 +- 0.595476      9        bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8986185654246286640

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5886882103164928


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Mar 7 2017 


=== BISECT JOB RESULTS ===
Bisect failed unexpectedly

Bisect was aborted with the following:
  Step('Re-testing reference range.Compare samples (4)') failed with return_code 1


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : avg_surface_fps/avg_surface_fps

Revision             Result                  N
chromium@450360      46.477 +- 0.898705      14      good
chromium@450438      46.1992 +- 1.00079      9       bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8985860845264728992

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5905791602655232


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 14 by 42576172...@developer.gserviceaccount.com, Mar 7 2017 


=== BISECT JOB RESULTS ===
Bisect failed unexpectedly

Bisect was aborted with the following:
  Step('Re-testing reference range.Compare samples (4)') failed with return_code 1


Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : mean_frame_time/mean_frame_time

Revision             Result                   N
chromium@450360      24.3415 +- 0.682013      14      good
chromium@450438      24.4876 +- 0.604313      9       bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8985860833649755184

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5790143601967104


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Comment 15 by benhenry@chromium.org, Mar 11 2017

Labels: Performance-Responsiveness

Comment 16 by ccameron@chromium.org, Mar 17 2017

Owner: sullivan@chromium.org
Status: Assigned (was: Untriaged)
Annie, could you see if we can get a better bisect here?

Comment 17 by ccameron@chromium.org, Mar 17 2017 

(perhaps there was a hardware change? ref changes at the same point and seems to become bimodal)

Comment 18 by sullivan@chromium.org, Mar 20 2017

Cc: simonhatch@chromium.org
Looks like the runs are all on device AG860440GI024GC, so no device swap. Looking at the subpages, lots of them jump so it's not a weirdness with adding a new page or something: https://chromeperf.appspot.com/report?sid=4725ef0bbb49af84326d5cbcc4a18020d34fb30c9f7561456bd856882120e2d6&rev=450438

Here is a trace from before and after the regression from http___mobile-news.sandbox.google.com_news_pt0?swipe:

Before: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_23-2017-02-13_21-04-28-51507.html
After: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_23-2017-02-14_12-58-28-27241.html

+simonhatch in case he has ideas what may be going on with the bisect. Will re-kick on that page.
Project Member

Comment 20 by 42576172...@developer.gserviceaccount.com, Mar 20 2017 


=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : mean_frame_time/http___mobile-news.sandbox.google.com_news_pt0?swipe


To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...mobile.news.sandbox.google.com.news.pt0.swipe smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8984648853739819904

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5012065799897088


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Comment 21 by simonhatch@chromium.org, Mar 20 2017 

re: #c18

Looks like compare_samples is failing. Digging a bit deeper shows a test failing and producing a None value in one of the retesting steps, which results in a summary with "Merging values containing a None value results in a None value." and compare_samples doesn't handle that case. Will put up a CL to fix.
Project Member

Comment 22 by bugdroid1@chromium.org, Mar 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c27e679d80dbdcb09c867f32a8a162964460df3a

commit c27e679d80dbdcb09c867f32a8a162964460df3a
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Tue Mar 21 00:39:48 2017

Roll src/third_party/catapult/ 4a1c8851a..d233eb291 (2 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/4a1c8851a20d..d233eb291f89

$ git log 4a1c8851a..d233eb291 --date=short --no-merges --format='%ad %ae %s'
2017-03-20 maxlg [Tracing] Add FMP CPU time and its breakdown
2017-03-20 simonhatch Compare Samples - Skip summaries with None values.

Created with:
  roll-dep src/third_party/catapult
BUG= 693533 

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2763783002
Cr-Commit-Position: refs/heads/master@{#458257}

[modify] https://crrev.com/c27e679d80dbdcb09c867f32a8a162964460df3a/DEPS
Project Member

Comment 24 by 42576172...@developer.gserviceaccount.com, Apr 10 2017 


=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : smoothness.key_silk_cases
  Metric       : mean_frame_time/http___mobile-news.sandbox.google.com_news_pt0?swipe


To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...mobile.news.sandbox.google.com.news.pt0.swipe smoothness.key_silk_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982659761714633120

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5012065799897088


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Comment 27 by sullivan@chromium.org, Aug 16 2017

Status: WontFix (was: Assigned)
Bisect wasn't able to reproduce.
►

Sign in to add a comment