New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 670722 link

Starred by 1 user
Status: Duplicate
Merged: issue 670797
Owner:
charliea@chromium.org
Closed: Dec 2016
Cc:
█ alexandermont@chromium.org
█ rnep...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 2
Type: Bug-Regression



Sign in to add a comment

30.8% regression in battor.trivial_pages/idle:power_avg on chromium-rel-mac-retina at 434308:435383

Project Member Reported by charliea@chromium.org, Dec 2 2016 
Performance dashboard identified a 30.8% regression in battor.trivial_pages/idle:power_avg on chromium-rel-mac-retina at revision range 434308:435383. Graph: https://chromeperf.appspot.com/report?masters=ChromiumPerf&bots=chromium-rel-mac-retina&tests=battor.trivial_pages%2Fidle%3Apower_avg&checked=idle%3Apower_avg%2Cidle%3Apower_avg_ref%2Cref&rev=435383

Looking at the graph, both the ref build and the non-ref build moved, but the non-ref build moved by significantly more. I'm going to kick off a bisect and see if we can find anything.
both_moved.png
81.7 KB View Download
Project Member

Comment 2 by 42576172...@developer.gserviceaccount.com, Dec 2 2016 

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1849
Failure reason: the build has failed due to infrastructure failure.
Project Member

Comment 4 by 42576172...@developer.gserviceaccount.com, Dec 2 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@434307  39.3897  9.94548  27  good
chromium@435383  38.1656  14.9503  27  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 670722

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=TrivialBlinkingCursorPageSharedPageState battor.trivial_pages
Test Metric: idle:power_avg/TrivialBlinkingCursorPageSharedPageState
Relative Change: 3.11%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1850
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994359895413787312


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5006234162298880

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 5 by nedngu...@google.com, Dec 2 2016 

For hardware/lab change issue, it's unlikely that a bisect will be successfully point out any thing.

Comment 6 by charliea@chromium.org, Dec 2 2016

Cc: -charliea@google.com

Comment 7 by charliea@chromium.org, Dec 2 2016

Mergedinto: 670797
Status: Duplicate (was: Untriaged)
Looking more closely at this, it's a duplicate of https://bugs.chromium.org/p/chromium/issues/detail?id=670797

See: ref_before_regression.png and ref_after_regression.png. You can clearly see that the thing that changed is that "before" is divided up into user expectations and "after" is not.
ref_before_regression.png
144 KB View Download
ref_after_regression.png
195 KB View Download
Project Member

Comment 9 by 42576172...@developer.gserviceaccount.com, Mar 23 2017 


=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: mac_retina_perf_bisect
  Benchmark    : battor.trivial_pages
  Metric       : idle:power_avg/TrivialBlinkingCursorPageSharedPageState

Revision             Result                  N
chromium@434307      27.0153 +- 8.68919      21      good
chromium@435383      27.2097 +- 8.55279      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=TrivialBlinkingCursorPageSharedPageState battor.trivial_pages

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8984378551044336464

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4942528187465728


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 11 by 42576172...@developer.gserviceaccount.com, Apr 11 2017 


=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: mac_retina_perf_bisect
  Benchmark    : battor.trivial_pages
  Metric       : idle:power_avg/TrivialBlinkingCursorPageSharedPageState

Revision             Result                  N
chromium@434307      20.6672 +- 6.15468      21      good
chromium@435383      21.0844 +- 6.2745       21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=TrivialBlinkingCursorPageSharedPageState battor.trivial_pages

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982659788405316416

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5006234162298880


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
►

Sign in to add a comment