New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 687028 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

10.0% regression in battor.power_cases/story:power_avg on chromium-rel-mac-retina at 446334:446433

Project Member Reported by charliea@chromium.org, Jan 31 2017

Issue description

Performance dashboard identified a 10.0% regression in battor.power_cases/story:power_avg on chromium-rel-mac-retina at revision range 446334:446433. Graph: https://chromeperf.appspot.com/report?masters=ChromiumPerf&bots=chromium-rel-mac-retina&tests=battor.power_cases%2Fstory%3Apower_avg&checked=story%3Apower_avg%2Cstory%3Apower_avg_ref%2Cref&rev=446433

 
Quoting my email about the alert:

"""
I uh... have no idea how to interpret this. It seems like it's an even increase across all user stories. It also matches with a Catapult roll with a whole schload of changes because the roll had been blocked for several days. Just to be sure that this is an environmental change and not a real regression, I'm going to file a bug and launch a bisect.
"""
Just to be clear, the reason that we're reasonably confident this isn't a real regression is that it happened on both the ref build and non-ref builds. However, the ref build seems to have regressed twice as much as the non-ref build, which is strange for a purely environmental change.
Project Member

Comment 4 by 42576172...@developer.gserviceaccount.com, Jan 31 2017

Owner: alexandermont@chromium.org

=== Auto-CCing suspected CL author alexandermont@chromium.org ===

Hi alexandermont@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : alexandermont
  Commit : 6e1486bd6ffe5f45c70b411d55ae3f2efcdb2171
  Date   : Thu Jan 26 01:06:17 2017
  Subject: Make the whole story power metric not depend on Chrome trace.

Bisect Details
  Configuration: mac_retina_perf_bisect
  Benchmark    : battor.power_cases
  Metric       : story:power_avg/about_blank
  Change       : 11.80% | 6.83160820648 -> 7.63744493596

Revision                                 Result                    N
chromium@446333                          6.83161 +- 0.273522       6      good
chromium@446383                          6.82821 +- 0.0773776      6      good
chromium@446408                          6.81337 +- 0.172974       9      good
chromium@446415                          7.0993 +- 2.2252          9      good
chromium@446415,catapult@fd4a29f5be      6.85942 +- 0.366953       6      good
chromium@446415,catapult@866671b725      6.79895 +- 0.139586       6      good
chromium@446415,catapult@6f3efa6082      6.76057 +- 0.190365       6      good
chromium@446415,catapult@6e1486bd6f      7.60336 +- 0.216114       5      bad       <--
chromium@446415,catapult@3584a988b4      7.66512 +- 0.473752       6      bad
chromium@446416                          7.66324 +- 0.250959       9      bad
chromium@446417                          7.69448 +- 0.43388        9      bad
chromium@446418                          7.71325 +- 0.685739       9      bad
chromium@446421                          7.74521 +- 0.537539       9      bad
chromium@446433                          7.63744 +- 0.201502       6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=about.blank battor.power_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8988976888744695824

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5347781373329408


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Hey Alex: could you look into why your change affected the power metric for this benchmark?
Hey Alex,

Can you set this bug's status WontFix with an explanation of why we're not going to fix it?
Closing this as it isn't a real regression: it just reflects a change in the metric logic.
Status: WontFix (was: Untriaged)

Sign in to add a comment