New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 645313 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Nov 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

23.3% regression in blink_perf.canvas at 416452:416453

Project Member Reported by briander...@chromium.org, Sep 9 2016

Issue description

See the link to graphs below.
 
All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=645313

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgyeD2pgoM


Bot(s) for this bug's original alert(s):

android-nexus5

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N   Good?
chromium@416451  110.322  1.28027   18  good
chromium@416453  110.345  0.994471  18  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 645313

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.canvas
Test Metric: putImageData/putImageData
Relative Change: 0.90%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4104
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9002041730709104096


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5816418812035072

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Owner: ----
Perf fixit: removing perf sheriff as owner of bugs to clarify that the rotation is responsible for triage.

Comment 5 by benhenry@google.com, Nov 17 2016

Status: Fixed (was: Assigned)
Why would we bisect this? It's an improvement!
Status: Available (was: Fixed)
Re-opening. Looks like a valid regression. The y-axis is runs/s, so higher is better.

Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Nov 19 2016

Cc: perezju@chromium.org
Owner: perezju@chromium.org

=== Auto-CCing suspected CL author perezju@chromium.org ===

Hi perezju@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [Telemetry] Switch ShouldTearDownStateAfterEachStoryRun to True as default
Author  : perezju
Commit description:
  
To reduce complexity of benchmarks and imcrease stability of measurements
benchmarks will now tear down the state (i.e. restart the browser) between
each page/story of a test.

Perf sherfiffs warning: This CL is expected to affect measurements of
many benchmarks.

BUG= chromium:640990 

Review-Url: https://codereview.chromium.org/2304893002
Commit  : 4f812c9035e6cf9bad6774bb6ff71c1822167e0a
Date    : Sat Sep 03 10:45:26 2016


===== TESTED REVISIONS =====
Revision                             Mean     Std Dev  N   Good?
chromium@416263                      134.479  32.1516  25  good
chromium@416384                      135.305  17.4451  25  good
chromium@416445                      143.976  17.344   25  good
chromium@416449                      139.348  19.75    25  good
chromium@416451                      143.14   19.0927  25  good
chromium@416452                      146.929  14.4159  25  good
chromium@416452,catapult@4f812c9035  110.431  6.88509  25  bad    <--
chromium@416453                      111.028  7.5283   25  bad
chromium@416460                      110.697  8.47733  25  bad
chromium@416475                      110.972  7.2581   25  bad
chromium@416505                      110.156  7.96942  25  bad
chromium@416747                      110.575  8.25142  25  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 645313

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests blink_perf.canvas
Test Metric: putImageData/putImageData
Relative Change: 17.77%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4357
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995620069876942016


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5809242664075264

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Cc: junov@chromium.org nednguyen@chromium.org
Status: WontFix (was: Available)
Not a real regression, but the test being re-baselined.

The benchmark now restarts the browser between individual stories, and so we're measuring in a more "cold" scenario.

+Ned FYI, +junov benchmark owner.

Sign in to add a comment