New issue
Advanced search Search tips

Issue 670882 link

Starred by 1 user
Status: WontFix
Owner: ----
Closed: Jul 2017
Cc:
qyears...@chromium.org
iclell...@chromium.org
sullivan@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

63.4%-68.5% regression in startup.cold.blank_page at 435386:435537

Project Member Reported by qyears...@chromium.org, Dec 2 2016 
See the link to graphs below.
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Dec 3 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@435385  3770.78  2623.77  27  good
chromium@435537  3825.15  2409.59  27  bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 670882

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests startup.cold.blank_page
Test Metric: first_main_frame_load_time/first_main_frame_load_time
Relative Change: 1.44%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2289
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994346768679505648


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5295531348197376

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 6 by 42576172...@developer.gserviceaccount.com, Dec 6 2016

Cc: iclell...@chromium.org
Owner: iclell...@chromium.org

=== PERF REGRESSION ===


=== Auto-CCing suspected CL author iclelland@chromium.org ===

Hi iclelland@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Disable flaky virtual/threaded/fast/scroll-behavior/smooth-scroll/ongoing-smooth-scroll-vertical-rl-anchors.html test on Windows and Linux
Author  : iclelland
Commit description:
  
BUG= 670066 
TBR=flackr@chromium.org

Review-Url: https://codereview.chromium.org/2546573002
Cr-Commit-Position: refs/heads/master@{#435457}
Commit  : 61ffd09bb211792738af0e912b553bbe38e9ba0c
Date    : Wed Nov 30 22:49:55 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@435385  3652.44  1707.32  18  good
chromium@435423  3831.63  492.451  8   good
chromium@435442  4530.5   707.209  8   good
chromium@435452  4126.38  433.188  8   good
chromium@435455  3795.63  511.378  8   good
chromium@435456  3847.0   767.88   8   good
chromium@435457  3201.2   80.8505  5   bad    <--
chromium@435461  3090.8   220.85   5   bad
chromium@435537  3316.33  1117.23  18  bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 670882

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests startup.cold.blank_page
Test Metric: first_main_frame_load_time/first_main_frame_load_time
Relative Change: 9.20%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2293
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994096495056471152


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5051898455916544

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 7 by 42576172...@developer.gserviceaccount.com, Dec 6 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@435385  3767.22  2983.99  27  good
chromium@435537  3722.19  1962.97  27  bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 670882

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests startup.cold.blank_page
Test Metric: first_main_frame_load_time/first_main_frame_load_time
Relative Change: 1.20%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2295
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994061645062900576


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5212853932392448

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 8 by iclell...@chromium.org, Dec 6 2016

Owner: qyears...@chromium.org
This test looks like it has a very high variance -- stddev is often > 50% of the mean. It identified a CL that disabled a flaky layout test, which I'm pretty certain can't possibly be correct.

Reassigning back :(

Comment 9 by qyears...@chromium.org, Dec 6 2016

Cc: sullivan@chromium.org
Owner: ----
It does look like it has high variance, or the result distribution is bimodal, which looks like it led to an incorrect bisect result.

These alerts are on one platform, with no ref build results, and a pattern of going up and down by at least 60% after the alert.

There's no owner for this test, so I think that this should be closed, based on the above.

I'm not sure if the startup tests are generally too noisy, or whether it might just be noisy some of the time, on some platforms.
Project Member

Comment 11 by 42576172...@developer.gserviceaccount.com, Dec 7 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@435385  3756.81  3028.62  27  good
chromium@435537  3919.44  2567.95  27  bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 670882

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests startup.cold.blank_page
Test Metric: first_main_frame_load_time/first_main_frame_load_time
Relative Change: 4.33%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2298
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993981609269813616


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5575388263088128

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Dec 13 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@435385  1753.81  2440.4   21  good
chromium@435537  1612.43  704.948  21  bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 670882

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests startup.cold.blank_page
Test Metric: window_display_time/window_display_time
Relative Change: 8.06%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2303
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993379224925602848


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5789322496704512

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 17 by 42576172...@developer.gserviceaccount.com, Apr 11 2017 


=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: win_8_perf_bisect
  Benchmark    : startup.cold.blank_page
  Metric       : first_main_frame_load_time/first_main_frame_load_time

Revision             Result                  N
chromium@435385      4091.14 +- 2739.55      21      good
chromium@435537      3808.48 +- 1977.44      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests startup.cold.blank_page

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982651827203854624

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5212853932392448


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Comment 19 by benhenry@chromium.org, Jul 27 2017

Status: WontFix (was: Untriaged)
This alert was found before M-60 branched. Closing as WontFix as this is believed to either be invalid or non-reproducible.
►

Sign in to add a comment