New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 666312 link

Starred by 1 user
Status: WontFix
Owner:
█ aelias@chromium.org
inactive
Closed: Apr 2017
Cc:
simonhatch@chromium.org
█ aelias@chromium.org
mustaq@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression

Blocked on:
issue 668529


Sign in to add a comment

1.1%-74% regression in smoothness.top_25_smooth at 432301:432475

Project Member Reported by mustaq@chromium.org, Nov 17 2016 
See the link to graphs below.
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Nov 17 2016 

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_10_perf_bisect/builds/781
Failure reason: the build has failed.
Additional errors:
Either of the initial "good" or "bad" revisions failed to be tested or built.

Comment 4 by sullivan@chromium.org, Nov 17 2016 

r432340 didn't build on Windows. Trying on linux.
Project Member

Comment 6 by 42576172...@developer.gserviceaccount.com, Nov 17 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N    Good?
chromium@432340  16.7654  53.1318  350  good
chromium@432405  16.9313  60.0903  350  bad

Bisect job ran on: linux_perf_bisect
Bug ID: 666312

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=https...www.google.com..hl.en.q.barack.obama smoothness.top_25_smooth
Test Metric: frame_times/https___www.google.com__hl_en_q_barack+obama
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6846
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995730330838136576


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4970184899559424

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 9 by 42576172...@developer.gserviceaccount.com, Nov 21 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N    Good?
chromium@432300  16.7538  24.2632  379  good
chromium@432475  16.8928  33.9664  380  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 666312

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=https...www.google.com..hl.en.q.barack.obama smoothness.top_25_smooth
Test Metric: frame_times/https___www.google.com__hl_en_q_barack+obama
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/608
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995363091932977600


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5618812940976128

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 10 by 42576172...@developer.gserviceaccount.com, Nov 21 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N    Good?
chromium@432300  17.4509  57.5669  135  good
chromium@432475  18.0044  90.3747  135  bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 666312

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=https...www.google.com.calendar. smoothness.top_25_smooth
Test Metric: frame_times/https___www.google.com_calendar_
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2278
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995363111082017056


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6465464677433344

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 14 by 42576172...@developer.gserviceaccount.com, Nov 22 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N    Good?
chromium@432300  17.3196  52.5541  135  good
chromium@432475  17.9924  89.9459  135  bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 666312

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=https...www.google.com.calendar. smoothness.top_25_smooth
Test Metric: frame_times/https___www.google.com_calendar_
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2281
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995271089992504256


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5287844875075584

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 15 by 42576172...@developer.gserviceaccount.com, Nov 22 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N    Good?
chromium@432300  16.7127  14.0779  175  good
chromium@432475  17.0981  32.3556  175  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 666312

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=https...www.google.com.calendar. smoothness.top_25_smooth
Test Metric: frame_times/https___www.google.com_calendar_
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/610
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995271061928057920


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5883256191320064

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 16 by 42576172...@developer.gserviceaccount.com, Nov 22 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Step('Compare samples') failed with return_code 1
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean  Std Dev  N  Good?
chromium@432311  N/A   N/A      0  good
chromium@432475  N/A   N/A      0  bad

Bisect job ran on: win_perf_bisect
Bug ID: 666312

Test Command: .\src\out\Release\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: CastV2Performance_gpu_60fps/av_sync
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7050
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995271106034781744


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4970729454436352

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 17 by sullivan@chromium.org, Nov 23 2016

Cc: simonhatch@chromium.org
Simon, can you take a look at the failure here?

Comment 18 by simonhatch@chromium.org, Nov 24 2016

Blockedon: 668529
So looks like "Compare Samples" is failing since it's been given no inputs, which means no values were produced from the "Gathering Reference Values" step, this will get cleared up when 666447 lands and "Gathering Reference Values" will fail if it doesn't generate any actual values to bisect on. But that'll just make this a bit clearer where it's failing.


The test is actually generating the metrics. Taking a look at the captured output, can see that the test metric is generated:

@@@STEP_LOG_LINE@Captured Output@*RESULT CastV2Performance_gpu_60fps: av_sync= {-38.556311,27.849332} ms@@@


Looks like the bisect recipe just isn't reading them back in. In a normal run we'd see:

Gathering reference values.write buildbot output to disk (10)
Gathering reference values.Reading chartjson results (10)

The bisect recipe determines if it's chartjson or valueset output by looking at the command (perf_test.py:94) and reads in the appropriate file. Since neither of these are set (I'm not clear if these gtest can output either), the recipe doesn't read any usable values back in.

Logged 668529.
Project Member

Comment 20 by 42576172...@developer.gserviceaccount.com, Dec 6 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Testing the "good" revision failed: Test runs failed to produce output.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean  Std Dev  N  Good?
chromium@432311  N/A   N/A      0  good
chromium@432475  N/A   N/A      0  bad

Bisect job ran on: win_perf_bisect
Bug ID: 666312

Test Command: .\src\out\Release\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: CastV2Performance_gpu_60fps/av_sync
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/7064
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994012134011163232


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5833940999340032

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 22 by 42576172...@developer.gserviceaccount.com, Dec 14 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N  Good?
chromium@432300  32.7406  0.137614  6  good
chromium@432475  33.206   0.111976  6  bad

Bisect job ran on: win_x64_perf_bisect
Bug ID: 666312

Test Command: .\src\out\Release_x64\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: CastV2Performance_gpu_60fps/time_between_captures
Relative Change: 1.42%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_x64_perf_bisect/builds/1552
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993449711018718848


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5779709957242880

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 25 by 42576172...@developer.gserviceaccount.com, Apr 10 2017

Cc: aelias@chromium.org
Owner: aelias@chromium.org

=== Auto-CCing suspected CL author aelias@chromium.org ===

Hi aelias@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : aelias
  Commit : eb9ecefc55d1e3cf8498d0865568097fb9bbf285
  Date   : Wed Nov 16 06:29:13 2016
  Subject: Revert "Getting rid of DelegatedFrameData"

Bisect Details
  Configuration: linux_perf_bisect
  Benchmark    : smoothness.top_25_smooth
  Metric       : frame_times/https___www.google.com__hl_en_q_barack+obama
  Change       : 1.17% | 16.7601261905 -> 16.9556071429

Revision             Result                    N
chromium@432340      16.7601 +- 0.0654079      6      good
chromium@432373      16.7593 +- 0.0522174      6      good
chromium@432389      16.7534 +- 0.0662199      6      good
chromium@432390      16.9608 +- 0.0309214      6      bad       <--
chromium@432391      16.9444 +- 0.0884102      6      bad
chromium@432393      16.9814 +- 0.0579045      6      bad
chromium@432397      16.9509 +- 0.0724257      6      bad
chromium@432405      16.9556 +- 0.0637154      6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=https...www.google.com..hl.en.q.barack.obama smoothness.top_25_smooth

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982667466541815328

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4970184899559424


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Comment 26 by aelias@chromium.org, Apr 10 2017

Status: WontFix (was: Untriaged)
Right, that was a revert of a buggy performance optimizing CL, presumably it recovered on the reland.

Comment 27 by simonhatch@chromium.org, Apr 10 2017 

Sorry for the noise here, we fixed some issues on the dashboard and some ancient bisects happened to get restarted as a result of the fix. Or rather, they were finally able to run again.
►

Sign in to add a comment