Issue metadata
Sign in to add a comment
|
9.2% regression in media.tough_video_cases at 437363:437418 |
||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Dec 12 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993457878542857136
,
Dec 12 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@437362 671.8 8911.98 156 good chromium@437418 743.611 9535.98 156 bad Bisect job ran on: mac_10_11_perf_bisect Bug ID: 673404 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases Test Metric: cputime/cputime Relative Change: 10.69% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_10_11_perf_bisect/builds/1114 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993457878542857136 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5800954207666176 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 13 2016
,
Dec 13 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993435353447943952
,
Dec 13 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@437362 677.517 8984.26 156 good chromium@437418 740.549 9344.46 156 bad Bisect job ran on: mac_10_11_perf_bisect Bug ID: 673404 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases Test Metric: cputime/cputime Relative Change: 9.30% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_10_11_perf_bisect/builds/1120 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993435353447943952 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5820690253479936 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 14 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993275310238875088
,
Dec 14 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@437362 688.681 9016.64 156 good chromium@437418 742.494 9375.68 156 bad Bisect job ran on: mac_10_11_perf_bisect Bug ID: 673404 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases Test Metric: cputime/cputime Relative Change: 7.81% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_10_11_perf_bisect/builds/1126 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993275310238875088 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5884225486585856 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jan 9 2017
Simon, Is there anything we can do at this point to figure out what caused the regression?
,
Jan 9 2017
Looks like the bisect is actually reproducing this quite reliably, but we're failing the MWU test. I have a bug out to try averaging the values, in the same way the dashboard does, just haven't had time to get to it. I'll block on that for now.
,
Jan 10 2017
For anyone reading this later: MWU = https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test Simon, Is the loop here https://cs.chromium.org/chromium/src/tools/auto_bisect/bisect_perf_regression.py?l=2365 succeeding in finding a possible culprit but then later on a test of confidence in that culprit is failing? Would it be possible to print the suspected culprit to logs that way we could look it up and see what we think? Also, why does the TESTED REVISIONS list above (comment #8) only list two tested revisions? It seems like it should have tested a full bisect's worth of revisions. I might be helpful to share that data so that developers could analyze it and understand what was happening.
,
Jan 10 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8990895199863938368
,
Jan 10 2017
re: #c11 It does print the suspected culprit, or even a range, if it has one to print. In this case, the bisect tested the first/last revisions, but it wasn't confident there was enough of a difference in the values from those revisions and aborted. Kinda wanna try the fix suggested in crbug.com/669608 to see if it would help, might have some time this week or next depending on what takes priority. I'd be happy to chat offline if you still have any questions.
,
Jan 10 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : media.tough_video_cases Metric : cputime/cputime Revision Result N chromium@437362 681.524 +- 9141.99 156 good chromium@437418 740.906 +- 9372.33 156 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8990895199863938368 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5227323898462208 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jan 12 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8990641799651861520
,
Jan 12 2017
permalink to bisect logs from comment 14: https://luci-milo.appspot.com/buildbot/tryserver.chromium.perf/mac_10_11_perf_bisect/1160
,
Jan 13 2017
=== BISECT JOB RESULTS === Perf regression found but unable to narrow commit range Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : media.tough_video_cases Metric : cputime/cputime Suspected Commit Range 16 commits in range https://chromium.googlesource.com/chromium/src/+log/e433914989869a14fe15f6369fd3149a1b3b7eb8..5c49d183f4713e682965e831387a1f4bdac32034 Revision Result N chromium@437362 682.049 +- 17.1962 6 good chromium@437379 684.781 +- 9.18838 6 good --- --- --- too many build failures to list chromium@437394 --- --- build failure chromium@437395 738.345 +- 9.97353 6 bad chromium@437418 741.742 +- 10.4483 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8990641799651861520 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5548471890739200 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jan 14 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8990533899224542752
,
Jan 14 2017
=== BISECT JOB RESULTS === Perf regression found but unable to narrow commit range Build failures prevented the bisect from narrowing the range further. Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : media.tough_video_cases Metric : cputime/cputime Change : 9.85% | 674.161241282 -> 740.561521603 Suspected Commit Range 15 commits in range https://chromium.googlesource.com/chromium/src/+log/f32ade75b611e31c6f43914a93c3f7067bcc8cce..5c49d183f4713e682965e831387a1f4bdac32034 Revision Result N chromium@437362 674.161 +- 20.813 6 good chromium@437379 685.498 +- 16.329 6 good chromium@437380 684.21 +- 20.0923 6 good chromium@437381 --- --- build failure --- --- --- too many build failures to list chromium@437394 --- --- build failure chromium@437395 740.754 +- 12.8966 6 bad chromium@437418 740.562 +- 9.64975 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8990533899224542752 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6394481932238848 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Mar 1 2017
All CLs in range: http://test-results.appspot.com/revision_range?start=437379&end=437395
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982651630306108320
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982611977913662512
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982580391148558544
,
Apr 12 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: mac_10_11_perf_bisect Benchmark : media.tough_video_cases Metric : cputime/cputime Revision Result N chromium@437362 681.524 +- 9141.99 156 good chromium@437418 740.906 +- 9372.33 156 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982651630306108320 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5227323898462208 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Jun 23 2017
|
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by hubbe@chromium.org
, Dec 12 2016