Issue metadata
Sign in to add a comment
|
14.2%-72.3% regression in blink_perf.layout at 406285:406326 |
||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Jul 21 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9006484586050697904
,
Jul 21 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9006484394584593040
,
Jul 22 2016
===== BISECT JOB RESULTS ===== Status: completed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. === Warnings === The following warnings were raised by the bisect job: * Bisect failed to reproduce the regression with enough confidence. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@406287 191.006 4.40163 18 good chromium@406326 191.43 3.62163 18 bad Bisect job ran on: winx64_zen_perf_bisect Bug ID: 630462 Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.layout Test Metric: floats_2_100_nested/floats_2_100_nested Relative Change: 1.26% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/288 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006484586050697904 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5876423835779072 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jul 22 2016
=== Auto-CCing suspected CL author halliwell@chromium.org === Hi halliwell@chromium.org, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : [Chromecast] Default desktop windows to 720p + switch for 1080p Author : halliwell Commit description: This should prevent need for resizing windows when taking screenshots, although note that 1080p window option won't work on a 1080p monitor, because the window actually gets forced to be slightly smaller. 1080p window testing should be done with 4K monitor. BUG=internal b/29226425 TEST=created both 720 and 1080p windows, cast the viewport dims test page Review-Url: https://codereview.chromium.org/2155863002 Cr-Commit-Position: refs/heads/master@{#406308} Commit : e63dea9d25f8c7d7c2f562c1bea27ba1f2d57a12 Date : Tue Jul 19 18:03:06 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@406287 700.892 66.9972 12 good chromium@406307 627.376 109.258 5 good chromium@406308 792.003 12.2937 5 bad <-- chromium@406309 816.077 10.8124 5 bad chromium@406310 785.187 5.79939 5 bad chromium@406312 829.246 4.16506 5 bad chromium@406317 808.047 19.2913 5 bad chromium@406326 724.238 10.903 18 bad Bisect job ran on: winx64_zen_perf_bisect Bug ID: 630462 Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.layout Test Metric: fixed-grid-lots-of-data/fixed-grid-lots-of-data Relative Change: 0.86% Score: 98.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/289 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006484394584593040 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5796296925904896 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jul 22 2016
Can't possibly be caused by chromecast/ CL.
,
Jul 22 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9006419094278491392
,
Jul 22 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9006419085884365088
,
Jul 22 2016
Yeah, gonna have to agree with you there. Pretty noisy bisect, sent some more with more repeats.
,
Jul 23 2016
=== Auto-CCing suspected CL author szurgotc@amazon.com === Hi szurgotc@amazon.com, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : [chromedriver] Handle missing WCHAN field in output from ps. Author : szurgotc Commit description: Accounts for issue where ps command isn't returning a - when a field is blank, throwing off the tokenization. This update takes last field as the app name (instead of absolute index of [8]) and allows for 1 missing field. BUG= chromedriver:1378 Review-Url: https://codereview.chromium.org/2127993002 Cr-Commit-Position: refs/heads/master@{#406296} Commit : 9ac8817f503a25ed86db93584813e8b6e5c7f6a8 Date : Tue Jul 19 17:01:35 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@406287 15.6645 0.530972 18 good chromium@406292 15.7868 0.0483372 5 good chromium@406295 15.6687 0.0716607 5 good chromium@406296 15.8472 0.0690788 5 bad <-- chromium@406297 15.895 0.0421691 8 bad chromium@406307 15.8264 0.347309 18 bad chromium@406326 15.9711 0.117602 12 bad Bisect job ran on: winx64_zen_perf_bisect Bug ID: 630462 Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.layout Test Metric: flexbox-lots-of-data/flexbox-lots-of-data Relative Change: 1.15% Score: 99.8 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/292 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006419085884365088 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5826635635359744 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jul 23 2016
===== BISECT JOB RESULTS ===== Status: completed ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@406287 190.378 2.28854 5 good chromium@406297 191.84 1.12764 5 good chromium@406302 191.804 0.586013 5 good chromium@406305 196.952 4.49165 27 good chromium@406306 201.224 4.05891 27 bad chromium@406307 201.301 4.11884 27 bad chromium@406326 203.041 1.14525 5 bad Bisect job ran on: winx64_zen_perf_bisect Bug ID: 630462 Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.layout Test Metric: floats_2_100_nested/floats_2_100_nested Relative Change: 6.65% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/291 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006419094278491392 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5866664730558464 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jul 25 2016
szurgotc's CL only affects chromedriver, and only when running tests against Android, so it's unlikely to be related to this.
,
Aug 1 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9005515656196398704
,
Aug 1 2016
Trying another bisect.
,
Aug 1 2016
=== Auto-CCing suspected CL author wangxianzhu@chromium.org === Hi wangxianzhu@chromium.org, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Fix PaintPropertyTreePrinter Author : wangxianzhu Commit description: Previously it printed nothing because it didn't collect the root nodes. Review-Url: https://codereview.chromium.org/2157173002 Cr-Commit-Position: refs/heads/master@{#406132} Commit : f070db9ec34a891c85e375827696f733db8d21c1 Date : Mon Jul 18 23:32:16 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@406100 183.029 0.874071 8 good chromium@406116 181.86 1.1669 5 good chromium@406125 181.227 1.65453 5 good chromium@406129 181.747 0.925614 5 good chromium@406131 181.436 1.30174 5 good chromium@406132 193.801 7.55922 5 bad <-- chromium@406176 197.296 9.53877 5 bad chromium@406250 188.32 0.897887 5 bad chromium@406400 188.704 8.41519 8 bad Bisect job ran on: winx64_zen_perf_bisect Bug ID: 630462 Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.layout Test Metric: floats_100_100_nested/floats_100_100_nested Relative Change: 4.38% Score: 95.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/332 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005515656196398704 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5219634106597376 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 1 2016
My CL just changes how to print a PaintPropertyTree, which is used for slimming paint v2 only, and not used in perf tests.
,
Aug 8 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004884502652557696
,
Aug 8 2016
Fired off another bisect.
,
Aug 9 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004761989321687744
,
Aug 10 2016
===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Moves WindowCycleList/Controller to ash/common Author : sky Commit description: BUG=582590, 629497 TEST=covered by tests R=estade@chromium.org, jamescook@chromium.org Review-Url: https://codereview.chromium.org/2157393002 Cr-Commit-Position: refs/heads/master@{#406310} Commit : 37f0dd1f6a3631a8d320f5dec3dc1111f94f640c Date : Tue Jul 19 18:03:08 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@406287 1859.09 8.58477 5 good chromium@406307 1861.11 53.5077 27 good chromium@406309 1771.02 196.348 8 good chromium@406310 1815.69 194.838 27 bad <-- chromium@406312 1851.12 145.801 18 bad chromium@406317 1759.22 406.029 8 bad chromium@406326 1865.42 42.9921 8 bad Bisect job ran on: winx64_zen_perf_bisect Bug ID: 630462 Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.layout Test Metric: flexbox-column-nowrap/flexbox-column-nowrap Relative Change: 1.38% Score: 0.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/370 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004884502652557696 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5329435583053824 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 10 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004759226589713568
,
Aug 10 2016
=== Auto-CCing suspected CL author yunchao.he@intel.com === Hi yunchao.he@intel.com, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : command buffer: remove unused code in GLES2DecoderImpl. Author : yunchao.he Commit description: ERRORSTATE_SET_GL_ERROR_INVALID_PARAM has been removed. It is not defined at all now. BUG= CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Review-Url: https://codereview.chromium.org/2165453003 Cr-Commit-Position: refs/heads/master@{#406290} Commit : dfabf15df7689b76d0a983197f7365d6b35738ef Date : Tue Jul 19 16:20:44 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@406287 46.2662 0.569368 12 good chromium@406289 45.8499 0.204964 5 good chromium@406290 44.6094 0.174865 5 bad <-- chromium@406292 44.3257 0.268977 5 bad chromium@406297 45.1525 0.391779 5 bad chromium@406307 45.7341 0.729512 12 bad chromium@406326 45.9526 0.237333 8 bad Bisect job ran on: winx64_zen_perf_bisect Bug ID: 630462 Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.layout Test Metric: chapter-reflow/chapter-reflow Relative Change: 0.81% Score: 99.9 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/375 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004759226589713568 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5812786840469504 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 10 2016
Hi, my CL just only remove a couple of unused code. It is very easy: https://codereview.chromium.org/2165453003. I have 100% confidence that it should not impact the performance of any component in Chromium. Please revisit the perf issue and assign to an appropriate guy.
,
Aug 10 2016
Emil, can you take a look at this bug? It's pretty strange and we're not sure how to prioritize: * We are seeing big performance regressions in many blink_perf.layout tests, but only on one bot (zenbook running Win10; we have another Win10 machine that doesn't see these regressions) * The ref build doesn't move, so it doesn't seem to be due to a change on the bot * The bisects have pointed to 6 different CLs in the range with varying degrees of confidence, none of which seem a likely culprit. The biggest regression found by a bisect bot was 6%, while the graphs show up to 74%. Any idea what could be going on here?
,
Aug 10 2016
This is quite puzzling. Any change affecting the two floats and one flexbox column tests should also have affected the other flexbox and float tests. These are all microbenchmarks with relatively low amounts of data (while the versions with more data din't move). Given that it only affects the one bot intermittently and only those three tests I'm inclined to believe that it's due to cache misses. Perhaps the test run order changed slightly?
,
Aug 10 2016
Hmmm, the stdio for the tests has expired but it looks like the list of test results was the same size/order for the before/after builds: https://build.chromium.org/p/chromium.perf/builders/Win%20Zenbook%20Perf%20%283%29/builds/2172/steps/blink_perf.layout https://build.chromium.org/p/chromium.perf/builders/Win%20Zenbook%20Perf%20%283%29/builds/2173/steps/blink_perf.layout
,
Aug 10 2016
It does indeed. Interesting. Is there any way to check if any of the configuration or test harness changed between those two runs?
,
Aug 11 2016
There doesn't seem to be anything about the test directory, telemetry, or catapult in the CL range. +dtu, stip, any ideas on how we could check what changed in recipes, master config, bot config between these two builds? https://build.chromium.org/p/chromium.perf/builders/Win%20Zenbook%20Perf%20%283%29/builds/2172 https://build.chromium.org/p/chromium.perf/builders/Win%20Zenbook%20Perf%20%283%29/builds/2173
,
Aug 11 2016
The bisect tasks seem not to show the regression. For example, the task in #22 showed: chromium@406287 46.2662 0.569368 12 good chromium@406289 45.8499 0.204964 5 good chromium@406290 44.6094 0.174865 5 bad <-- chromium@406292 44.3257 0.268977 5 bad chromium@406297 45.1525 0.391779 5 bad chromium@406307 45.7341 0.729512 12 bad chromium@406326 45.9526 0.237333 8 bad The difference between the good and bad results is much smaller than the difference in the graph.
,
Aug 17 2016
,
Oct 11 2016
Perf sheriff ping
,
Oct 11 2016
Not sure what the next step is here given that we haven't been able to reproduce the results and that it only affects certain bots.
,
Oct 11 2016
As suggested by sullivan in #28, it's possible that something could have changed in recipes, master config, or bot config... And given that it only affects one bot (win-zenbook), and there is no ref build to compare against, maybe there is the theoretical possibility that this 40-60% change in the tests is entirely attributable to a change in the state of the machine (more processes running, less available memory, low battery, or something else?) Mike or Dave, do you think this might be worth investigating? Otherwise I'd suggesting closing this bug since we couldn't find a cause, so it seems likely that there's no real perf regression in Chrome :-/
,
Oct 19 2016
I think this probably isn't a real regression (see last comment). |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by rsch...@chromium.org
, Jul 21 2016