Issue metadata
Sign in to add a comment
|
sunspider failing on 6 builders |
||||||||||||||||||||
Issue descriptionsunspider failing on 6 builders Builders failed on: - Linux Perf: https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf - Win 10 Perf: https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf - Win 7 ATI GPU Perf: https://build.chromium.org/p/chromium.perf/builders/Win%207%20ATI%20GPU%20Perf - Win 7 Intel GPU Perf: https://build.chromium.org/p/chromium.perf/builders/Win%207%20Intel%20GPU%20Perf - Win 8 Perf: https://build.chromium.org/p/chromium.perf/builders/Win%208%20Perf - Win Zenbook Perf: https://build.chromium.org/p/chromium.perf/builders/Win%20Zenbook%20Perf
,
Apr 17 2017
Lunching a regression bisect on the benchmark_duration since the benchmark currently timeouts.
,
Apr 17 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: linux_perf_bisect Benchmark : sunspider Metric : benchmark_duration/benchmark_duration Revision Result N chromium@464246 0.350898 +- 0.3909 21 good chromium@464349 2.29648 +- 11.3725 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982048640897775440 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5856316829990912 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 17 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982042919529290288
,
Apr 17 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: linux_perf_bisect Benchmark : sunspider Metric : benchmark_duration/benchmark_duration Revision Result N chromium@464246 0.33116 +- 0.178776 21 good chromium@464349 1.80441 +- 10.637 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982042919529290288 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5856316829990912 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 17 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982035951409017744
,
Apr 17 2017
Trying a return code bisect
,
Apr 17 2017
=== BISECT JOB RESULTS === NO Test failure found Bisect Details Configuration: linux_perf_bisect Benchmark : sunspider Metric : benchmark_duration/benchmark_duration Revision Exit Code N chromium@464246 0 +- N/A 5 good chromium@464349 0 +- N/A 5 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982035951409017744 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5803615232458752 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 18 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981964097060799168
,
Apr 18 2017
,
Apr 18 2017
I'm not sure where you got the revision range for the return code bisect? The debug button here: https://chromeperf.appspot.com/report?sid=aac0dd8b335248f537b95064ae5ee96d24d03480392f33be8586e16dff251fbf Says r464922:r465010. I'm going to try bisecting on that range.
,
Apr 18 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981959046088236400
,
Apr 18 2017
Also an interesting note: the screenshot on the failure has "undefined" as output metrics, I don't see this running locally on Mac (I see numbers): https://00e9e64bac430861b8072da2127de6df8da2a187abe8caff87-apidata.googleusercontent.com/download/storage/v1/b/chrome-telemetry-output/o/profiler-file-id_0-2017-04-18_03-01-17413.png?qk=AD5uMEvovSKSpWQFqux4zVvzNOL58eRyAceUijx19g7OSpHtKkLfmY8TQn6p9evUiZR-KjXUI-IBJBELwDf5KVayB11HuiY5i3CHhc2hHAjInDh-DkT0tGpabHh_B0XfA77qhdbLD172wlDCnwwaRLyV-v9QlS3fDftIeVuWuqZiKChcm9h1Xoy-UGFKb9aoYNP3vOAgz7IWSVq7ct5KUXewAFOxUClWnnomdLy-A7xlqUNkvPrYJD_Q_mRueNlzM4Nn-OugNLCKIKT2EZA0vguimJ4cYjYmuBNzomNZlPxGlj0pmR2vrilSVH5F0MzPFqA1DQ7rlwtYC2dxReH6h4FqHu0KMjvvlrjVRkTD5cH0dW6Sry1wHFaY5luCpiIHBrLSW1KoXT7E57hwvIXyvEYXQM6EWe54e0x49Yx7Ypc8BITtpIHKBsjMw1Fu3UfISr4wJ7L0eMMSk0JaW8wfDbO-IoOOpHfEb1h--mqNmurlo8zCgH5inGj4pKHt5h3GDAzvLWmctqGcFbHbxeBJI5yUnxsgVC8qFNK--TiQK-Z2Fa0aOvocHXgN7lJK5KyXat1-L5U3ZZlpPf3inXn8r7S18wkFfqlkykTvrIHcrFMVHFPyeMLPi0OgXyhwFXP7sUru0c4IWlxSVdqVJ5vzhFCVLQuKDzPahfGVswsr065yuCM8cz02v1X7LaB6FIK65PEgZMr4XOXPJ-AUihFyFRrxgXUL1xtQ20hMLvTaVJVS_EWQUvt1hHJ7aKSVY3OqTp5-ZzelySJmuop7tlAEK4nThV9EFjps63URPN0SWcwjMGhDnjYS79hvJD8sA4rGUb0blAIblMJ6RBUG3FBCsTP5Jn7MzdMPUw
,
Apr 18 2017
The range r464246:r464349 seems to match the chromium revisions on https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf where sunspider seemingly first started failing, ie. on builds 564/565. Sunspider had a successful run on build 577, so this looks flakey?
,
Apr 18 2017
Yes it is flaky but mostly failing, I changed the ranges in case the flakiness start point is earlier than the first failure. Most of my ranges are based on the benchmark_duration spike in the graph assuming that failing --> triggers timeout --> affects benchmark_duration.
,
Apr 18 2017
=== BISECT JOB RESULTS === NO Test failure found Bisect Details Configuration: linux_perf_bisect Benchmark : sunspider Metric : Total/Total Revision Exit Code N chromium@464922 1 +- N/A 20 good chromium@465010 1 +- N/A 20 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981959046088236400 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5888498818613248 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 18 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: linux_perf_bisect Benchmark : sunspider Metric : benchmark_duration/benchmark_duration Revision Result N chromium@461528 0.356595 +- 0.440258 21 good chromium@464350 2.0583 +- 11.0596 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981964097060799168 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5800358237962240 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 18 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981944772014837216
,
Apr 19 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: linux_perf_bisect Benchmark : sunspider Metric : benchmark_duration/benchmark_duration Revision Result N chromium@461528 0.352703 +- 0.425653 21 good chromium@464350 1.07455 +- 8.30167 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981944772014837216 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5800358237962240 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 26 2017
I think we are going to have to disable sunspider benchmark as it is no longer passing, but flaky enough that the failure doesn't bisect.
,
Apr 26 2017
(also failing on android)
,
Apr 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/19045842e1eff8063bd5aca05c8a94aa2035f7a5 commit 19045842e1eff8063bd5aca05c8a94aa2035f7a5 Author: sullivan <sullivan@chromium.org> Date: Thu Apr 27 00:04:35 2017 Disable failing sunspider test on all platforms BUG= 712208 TBR=nednguyen@google.com Review-Url: https://codereview.chromium.org/2844003003 Cr-Commit-Position: refs/heads/master@{#467521} [modify] https://crrev.com/19045842e1eff8063bd5aca05c8a94aa2035f7a5/tools/perf/benchmarks/sunspider.py
,
Apr 27 2017
Looks like the JS condition
window.location.pathname.indexOf("results.html") >= 0'
'&& typeof(output) != "undefined"
is not true anymore. Reference https://codesearch.chromium.org/chromium/src/tools/perf/benchmarks/sunspider.py?rcl=659c8284c33baf483121f1248d5aaa989d5dabe0&l=95
Adding benchmark owners.
,
Apr 27 2017
Assigning to perf sheriff
,
Apr 27 2017
Works for me. hablich@ thinks it might be infra related.
,
Apr 27 2017
Background: The JavaScript expression seems to still fire correctly when the test runs on a workstation without any flakyness. Seems like the 300 seconds timeout are not enough anymore though. It is strange that a 5 minute timeout is not enough when the benchmark locally runs for ... 15 seconds. This hints that something on the hardware side has changed.
,
Apr 27 2017
Ned, Stephen: see #26. Can you take a look at why this test doesn't finish on the bots?
,
Apr 27 2017
There is no reason to believe that this is a bot problem given that the "reference" benchmark are running fine on the same bot. https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf/builds/722 sunspider fails after 5 minutes sunspider.reference passes after 30s To me the flakiness is more likely to be a Chrome binary bug. Maybe v8, maybe blink, I am not sure.
,
Apr 27 2017
This graph shows the clear differences between the two binaries: https://chromeperf.appspot.com/report?sid=221fe881082a4a03021a16b78cc82e5ef4645370a14876e4aafc0c927e7a3e04
,
Apr 27 2017
,
Apr 28 2017
Hmmm, I doubt this is V8 related. I looked at a few of the spikes and the V8 changes are not related or there are no changes at all (e.g. http://test-results.appspot.com/revision_range?start=464857&end=464866). If it would be something V8 related the JavaScript expression would never succeed. Is the ref build running the same Telemetry/Catapult?
,
Apr 28 2017
Yes, it does. Everything is exactly the same, except Chrome binary. Note that since this is a flaky failure type, I don't think looking at the spike point in the graph can help us find out the culprit.
,
Apr 28 2017
Based on the debug screenshot https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_0-2017-04-13_12-50-0770910.png, it looks like the test finished but the output results is not in the DOM. Probably some blink bug?
,
Apr 28 2017
Possible or maybe a GC (V8 or oilpan) thing. We had one GC bug in the past where DOM elements were collected even if they were not dead. This bug is long fixed though and I would be surprised if this triggers only on Sunspider. +mlippautz and +haraken to confirm that this is not the case here.
,
Apr 28 2017
Hard to say. If it's only happening on M57 then it is likely the bug. We also recently had some hickups after some bindings change which got reverted in 4a97922bec93566369465c2e3d7facc45d165005. Is it gone after this one? (I don't have a full bindings overview; only the stuff that is related to wrapper tracing.)
,
May 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/14232dcc19c1dc66c2adad323dfda282b69cb0c8 commit 14232dcc19c1dc66c2adad323dfda282b69cb0c8 Author: nednguyen <nednguyen@google.com> Date: Mon May 15 17:14:04 2017 Remove sunspider benchmark Since this benchmark doesn't use press benchmark harness, we clean it up to reduce technical debt (also see https://bugs.chromium.org/p/chromium/issues/detail?id=708103#c13 for further context). In addition, the benchmark has been disabled everywhere due to a crash bug ( crbug.com/712208 ), removing this won't reduce the current coverage anyway. BUG= 712208 , 714231 Review-Url: https://codereview.chromium.org/2874983003 Cr-Commit-Position: refs/heads/master@{#471812} [modify] https://crrev.com/14232dcc19c1dc66c2adad323dfda282b69cb0c8/tools/perf/benchmark.csv [delete] https://crrev.com/8fef90770e83bf54f895c971ff08d5016ddabefc/tools/perf/benchmarks/sunspider.py
,
Aug 22 2017
|
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Apr 17 2017