Issue metadata
Sign in to add a comment
|
rasterize_and_record_micro.key_silk_cases failing on linux perf bots |
||||||||||||||||||||||||
Issue description
,
Oct 27 2016
===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Add serialization/deserialization for FakePictureLayer for testing. Author : xingliu Commit description: Previously we serialize/deserialize FakePictureLayer into PictureLayer. This introduced an issue in LayerTreeHostTestReadyToActivateNonEmpty and other unit test cases in layer_tree_host_unittests.cc that downcasting from PictureLayerImpl to FakePictureLayerImpl will fail. The data flow was: FakePictureLayer==>PictureLayer==>PictureLayerImpl==> FakePictureLayerImpl(downcast failure). This CL adds procedure to process FakePictureLayer, so it can be deserialized and down cast correctly. The data flow becomes: FakePictureLayer==>FakePictureLayer==>FakePictureLayerImpl BUG= 657871 CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel Review-Url: https://codereview.chromium.org/2451913002 Cr-Commit-Position: refs/heads/master@{#427886} Commit : b67229217f5b54fad888c74f6528be4af51e5a6e Date : Thu Oct 27 00:10:32 2016 ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@427874 0 N/A 2 good chromium@427880 0 N/A 2 good chromium@427883 0 N/A 2 good chromium@427885 0 N/A 2 good chromium@427886 1 N/A 2 bad <-- chromium@427898 1 N/A 2 bad chromium@427921 1 N/A 2 bad Bisect job ran on: linux_perf_bisect Bug ID: 660048 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests rasterize_and_record_micro.key_silk_cases Test Metric: pixels_rasterized/pixels_rasterized Relative Change: Zero to non-zero Score: 0.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6801 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997634694350438256 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5271682984443904 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Oct 27 2016
,
Oct 27 2016
,
Oct 27 2016
Can't possibly be that patch. The code touched by it is not used in any Chrome build.
,
Oct 27 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997618233885503744
,
Oct 27 2016
Started another bisect, hopefully it can catch the real culprit.
,
Oct 28 2016
===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : memory coordinator: Use free memory to guess available memory on Linux Author : bashi Commit description: Though there will be eaisly discardable memory such as buffers and caches, don't count them as "available" memory because discarding them will affect the overall performance of the system. BUG= 617492 Review-Url: https://codereview.chromium.org/2448923004 Cr-Commit-Position: refs/heads/master@{#427880} Commit : 8e656e05f77fb80739dc80bdfa1abfce1f45173d Date : Wed Oct 26 23:41:46 2016 ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@427870 0 N/A 4 good chromium@427878 0 N/A 4 good chromium@427879 0 N/A 4 good chromium@427880 1 N/A 4 bad <-- chromium@427882 1 N/A 4 bad chromium@427885 1 N/A 4 bad chromium@427900 1 N/A 4 bad chromium@427930 1 N/A 4 bad Bisect job ran on: linux_perf_bisect Bug ID: 660048 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests rasterize_and_record_micro.key_silk_cases Test Metric: pixels_rasterized/pixels_rasterized Relative Change: Zero to non-zero Score: 0.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6803 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997618233885503744 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5846541405782016 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Oct 28 2016
+bashi, do you mind taking a look.
,
Oct 28 2016
Thanks for triage but my CL won't affect rasterize_and_record_micro.key_silk_cases. I touched code which is only enabled when --enable-features=MemoryCoordinator is specified.
,
Oct 28 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997553806056924288
,
Oct 28 2016
I started another bisect, if it comes back the same I'm going to do a speculative revert and see if your cl is somehow interacting in strange ways.
,
Oct 28 2016
CCing Bashi so they can actually see my message above.
,
Oct 28 2016
=== Auto-CCing suspected CL author benjhayden@chromium.org === Hi benjhayden@chromium.org, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Relax RelatedHistogramBreakdown restrictions. Author : benjhayden Commit description: Currently, RelatedHistogramBreakdown keys must be suffixes of the referenced Histogram names. This would be annoying for memoryMetric, so this CL relaxes that constraint. In order to prevent the usability issue of clicking on a link that says X only to find that a histogram named Y is selected, this a title tooltip is shown when hovering on the legend key containing the full name of the referenced Histogram. BUG=catapult:#2933 Review-Url: https://codereview.chromium.org/2452983003 Commit : b1fcad56cd3362b942982ff8db6debb117d5005e Date : Wed Oct 26 23:09:35 2016 ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@427865 0 N/A 2 good chromium@427900 0 N/A 2 good chromium@427900,catapult@bd5139fa07 0 N/A 2 good chromium@427900,catapult@b1fcad56cd 1 N/A 2 bad <-- chromium@427900,catapult@eeeba06a98 1 N/A 2 bad chromium@427901 1 N/A 2 bad chromium@427902 1 N/A 2 bad chromium@427903 1 N/A 2 bad chromium@427905 1 N/A 2 bad chromium@427909 1 N/A 2 bad chromium@427918 1 N/A 2 bad chromium@427935 1 N/A 2 bad Bisect job ran on: linux_perf_bisect Bug ID: 660048 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests rasterize_and_record_micro.key_silk_cases Test Metric: pixels_rasterized/pixels_rasterized Relative Change: Zero to non-zero Score: 99.9 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6804 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997553806056924288 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5296895381995520 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Oct 28 2016
I'm going to kick off another bisect to confirm since this keeps on getting different results.
,
Oct 28 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997535302420391936
,
Oct 28 2016
,
Oct 28 2016
===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Fix RenderFrameHost's Mojo objects not being recreated after a process crash. Author : jam Commit description: This fixes flakiness in PolicyTest.ExtensionInstallBlacklistSharedModules with PlzNavigate. BUG= 504347 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_site_isolation Review-Url: https://codereview.chromium.org/2454833002 Cr-Commit-Position: refs/heads/master@{#427848} Commit : fc5ee5bae3a6c40e46a28a3e2410c2170745eb5d Date : Wed Oct 26 22:23:41 2016 ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@427847 0 N/A 5 good chromium@427848 1 N/A 5 bad <-- chromium@427849 1 N/A 5 bad chromium@427851 1 N/A 5 bad Bisect job ran on: linux_perf_bisect Bug ID: 660048 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests rasterize_and_record_micro.key_silk_cases Test Metric: pixels_rasterized/pixels_rasterized Relative Change: Zero to non-zero Score: 0.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6805 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997535302420391936 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5855824440721408 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Oct 28 2016
Starting another one since every one is getting a different one.
,
Oct 28 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997520292183478160
,
Oct 28 2016
===== BISECT JOB RESULTS ===== Status: completed ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@427840 0 N/A 5 good chromium@427855 0 N/A 5 bad Bisect job ran on: linux_perf_bisect Bug ID: 660048 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='list.animation.simple\.html' rasterize_and_record_micro.key_silk_cases Test Metric: pixels_rasterized/list_animation_simple.html Relative Change: 0.00% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6807 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997520292183478160 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5893437616816128 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 1 2016
Looks like this is a timeout that occurs on different test cases. Sometimes list-recycle-transform.html, sometimes list_animation_simple.html. First bot failure was in 427875:427920. I'll try looking for indication that the test started taking longer to complete...
,
Nov 1 2016
,
Nov 1 2016
Issue 660328 has been merged into this issue.
,
Nov 1 2016
Issue 656671 has been merged into this issue.
,
Nov 1 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997176356685694416
,
Nov 1 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997176309238926976
,
Nov 1 2016
I believe there's been flakiness here for awhile - seeing intermittent failures of this form back to Oct 9. There's also evidence of an increase in record time variability around Oct 4: https://chromeperf.appspot.com/report?sid=27d8972eb23851c88a12da741370ee1bea289d80de06958f78000dcf5ce14021&start_rev=412724&end_rev=427874 I've kicked off both a larger return code bisect (not sure if num_repeats will work there for flaky failures) as well as a stddev bisect for where the variability seems to go up. Worst case and there's no obvious culprit then I'll disable the test and assign this to the test owner (vmpstr) for follow-up.
,
Nov 1 2016
Oh, it's also been disabled since May on Mac, Windows and Android due to flakiness - issue 610424 . Perhaps nobody cares about the this test anymore? I'll disable on linux against that bug as well.
,
Nov 1 2016
,
Nov 2 2016
===== BISECT JOB RESULTS ===== Status: completed ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@422335 1 N/A 20 good chromium@427600 1 N/A 20 bad Bisect job ran on: linux_perf_bisect Bug ID: 660048 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests rasterize_and_record_micro.key_silk_cases Test Metric: record_time/record_time Relative Change: 0.00% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6811 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997176356685694416 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5883427201482752 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 2 2016
=== Auto-CCing suspected CL author bsalomon@google.com === Hi bsalomon@google.com, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Move clip CTM application to SkRasterClip and SkClipStack Author : Brian Salomon Commit description: GOLD_TRYBOT_URL= https://gold.skia.org/search?issue=2866 Change-Id: I914a57d6ba128acc457e12586c99ba6766eb940c Reviewed-on: https://skia-review.googlesource.com/2866 Reviewed-by: Mike Reed <reed@google.com> Reviewed-by: Florin Malita <fmalita@chromium.org> Commit : a3b45d4f7db953472df4f11ab1595964b65175f9 Date : Mon Oct 03 19:32:07 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@422347 0.0287103 0.00277566 5 good chromium@422550 0.0318621 0.00117004 4 good chromium@422601 0.0279448 0.00166406 5 good chromium@422626 0.0313655 0.00178456 5 good chromium@422626,skia@14f984bc6b 0.0321724 0.00176621 5 good chromium@422626,skia@a3b45d4f7d 0.0349241 0.000660775 5 bad <-- chromium@422626,skia@afb48b6227 0.0347862 0.000375688 5 bad chromium@422626,skia@f85d2a4fa1 0.0347724 0.000387529 5 bad chromium@422627 0.0350276 0.00049636 5 bad chromium@422628 0.035469 0.000411343 5 bad chromium@422630 0.0351897 0.000371924 4 bad chromium@422633 0.0354621 0.000388295 5 bad chromium@422639 0.0351293 0.000157706 4 bad chromium@422651 0.0354621 0.000710295 5 bad chromium@422752 0.0361586 0.000186653 5 bad Bisect job ran on: linux_perf_bisect Bug ID: 660048 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests rasterize_and_record_micro.key_silk_cases Test Metric: record_time/record_time Relative Change: 25.94% Score: 95.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6812 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997176309238926976 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5301149962338304 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 2 2016
Because of the flake, I'm re-opening this and assigning to Flakiness to see if there's a way we can improve our tools to make it easier for people to see if this was flake or not. We wasted a ton of time trying to re-run the builds, when we could've realized it was flake and disabled/fixed the test a week prior.
,
Nov 4 2016
Reassigning to sergiyb, who is working on test flakiness.
,
Nov 4 2016
,
Nov 7 2016
We do not collect data from waterfall and only detect flakiness in CQ. Therefore this build was not detected or reported as flaky by chromium-try-flakes. Adding support for waterfall depends on migrating chromium-try-flakes to event_mon, which is tracked in issue 596226. Unfortunately, I am busy working on other flakiness projects atm and therefore the work in that direction is currently stalled. The new Flakiness Surface project that I work on is already based on event_mon and thus will show up waterfall flakiness for tests, but not for steps as in this case.
,
Nov 8 2016
,
Nov 8 2016
|
|||||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Oct 27 2016