blink_perf.paint failing on chromium.perf/Android Nexus7v2 Perf and Nexus5 Perf |
||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of charliea@chromium.org blink_perf.paint failing on chromium.perf/Android Nexus7v2 Perf Builders failed on: - Android Nexus5 Perf - Android Nexus7v2 Perf: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf It seems like these failures have been happening for a *while* now: here's a link to the flakiness dashboard for Nexus5: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=blink_perf.paint&showAllRuns=true&sortOrder=forward&builder=chromium.perf%3AAndroid%20Nexus5%20Perf And here's the one for Nexus 7v2: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=blink_perf.paint&showAllRuns=true&sortOrder=forward&builder=chromium.perf%3AAndroid%20Nexus7v2%20Perf An excerpt from a failure on Nexus 5 is available at https://gist.github.com/zeptonaut/225f6fbb53526d7a93ee0fa971505a82. It looks like we're timing out waiting for some javascript expression to evaluate to true. I'm not sure if the problems on these two platforms are related. It seems like a lot of the Nexus 7v2 errors are related to devices that become unreachable. I want to blame this on a flaky device, but none of the other benchmarks running on that device seem to be experiencing similar errors, just blink_perf.paint. Here are two example failures on that platform: https://gist.github.com/zeptonaut/3871074b5af7e247a7b7f98addfcfc7e (Chrome crash) https://gist.github.com/zeptonaut/5e1c05b8e82548d50515ba73fba3c94c (Suspected Chrome crash, but it looks like the device became unreachable - possibly due to the crash?) Anyhow, I'm going to kick off a bisect after filing this bug that runs at a high iteration count over the suspected N5 regression range (something like 519695 as a first good version and 520167 as a first bad version). I'll kick this down to P1 and reassign to the owners after I disable on these two platforms and launch a bisect on N5. nednguyen@, is it a common pattern that a Chrome crash results in a DeviceUnreachableError? Given that other benchmarks running on this device seem to run fine, I'm hesitant to blame the story failure on device flakiness rather than the other way around.
,
Jan 5 2018
Kicking down to P2 and reassigning to the owner now that a bisect is started and disable is in flight.
,
Jan 5 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2fff258f1511d035d9f56efe4e27ec0fd8544d5c commit 2fff258f1511d035d9f56efe4e27ec0fd8544d5c Author: Charlie Andrews <charliea@chromium.org> Date: Fri Jan 05 19:44:34 2018 Disable blink_perf.paint on Nexus5 and Nexus7v2 The benchmark has been flaky for a while now. NOTRY=true TBR=nednguyen@google.com,wangxianzhu@chomium.org Bug: 799540 Change-Id: Ib720dc8f0e6be12069b4453aae46d34f9c5fb8e8 Reviewed-on: https://chromium-review.googlesource.com/852802 Reviewed-by: Charlie Andrews <charliea@chromium.org> Commit-Queue: Charlie Andrews <charliea@chromium.org> Cr-Commit-Position: refs/heads/master@{#527346} [modify] https://crrev.com/2fff258f1511d035d9f56efe4e27ec0fd8544d5c/tools/perf/expectations.config
,
Jan 6 2018
=== BISECT JOB RESULTS === Test failure found but unable to narrow commit range Build failures prevented the bisect from narrowing the range further. Bisect Details Configuration: android_nexus5_perf_bisect Benchmark : blink_perf.paint Metric : large-table-repaint/large-table-repaint.html Suspected Commit Range 2 commits in range https://chromium.googlesource.com/chromium/src/+log/4f59a790e19f97e8bbdbb6e7960883c1e1cf0234..c47b4d110a697afa6a6ee4d4748028312c65ec19 Revision Exit Code N chromium@519695 0 +- N/A 20 good chromium@519930 0 +- N/A 20 good chromium@520048 0 +- N/A 20 good chromium@520056 0 +- N/A 20 good chromium@520060 0 +- N/A 20 good chromium@520061 --- --- build failure chromium@520062 1 +- N/A 20 bad chromium@520063 1 +- N/A 20 bad chromium@520078 1 +- N/A 20 bad chromium@520108 1 +- N/A 20 bad chromium@520167 1 +- N/A 20 bad Please refer to the following doc on diagnosing blink_perf regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/speed/benchmark_harnesses/blink_perf.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=large.table.repaint.html blink_perf.paint More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8958212045262080816 For feedback, file a bug with component Speed>Bisection
,
Jan 9 2018
I think though the tests are a bit flaky, when they succeed, the results are still useful. Disabling the tests just remove all the useful results. How about running the tests while investigating the flakiness?
,
Jan 9 2018
Nexus7v2 and Nexus5 may be the slowest platforms. The loads of the tests are very high, and the frame times are sometime a few seconds. The high load and long frame times may cause the flakiness. I think it's fine to disable the tests on these two platforms. We still have other Android bots running the tests providing useful data. |
||||
►
Sign in to add a comment |
||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Jan 5 2018