Adding Telemetry benchmarks to the flakiness dashboard |
|||||||
Issue descriptionBug to track the work on uploading Telemetry benchmarks running on the perf waterfall to the flakiness dashboard. Design doc: go/telemetry-flakiness-and-failures Adding the Telemetry benchmarks to the flakiness dashboard will allow us to identify: * tests that are flaky on the buildbots * when tests started failing * when tests are failing for a different reason than they previously were
,
May 9 2017
,
Jun 23 2017
,
Jul 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c2e37410c66a143b428e1fe6ef3f19fc03c5509e commit c2e37410c66a143b428e1fe6ef3f19fc03c5509e Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Wed Jul 12 01:37:42 2017 Roll src/third_party/catapult/ fc9479ad5..08d8c9f08 (10 commits) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/fc9479ad5e3b..08d8c9f08602 $ git log fc9479ad5..08d8c9f08 --date=short --no-merges --format='%ad %ae %s' 2017-07-11 eakuefner [Dashboard] Include common/py_utils in dashboard deployment 2017-07-11 loloangela Removed abstract-class-little-used and abstract-class-not-used; Fixed errors related to bad-continuation 2017-07-11 etienneb Support for breakpad symbolisation from a symbols folder 2017-07-11 eakuefner [Tracing] use discover instead of REGISTRY to deserialize Python diagnostics 2017-07-11 benjhayden Split HistogramGrouping to its own file. 2017-07-11 etienneb Fix incorrect path with symbolisation script 2017-07-11 ashleymarie Add json3 output formatter 2017-07-11 simonhatch Dashboard - Rename OauthTest -> ApiAuthTest 2017-07-11 sullivan Cache whether an identity is a group member. 2017-07-11 etienneb Implements breakpad symbolisation Created with: roll-dep src/third_party/catapult BUG= 739783 , 717394 , 739783 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: Ia997fc6f7d00dc9c939c0eee4b0f9fde9d0710f4 Reviewed-on: https://chromium-review.googlesource.com/567640 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#485776} [modify] https://crrev.com/c2e37410c66a143b428e1fe6ef3f19fc03c5509e/DEPS
,
Jul 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1b0335dbe961d4ca985cb845c5bf7af37b24e7b2 commit 1b0335dbe961d4ca985cb845c5bf7af37b24e7b2 Author: ashleymarie <ashleymarie@chromium.org> Date: Wed Jul 12 16:14:01 2017 Outputing json test results to output.json BUG= chromium:717394 Review-Url: https://codereview.chromium.org/2979703002 Cr-Commit-Position: refs/heads/master@{#485984} [modify] https://crrev.com/1b0335dbe961d4ca985cb845c5bf7af37b24e7b2/testing/scripts/run_telemetry_benchmark_as_googletest.py
,
Jul 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a26d2ff2b4bf8e55e7219c7ec80014fcf2c6c534 commit a26d2ff2b4bf8e55e7219c7ec80014fcf2c6c534 Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Mon Jul 17 17:47:10 2017 Roll src/third_party/catapult/ af47a9399..6e3bfad0d (1 commit) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/af47a939971b..6e3bfad0debc $ git log af47a9399..6e3bfad0d --date=short --no-merges --format='%ad %ae %s' 2017-07-17 ashleymarie Fixing a line break Created with: roll-dep src/third_party/catapult BUG= 717394 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: I6002720bd981d8cd871563762578a3eab5e3ddc3 Reviewed-on: https://chromium-review.googlesource.com/574747 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#487148} [modify] https://crrev.com/a26d2ff2b4bf8e55e7219c7ec80014fcf2c6c534/DEPS
,
Jul 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/88a57b64c852326ba2a5d84c395882c01692ea32 commit 88a57b64c852326ba2a5d84c395882c01692ea32 Author: Ashley Enstad <ashleymarie@chromium.org> Date: Tue Jul 18 00:13:36 2017 Upload the system_health benchmarks to the flakiness dashboard. This change enables perf_data_generator to upload results from a list of benchmarks to the flakiness dashboard. BUG= chromium:717394 Change-Id: I3952887c527c40fb5d061e9ee8aa4875794075c7 Reviewed-on: https://chromium-review.googlesource.com/574687 Commit-Queue: Ashley Enstad <ashleymarie@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#487310} [modify] https://crrev.com/88a57b64c852326ba2a5d84c395882c01692ea32/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/88a57b64c852326ba2a5d84c395882c01692ea32/tools/perf/core/perf_data_generator.py [modify] https://crrev.com/88a57b64c852326ba2a5d84c395882c01692ea32/tools/perf/core/perf_data_generator_unittest.py
,
Jul 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5ce3442a9ad9bff805f89ab6ebeadbf7bffc0429 commit 5ce3442a9ad9bff805f89ab6ebeadbf7bffc0429 Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Wed Jul 19 16:39:20 2017 Roll src/third_party/catapult/ 8641322e4..1195fa0a2 (1 commit) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/8641322e4a40..1195fa0a2590 $ git log 8641322e4..1195fa0a2 --date=short --no-merges --format='%ad %ae %s' 2017-07-19 ashleymarie Changing version from "3" to 3 in json test results format Created with: roll-dep src/third_party/catapult BUG= 717394 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: I37876d87e72f59c4e0eff02dbf6291bca41e6138 Reviewed-on: https://chromium-review.googlesource.com/577727 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#487887} [modify] https://crrev.com/5ce3442a9ad9bff805f89ab6ebeadbf7bffc0429/DEPS
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/95170321e54b208637ca581fd848560975a6711a commit 95170321e54b208637ca581fd848560975a6711a Author: Ashley Enstad <ashleymarie@chromium.org> Date: Thu Jul 20 18:41:53 2017 Adding chromium.perf and chromium.perf.fyi to masters list We'd like to upload test results to the flakiness dashboard for Telemetry benchmarks which are run on chromium.perf and chromium.perf.fyi The error at https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf.fyi%2FWin_10_4_Core_Low-End_Perf_Tests%2F1373%2F%2B%2Frecipes%2Fsteps%2FUpload_to_test-results__system_health.common_desktop_on_Intel_GPU_on_Windows_%2F0%2Fstdout says that those masters aren't whitelisted here. BUG= chromium:717394 Change-Id: I98438b59bfc4155d2c6fcb9df5df5e80781ac386 Reviewed-on: https://chromium-review.googlesource.com/579579 Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org> Commit-Queue: Ashley Enstad <ashleymarie@chromium.org> [modify] https://crrev.com/95170321e54b208637ca581fd848560975a6711a/go/src/infra/appengine/test-results/masters/masters.go
,
Jul 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2c2a34b1d1c9dcb7c695419f66e1d7dc554c7a3f commit 2c2a34b1d1c9dcb7c695419f66e1d7dc554c7a3f Author: Ashley Enstad <ashleymarie@chromium.org> Date: Mon Jul 24 16:12:45 2017 Upload the system_health benchmarks to the flakiness dashboard The previous cl https://chromium-review.googlesource.com/c/574687/ handled uploading specifically for fyi bots only. This one enables uploading for both chromium.perf.fyi and chromium.perf BUG= chromium:717394 Change-Id: Ifffa6fa75a45633d20a92410f8a3246ba213f0c7 Reviewed-on: https://chromium-review.googlesource.com/581414 Commit-Queue: Ashley Enstad <ashleymarie@chromium.org> Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#488987} [modify] https://crrev.com/2c2a34b1d1c9dcb7c695419f66e1d7dc554c7a3f/testing/buildbot/chromium.perf.json [modify] https://crrev.com/2c2a34b1d1c9dcb7c695419f66e1d7dc554c7a3f/tools/perf/core/perf_data_generator.py [modify] https://crrev.com/2c2a34b1d1c9dcb7c695419f66e1d7dc554c7a3f/tools/perf/core/perf_data_generator_unittest.py
,
Jul 25 2017
,
Jul 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/88b7285575ddbe56cc1f35f2408d6cdffbad7519 commit 88b7285575ddbe56cc1f35f2408d6cdffbad7519 Author: Ashley Enstad <ashleymarie@chromium.org> Date: Thu Jul 27 14:08:43 2017 Only upload test results for specific benchmarks BUG=chromium:748638 BUG= chromium:717394 NOTRY=true Change-Id: I881e4e33d1803837ed373b863ae6a0c0d33762c0 Reviewed-on: https://chromium-review.googlesource.com/587092 Commit-Queue: Ashley Enstad <ashleymarie@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#489913} [modify] https://crrev.com/88b7285575ddbe56cc1f35f2408d6cdffbad7519/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/88b7285575ddbe56cc1f35f2408d6cdffbad7519/testing/buildbot/chromium.perf.json [modify] https://crrev.com/88b7285575ddbe56cc1f35f2408d6cdffbad7519/tools/perf/core/perf_data_generator.py [modify] https://crrev.com/88b7285575ddbe56cc1f35f2408d6cdffbad7519/tools/perf/core/perf_data_generator_unittest.py
,
Aug 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/88b8d31f2f4384243206306bbb9cc7393a49a871 commit 88b8d31f2f4384243206306bbb9cc7393a49a871 Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Wed Aug 09 20:24:34 2017 Roll src/third_party/catapult/ 868342d85..656df4e2f (6 commits) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/868342d8543e..656df4e2fcc8 $ git log 868342d85..656df4e2f --date=short --no-merges --format='%ad %ae %s' 2017-08-09 benjhayden [results.html] Stop creating history entries for now. 2017-08-09 benjhayden results.html: display sets of strings as text instead of JSON. 2017-08-09 ashleymarie Adding optional param is_unexpected to json_3_output_formatter 2017-08-09 phsilva Plumb SparseDiagnostics by Name 2017-08-09 xunjieli [wpr-go] Add support install test root CA on Android 2017-08-09 benjhayden Add paper-tabs to third_party/polymer/components. Created with: roll-dep src/third_party/catapult BUG= 717394 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: Id62600933fbdd31250d8a28eebac055cbce8f168 Reviewed-on: https://chromium-review.googlesource.com/608755 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#493123} [modify] https://crrev.com/88b8d31f2f4384243206306bbb9cc7393a49a871/DEPS
,
Sep 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e44b98077549c5901abeeb22d372abdb7ba4dbda commit e44b98077549c5901abeeb22d372abdb7ba4dbda Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Sep 13 01:50:09 2017 //tools/perf: Upload more benchmarks to the flakiness dashboard Bug: 717394 Change-Id: I4f3851f290d996746f209760f00ea8668e6091c8 Reviewed-on: https://chromium-review.googlesource.com/639577 Reviewed-by: Ned Nguyen <nednguyen@google.com> Reviewed-by: Sean McCullough <seanmccullough@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#501508} [modify] https://crrev.com/e44b98077549c5901abeeb22d372abdb7ba4dbda/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/e44b98077549c5901abeeb22d372abdb7ba4dbda/testing/buildbot/chromium.perf.json [modify] https://crrev.com/e44b98077549c5901abeeb22d372abdb7ba4dbda/tools/perf/core/perf_data_generator.py
,
Sep 25 2017
I've just noted that both SKIP (for disabled stories) and FAIL (actual story failures) show up as black in the test results dashboard, e.g.: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=system_health.memory_mobile%2Flong_running%3Atools%3Agmail-background&testType=system_health.memory_mobile on Nexus5X the black squares are failures, but on "Nexus5X WebView" the story is disabled. Would it be possible to make it easier to distinguish those?
,
Sep 25 2017
Sean: is it intentional that "both SKIP (for disabled stories) and FAIL (actual story failures) show up as black in the test results dashboard"? +Ashley in case it's due to how we output json test results.
,
Sep 26 2017
Since we skip the tests three times, we upload SKIP SKIP SKIP as the actual results. https://cs.chromium.org/chromium/infra/go/src/infra/appengine/test-results/frontend/static/dashboards/js/flakiness_dashboard.js?l=631&rcl=cab065eb16561e93ef7c956a21d6db735e5ffc4e logic dictates that any test with more than 2 actual results is "VERYFLAKY" and thus shows up black (failures are also ran multiple times so they show up the same way as "VERYFLAKY" tests). If we want to distinguish between those on the flakiness dashboard, we can treat SKIP the way we treat PASS and if all three results are the same, only upload one SKIP or PASS. We could make the change on the flakiness dashboard side but there would be a delay on that since the team is still getting up to speed on that codebase
,
Sep 26 2017
the bug that tracked this for PASS PASS PASS is here: https://bugs.chromium.org/p/chromium/issues/detail?id=754825 So I'll make the same temporary fix for SKIP SKIP SKIP and update that bug accordingly
,
Apr 12 2018
I'm pretty sure this functionality is complete Feel free to reopen if you disagree :)
,
Jan 16
,
Jan 16
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by nedngu...@google.com
, May 2 2017