Update telemetry logic to not fail when running disabled tests |
||
Issue description
In order to get perf on swarming we need to be able to statically generate the list of benchmarks to run. Therefore, needed information to determine if a benchmark is disabled or not can not be determined when we generate this list since it is dependent on the bot it is running on.
Therefore, we have decided to update the telemetry logic to not return 1 on disabled benchmarks but instead return 0 (success) and just not execute the test.
The steps to achieve this are as follows:
1) Update story_runner to generate chartjson before returning 0 on a disabled benchmarks. The format of this json will be {benchmark_name: name1, disabled: True}
2) Update all recipes to read this new chart json and not send results to the dashboard when test is disabled
3) Update all telemetry logic to continue executing (in dual_browser_story and
shared_page_state) when a test is disabled
4) Have the StoryRunner to return 0 instead of 1 (original CL
crrev.com/2342023005)
,
Sep 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e83042ee5983abc9d0e2ae34134b5cd213bd618f commit e83042ee5983abc9d0e2ae34134b5cd213bd618f Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Mon Sep 26 22:43:28 2016 Roll src/third_party/catapult/ efbf303a5..28c78c375 (6 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/efbf303a5360..28c78c375477 $ git log efbf303a5..28c78c375 --date=short --no-merges --format='%ad %ae %s' 2016-09-26 eyaich Outputting chart json for disabled tests. 2016-09-26 alexandermont Add load and after_load time to power metrics. 2016-09-26 benjhayden Improve some docs, comments, and tests. 2016-09-26 benjhayden Remove font-size:12px from tr-ui-b-table. 2016-09-26 petrcermak [tracing] Use 'count' unit in memory-infra code 2016-09-26 petrcermak [tracing] Enable setting context group in MemoryColumn.fromRows BUG= 647714 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2366413003 Cr-Commit-Position: refs/heads/master@{#421026} [modify] https://crrev.com/e83042ee5983abc9d0e2ae34134b5cd213bd618f/DEPS
,
Sep 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/b7b68b4c0396453d775c7fb924d45bbfdff6f349 commit b7b68b4c0396453d775c7fb924d45bbfdff6f349 Author: eyaich <eyaich@chromium.org> Date: Tue Sep 27 13:16:37 2016 Updating perf recipe for disabled chartjson data. BUG= chromium:647714 Review-Url: https://codereview.chromium.org/2346963003 [modify] https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349/scripts/slave/runtest.py [modify] https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349/scripts/slave/unittests/results_dashboard_test.py [modify] https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349/scripts/slave/unittests/runtest_test.py
,
Sep 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/b7b68b4c0396453d775c7fb924d45bbfdff6f349 commit b7b68b4c0396453d775c7fb924d45bbfdff6f349 Author: eyaich <eyaich@chromium.org> Date: Tue Sep 27 13:16:37 2016 Updating perf recipe for disabled chartjson data. BUG= chromium:647714 Review-Url: https://codereview.chromium.org/2346963003 [modify] https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349/scripts/slave/runtest.py [modify] https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349/scripts/slave/unittests/results_dashboard_test.py [modify] https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349/scripts/slave/unittests/runtest_test.py
,
Sep 27 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/5f815c0ac400d2fc33283e7705a54af5d0e0d43c commit 5f815c0ac400d2fc33283e7705a54af5d0e0d43c Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Sep 27 13:24:32 2016
,
Sep 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dbf9fd7abb53f906b59e4589dcbbdf1ee190f15b commit dbf9fd7abb53f906b59e4589dcbbdf1ee190f15b Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Sep 27 13:35:27 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349 Updating perf recipe for disabled chartjson data. (eyaich@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= chromium:647714 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2373793002 Cr-Commit-Position: refs/heads/master@{#421192} [modify] https://crrev.com/dbf9fd7abb53f906b59e4589dcbbdf1ee190f15b/infra/config/recipes.cfg
,
Sep 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/5d8747c4ca7e370ee284af76123fa6f5ac6ca520 commit 5d8747c4ca7e370ee284af76123fa6f5ac6ca520 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Sep 27 13:43:04 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/b7b68b4c0396453d775c7fb924d45bbfdff6f349 Updating perf recipe for disabled chartjson data. (eyaich@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= chromium:647714 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2374713002 [modify] https://crrev.com/5d8747c4ca7e370ee284af76123fa6f5ac6ca520/infra/config/recipes.cfg
,
Sep 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1e092ccae355f4430ad1a1707fa83b5ab1fe7a17 commit 1e092ccae355f4430ad1a1707fa83b5ab1fe7a17 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Fri Sep 30 16:30:40 2016 Roll src/third_party/catapult/ f5592efd0..addb27e71 (3 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/f5592efd0cf7..addb27e71cc9 $ git log f5592efd0..addb27e71 --date=short --no-merges --format='%ad %ae %s' 2016-09-30 eyaich Updating telemetry logic to return success when a disabled test is run. 2016-09-30 hjd [tracing] Add keyboard navigation to the tab view 2016-09-30 hjd [tracing] Correct usage of tr.b.Event BUG= 647714 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2376103006 Cr-Commit-Position: refs/heads/master@{#422129} [modify] https://crrev.com/1e092ccae355f4430ad1a1707fa83b5ab1fe7a17/DEPS
,
Oct 4 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/89b1e1527284754aa13e7a9fc5e7731105bc4886 commit 89b1e1527284754aa13e7a9fc5e7731105bc4886 Author: eyaich <eyaich@chromium.org> Date: Tue Oct 04 00:34:18 2016 Ignoring all results but chartjson on disabled benchmark run. Currently we are writing out only chartjson results when a disabled test is run but we still check for json results. BUG= chromium:647714 Review-Url: https://codereview.chromium.org/2385183002 Cr-Commit-Position: refs/heads/master@{#422630} [modify] https://crrev.com/89b1e1527284754aa13e7a9fc5e7731105bc4886/testing/scripts/run_telemetry_benchmark_as_googletest.py
,
Oct 10 2016
|
||
►
Sign in to add a comment |
||
Comment 1 by bugdroid1@chromium.org
, Sep 26 2016