tools/perf PRESUBMIT always fail, blocking CQ |
||||||
Issue description
The error message is:
** Presubmit ERRORS **
Validating story expectation data failed.
***************
Running all tests in /b/swarming/w/ir/cache/builder/chromium_presubmit/src/third_party/blink/perf_tests
ERROR:root:Story worker-structured-clone-json-serialize.html is not in the story set.
Traceback (most recent call last):
<module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/validate_story_expectation_data:12
sys.exit(story_expectation_validator.main(sys.argv[1:]))
main at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/core/story_expectation_validator.py:80
validate_story_names(benchmarks, raw_expectations_data)
validate_story_names at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/core/story_expectation_validator.py:35
assert not failed_stories, 'Incorrect story names: %s' % str(failed_stories)
AssertionError: Incorrect story names: ['worker-structured-clone-json-serialize.html']
Locals:
b : <benchmarks.blink_perf.BlinkPerfBindings object at 0x7fc4fd714510>
benchmark : <class 'benchmarks.blink_perf.BlinkPerfBindings'>
benchmarks : [<class 'contrib.cluster_telemetry.analysis_metrics_ct.AnalysisMetricsCT'>, <class 'contrib.blink_perf_cmdline.blink_perf_cmdline.BlinkPerfAll'>, <class 'benchmarks.blink_perf.BlinkPerfAccessibility'>, <class 'benchmarks.blink_perf.BlinkPerfBindings'>, <class 'benchmarks.blink_perf.BlinkPerfCanvas'>, <class 'benchmarks.blink_perf.BlinkPerfCSS'>, <class 'benchmarks.blink_perf.BlinkPerfDOM'>, <class 'benchmarks.blink_perf.BlinkPerfEvents'>, <class 'benchmarks.blink_perf.BlinkPerfImageDecoder'>, <class 'ben ... .Wasm'>, <class 'benchmarks.webrtc.WebrtcPerfBenchmark'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrBrowsingStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrBrowsingWprSmoothness'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrBrowsingWprStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebVrLiveStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebVrStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebVrWprStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebXrStatic'>] (truncated)
failed_stories : ['worker-structured-clone-json-serialize.html']
raw_expectations_data : '# Test Expectation file for telemetry tests.\n# Instructions of how to use this file:\n# https://chromium.googlesource.com/chromium/src/+/master/docs/speed/bot_health_sheriffing/how_to_disable_a_story.md\n\n# tags: All Android_Go Android_One Android_Svelte Android_Low_End\n# tags: Android_Webview Android_but_not_webview Mac Win Linux\n# tags: ChromeOS Android Desktop Nexus_5 Nexus_5X Nexus_6P\n# tags: Nexus_7 Mac_10.11 Mac_10.12 Nexus6_Webview Nexus5X_Webview\n# tags: Pixel_2 Win_7 Win_10\n\n# Benchmark ... Linux ] loading.desktop.network_service/Taobao_cold [ Skip ]\ncrbug.com/853835 [ Linux ] loading.desktop.network_service/Taobao_warm [ Skip ]\ncrbug.com/879833 [ Linux ] loading.desktop.network_service/Walgreens_cold [ Skip ]\ncrbug.com/879833 [ Linux ] loading.desktop.network_service/Walgreens_warm [ Skip ]\n\n# Benchmark: loading.desktop_layout_ng\ncrbug.com/879833 [ Linux ] loading.desktop_layout_ng/Walgreens_cold [ Skip ]\ncrbug.com/879833 [ Linux ] loading.desktop_layout_ng/Walgreens_warm [ Skip ]\n' (truncated)
story_set : <telemetry.story.story_set.StorySet object at 0x7fc4fd59eb10>
***************
Presubmit checks took 25.8s to calculate.
step returned non-zero exit code: 1
,
Oct 10
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6e911bb529976d3bd030cd98da922ff2fa9635c3 commit 6e911bb529976d3bd030cd98da922ff2fa9635c3 Author: Ned Nguyen <nednguyen@google.com> Date: Wed Oct 10 02:17:03 2018 Remove expecation of removed story blink_perf.bindings/worker-structured-clone-json-serialize.html TBR=crouleau@chromium.org NOTRY=true Bug: 893936 Change-Id: Ib2109cdf8b72ecb33f87a771f9a8c2cae8d694c1 Reviewed-on: https://chromium-review.googlesource.com/c/1272806 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#598180} [modify] https://crrev.com/6e911bb529976d3bd030cd98da922ff2fa9635c3/tools/perf/expectations.config
,
Oct 10
Downgrade priority since the fix is land. At next step, we should make sure tools/perf/PRESUBMIT.py is also executed for changes in third_party/blink/perf_tests/... Dirk/John: you know is there any way to do so?
,
Oct 10
Issue 894152 has been merged into this issue.
,
Oct 17
There's no built in way to do so, but you could write a custom presubmit hook to do that.
,
Oct 17
#5: thanks Dirk. is there a commandline to run a PRESUBMIT script?
,
Oct 17
,
Oct 17
not easily, no :(. You might be best off refactoring things to push as much logic as you can in a script that is called from both places.
,
Jan 16
(6 days ago)
,
Jan 16
(6 days ago)
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by nedngu...@google.com
, Oct 10