New issue
Advanced search Search tips

Issue 893936 link

Starred by 2 users

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

tools/perf PRESUBMIT always fail, blocking CQ

Project Member Reported by nedngu...@google.com, Oct 10

Issue description

The error message is:
  ** Presubmit ERRORS **
  Validating story expectation data failed.
  ***************
  Running all tests in /b/swarming/w/ir/cache/builder/chromium_presubmit/src/third_party/blink/perf_tests
  ERROR:root:Story worker-structured-clone-json-serialize.html is not in the story set.
  Traceback (most recent call last):
    <module> at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/validate_story_expectation_data:12
      sys.exit(story_expectation_validator.main(sys.argv[1:]))
    main at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/core/story_expectation_validator.py:80
      validate_story_names(benchmarks, raw_expectations_data)
    validate_story_names at /b/swarming/w/ir/cache/builder/chromium_presubmit/src/tools/perf/core/story_expectation_validator.py:35
      assert not failed_stories, 'Incorrect story names: %s' % str(failed_stories)
  AssertionError: Incorrect story names: ['worker-structured-clone-json-serialize.html']
  Locals:
    b                     : <benchmarks.blink_perf.BlinkPerfBindings object at 0x7fc4fd714510>
    benchmark             : <class 'benchmarks.blink_perf.BlinkPerfBindings'>
    benchmarks            : [<class 'contrib.cluster_telemetry.analysis_metrics_ct.AnalysisMetricsCT'>, <class 'contrib.blink_perf_cmdline.blink_perf_cmdline.BlinkPerfAll'>, <class 'benchmarks.blink_perf.BlinkPerfAccessibility'>, <class 'benchmarks.blink_perf.BlinkPerfBindings'>, <class 'benchmarks.blink_perf.BlinkPerfCanvas'>, <class 'benchmarks.blink_perf.BlinkPerfCSS'>, <class 'benchmarks.blink_perf.BlinkPerfDOM'>, <class 'benchmarks.blink_perf.BlinkPerfEvents'>, <class 'benchmarks.blink_perf.BlinkPerfImageDecoder'>, <class 'ben ... .Wasm'>, <class 'benchmarks.webrtc.WebrtcPerfBenchmark'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrBrowsingStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrBrowsingWprSmoothness'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrBrowsingWprStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebVrLiveStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebVrStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebVrWprStatic'>, <class 'contrib.vr_benchmarks.vr_benchmarks.XrWebXrStatic'>] (truncated)
    failed_stories        : ['worker-structured-clone-json-serialize.html']
    raw_expectations_data : '# Test Expectation file for telemetry tests.\n# Instructions of how to use this file:\n# https://chromium.googlesource.com/chromium/src/+/master/docs/speed/bot_health_sheriffing/how_to_disable_a_story.md\n\n# tags: All Android_Go Android_One Android_Svelte Android_Low_End\n# tags: Android_Webview Android_but_not_webview Mac Win Linux\n# tags: ChromeOS Android Desktop Nexus_5 Nexus_5X Nexus_6P\n# tags: Nexus_7 Mac_10.11 Mac_10.12 Nexus6_Webview Nexus5X_Webview\n# tags: Pixel_2 Win_7 Win_10\n\n# Benchmark ... Linux ] loading.desktop.network_service/Taobao_cold [ Skip ]\ncrbug.com/853835 [ Linux ] loading.desktop.network_service/Taobao_warm [ Skip ]\ncrbug.com/879833 [ Linux ] loading.desktop.network_service/Walgreens_cold [ Skip ]\ncrbug.com/879833 [ Linux ] loading.desktop.network_service/Walgreens_warm [ Skip ]\n\n# Benchmark: loading.desktop_layout_ng\ncrbug.com/879833 [ Linux ] loading.desktop_layout_ng/Walgreens_cold [ Skip ]\ncrbug.com/879833 [ Linux ] loading.desktop_layout_ng/Walgreens_warm [ Skip ]\n' (truncated)
    story_set             : <telemetry.story.story_set.StorySet object at 0x7fc4fd59eb10>
  ***************
  Presubmit checks took 25.8s to calculate.
  step returned non-zero exit code: 1
 
This is due to https://chromium-review.googlesource.com/c/chromium/src/+/1265085 which split worker-structured-clone-json-serialize to worker-structured-clone-json-to-worker.html & worker-structured-clone-json-from-worker.html
Project Member

Comment 2 by bugdroid1@chromium.org, Oct 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6e911bb529976d3bd030cd98da922ff2fa9635c3

commit 6e911bb529976d3bd030cd98da922ff2fa9635c3
Author: Ned Nguyen <nednguyen@google.com>
Date: Wed Oct 10 02:17:03 2018

Remove expecation of removed story blink_perf.bindings/worker-structured-clone-json-serialize.html

TBR=crouleau@chromium.org
NOTRY=true

Bug: 893936
Change-Id: Ib2109cdf8b72ecb33f87a771f9a8c2cae8d694c1
Reviewed-on: https://chromium-review.googlesource.com/c/1272806
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#598180}
[modify] https://crrev.com/6e911bb529976d3bd030cd98da922ff2fa9635c3/tools/perf/expectations.config

Cc: dpranke@chromium.org jbudorick@chromium.org
Labels: -Pri-0 Pri-2
Downgrade priority since the fix is land.

At next step, we should make sure tools/perf/PRESUBMIT.py is also executed for changes in 
third_party/blink/perf_tests/...

Dirk/John: you know is there any way to do so?
Cc: nednguyen@chromium.org pdr@chromium.org vmp...@chromium.org
 Issue 894152  has been merged into this issue.
There's no built in way to do so, but you could write a custom presubmit hook to do that.
#5: thanks Dirk. is there a commandline to run a PRESUBMIT script?
Cc: -nednguyen@chromium.org
not easily, no :(. You might be best off refactoring things to push as much logic as you can in a script that is called from both places.

Comment 9 by benhenry@google.com, Jan 16 (6 days ago)

Components: Test>Telemetry

Comment 10 by benhenry@google.com, Jan 16 (6 days ago)

Components: -Speed>Telemetry

Sign in to add a comment