Refer to "timeout" field in external/wpt/MANIFEST.json when deciding whether a test is "slow" |
|||||||
Issue description
What steps will reproduce the problem?
(1) Import slow tests from web-platform-tests
(2) Observe webkit_tests build bots
(3)
What is the expected result?
No flaky timeout
What happens instead?
Updated tests might have flaky timeout.
Test owners or sheriffs need to add flaky entries to TestExepctations, or add entries to SlowTests.
Please use labels and text to provide additional information.
MANIFEST.json contains slowness information like:
"dom/ranges/Range-attributes.html": [
[
"/dom/ranges/Range-attributes.html",
{
"timeout": "long"
}
]
],
run-webkit-tests should refer to the "timeout" fields, should set longer timeout for the test.
,
Jan 23 2017
Not sure if I'll have time this week, but I agree that this sounds like a good idea. Note that we ignore the testharness.js timeout ( bug 664268 ), but reading MANIFEST.json to decide how to modify the run-webkit-tests timeout would work. Fortunately, the testharness timeout mechanism also supports two options for how to set the timeout for a test: "normal" and "long" (slow), according to http://testthewebforward.org/docs/testharness-library.html#harness-timeout. It looks like the place to change would be Manager._test_is_slow (https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py), which could access the Port object, which could read the manifest file and check the value for a particular test.
,
Jan 23 2017
,
Jan 25 2017
,
Jan 26 2017
,
Jan 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c96e02d34a39e6a3e790c563cb506b73aa134e2b commit c96e02d34a39e6a3e790c563cb506b73aa134e2b Author: tkent <tkent@chromium.org> Date: Thu Jan 26 00:53:36 2017 run-webkit-tests: Extract slowness information from MANIFEST.json. BUG= 683807 Review-Url: https://codereview.chromium.org/2656813002 Cr-Commit-Position: refs/heads/master@{#446178} [modify] https://crrev.com/c96e02d34a39e6a3e790c563cb506b73aa134e2b/third_party/WebKit/LayoutTests/SlowTests [modify] https://crrev.com/c96e02d34a39e6a3e790c563cb506b73aa134e2b/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py [modify] https://crrev.com/c96e02d34a39e6a3e790c563cb506b73aa134e2b/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py [modify] https://crrev.com/c96e02d34a39e6a3e790c563cb506b73aa134e2b/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py
,
Jan 26 2017
,
Jul 3 2017
,
Jul 3 2017
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by jeffcarp@chromium.org
, Jan 23 2017Components: -Blink>Infra Blink>Infra>Predictability
Labels: -Pri-3 Pri-2
Status: Available (was: Untriaged)