experiment with 10x testing for layout tests |
||
Issue descriptionTest flakiness is being masked by retries at different layers. There is a proposal to instead require tests to pass multiple times, preventing flakiness from ever getting committed in the repo: https://docs.google.com/document/d/1CR0pKaC1bZ8Mk2-tbmrKJlqMH8UNwvATlUJ-56qVrjI/edit#heading=h.fdobmxs62fu Thoughts on prototyping this idea: * Add a src/third_party/WebKit/LayoutTests/10xTestExpectations of all existing flaky layout tests * Add a 10x mode to layout tests runner. Blacklisted tests won't be run but existing good tests and all new tests will * Add an experimental builder to tryserver.chromium.linux (or tryserver.chromium.fyi if the split happens first) that runs layout tests, passing in a property to run in 10x mode
,
Feb 21 2017
,
Jul 11 2017
See also 697215, filed for removing the retries from the standard bot test runs, except for explicitly-annotated tests. FWIW I've spent a few days debugging issue 740692 , which boiled down to raciness in some threaded tests - I'm not sure that running them 10x would necessarily spot the issues, which only showed up for me due to platform thread scheduling differences, though. |
||
►
Sign in to add a comment |
||
Comment 1 by kbr@chromium.org
, Feb 21 2017