New issue
Advanced search Search tips

Issue 833112 link

Starred by 1 user
Status: Available
Owner: ----
Cc:
a...@chromium.org
qyears...@chromium.org
robertma@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature
WPT



Sign in to add a comment

WPT importer is not aggressive enough in deciding which platforms to mark as flaky/fail

Project Member Reported by a...@chromium.org, Apr 15 2018 
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.11%20Tests/25392 is burning. Failure on the external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-flex-basis-content-003a.html test imported.

Apparently the machinery does some testing, but in https://chromium-review.googlesource.com/c/chromium/src/+/1013246 it concluded it should do

crbug.com/626703 [ Linux Mac10.10 Mac10.12 Mac10.13 Retina Win ] external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/flexbox-flex-basis-content-003a.html [ Failure ]

It fails on 10.11 too.

This is the second WPT failure after just one day of sheriffing. Can you make your testing bot more aggressive in disabling? Perhaps disable entirely on the Mac if most Mac bots fail? WPT peeps are more capable of evaluating imported tests accidentally disabled, and it would be nicer to the sheriffs.

Comment 1 by qyears...@chromium.org, Apr 16 2018

Cc: robertma@chromium.org qyears...@chromium.org
Components: -Tests>Fails Blink>Infra>Ecosystem
Owner: ----
Status: Available (was: Assigned)
Summary: WPT importer is not aggressive enough in deciding which platforms to mark as flaky/fail (was: WPT import failures)
Thanks for the report and the suggestion. Being extra aggressive when marking tests flaky/fail could reduce some unnecessary trouble for sheriffs when a test is flaky and happens to pass on a platform during import but fails later.

It seems like there could potentially be a change in:
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py

Moving this under Blink>Infra>Ecosystem. (I'm not the main owner of WPT import process now.)

Comment 2 by foolip@chromium.org, Jul 12 

robertma@, this can still happen, right?

Comment 3 by robertma@chromium.org, Jul 13

Labels: -Type-Bug Type-Feature
Yes. And it happens whenever a test fails but some try job doesn't finish because of infra issues. The test will be marked as expected to fail on all platforms except the one that doesn't finish.

Comment 4 by foolip@chromium.org, Sep 25 

robertma@, this is a P2. What should we do to fix this, would it be to generalize expectations in the face of incomplete results?
►

Sign in to add a comment