New issue
Advanced search Search tips

Issue 660888 link

Starred by 1 user
Status: Fixed
Owner:
qyears...@chromium.org
Closed: Nov 2016
Cc:
jeffcarp@chromium.org
jsb...@chromium.org
foolip@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 621599



Sign in to add a comment

w3c test imports sometimes also require rebaselining non-wpt tests

Project Member Reported by qyears...@chromium.org, Oct 31 2016 
In the latest w3c-test-autoroller CLs (eg https://codereview.chromium.org/2464743003/), there are some non-imported tests which appear to fail consistently.

foolip@ also noticed this in a recent change.

My hypothesis for this is that when importing, we update third_party/WebKit/LayoutTests/resources/testharness.js, and this affects tests outside of imported/wpt.

Possible change 1: Stop updating files in LayoutTests/resources/, require humans to manually update those files.
Possible change 2: When there are changes in LayoutTests/resources/, update baselines and expectations for any newly-failing tests, even if they're not in imported/wpt.

I think change 2 sounds easier to manage, even though it means that "updating wpt" implies "updating testharness.js and potentially anything that depends on it".
Project Member

Comment 1 by bugdroid1@chromium.org, Oct 31 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6d2089d2dbb5a9ceefa7b2c4fd23d48288745c24

commit 6d2089d2dbb5a9ceefa7b2c4fd23d48288745c24
Author: qyearsley <qyearsley@chromium.org>
Date: Mon Oct 31 23:54:34 2016

In update-w3c-test-expectations, also download baselines for non-imported tests.

BUG= 660888 

Review-Url: https://codereview.chromium.org/2468553002
Cr-Commit-Position: refs/heads/master@{#428871}

[modify] https://crrev.com/6d2089d2dbb5a9ceefa7b2c4fd23d48288745c24/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations.py

Comment 2 by qyears...@chromium.org, Nov 2 2016

Cc: jsb...@chromium.org
Status: Started (was: Assigned)
So now I've made it so that when update-w3c-test-expectations invokes webkit-patch rebaseline-cl, it doesn't use --only-changed-tests, so it will try to download new baselines for any tests which seem to be newly failing.

However, when there are flaky tests that fail with the patch but not without, then new baselines may (wrongly) be downloaded for those tests, which will cause the import to fail. I noticed this when trying to manually update wpt in https://codereview.chromium.org/2468053002.

In the best case, this flakiness would be an occasional but not common problem, and would just result in imports occasionally failing.

Comment 3 by qyears...@chromium.org, Nov 15 2016

Status: Fixed (was: Started)

Comment 4 by qyears...@chromium.org, Nov 17 2016 

 Issue 659180  has been merged into this issue.
►

Sign in to add a comment