Filed https://github.com/w3c/web-platform-tests/issues/3850 to move some tests in old-tests/submission/Opera/script_scheduling

I have finished filing bugs now, but not for these that are due to https://www.w3.org/Bugs/Public/show_bug.cgi?id=28244

These also seem to pass, so either the test or Blink changed since the run:
http://w3c-test.org/compat/webkit-text-fill-color-property-002.html
http://w3c-test.org/webgl/conformance-1.0.3/conformance/glsl/bugs/uniforms-should-not-lose-values.html

Adding Blink>HTML component because we need something to avoid constant triage and there is no good component for cross-area meta-bugs.

Thanks for doing this, foolip@ !

From conversation w/ jgraham on #whatwg, the dataset was likely testing something before Chrome 53, so we'll have made several fixes since.

Yeah first step for any of these is probably to verify that the test is indeed still failing (with a current chrome and current WPT checkout).

For people who aren't familiar, individual web-platform-tests can be easily run via https://w3c-test.org/, and there's a simplistic runner at https://w3c-test.org/tools/runner/index.html which can automate running most of the tests in-browser (relies on an iframe, so not perfect).

Assigning to foolip@ to track the meta-issue here, obviously the main work is being done in the blocking bugs.

I don't expect this to be a long-lived tracking effort, when the blocking bugs have been looked at or turned into more specific bugs, I'll close them and try to summarize how valuable this exercise was, if it's worth tracking 2 vs. 1 failures specifically in a future dashboard.

I guess these are all not in imported/ yet. Is the plan to put these tests in imported/ so they run on the CQ?

> Is the plan to put these tests in imported/ so they run on the CQ?

It depends on a WPT directory.

For example, wpt/workers is not imported because no one owns it and haven't imported yet.  If someone owns it, it can be imported immediately.
wpt/html is owned and was imported, but many tests are skipped to be imported because of Blink infra issues. Importing such tests is not a easy task.

ok worker team will own and import workers

Great, thank you falken@!  In general it's up to each team to decide whether the cost of monitoring their relevant WPTs is worth the benefit they're getting (in some cases we know the test quality is low).  The idea of this exercise is to hopefully identify some tests whose results are valuable - pointing to real bugs in blink that we do actually care about.  Does that appear to be the case for workers?

>  Does that appear to be the case for workers?

Makes sense. We'll investigate at Issue 651778.

Attaching results.db.gz in case http://hoppipolla.co.uk/410/results.db.gz goes away.

Deleted: results.db.gz 23.3 MB

results.db.gz 23.3 MB Download

I've computed some aggregate stats from this data here: https://docs.google.com/spreadsheets/d/1n01Y67B_pL_JnXgBxSEF7Jok7XYmH63WhxzF_JYoDR8/edit#gid=262399575

Wanted to update this bug since it's relevant: the Web Platform Tests Dashboard now gives you the ability to see which tests are least interoperable per (browser, spec) combination:

e.g. https://wptdashboard.appspot.com/interop/chrome/cssom

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/582bd4177742be6f5b06056170939778830d1f1f

commit 582bd4177742be6f5b06056170939778830d1f1f
Author: Rob Buis <rob.buis@samsung.com>
Date: Tue Oct 17 23:25:21 2017

Thead setter should throw if not handed a thead parameter

Thead setter should throw if not handed a thead parameter [1]:
"If the new value is neither null nor a thead element, then
a "HierarchyRequestError" DOMException must be thrown instead."

[1] https://html.spec.whatwg.org/#dom-table-thead

Bug:  651572 
Change-Id: Ia65a259b7bc6e9cca50d5491c497012149d1fa8e
Reviewed-on: https://chromium-review.googlesource.com/721860
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509588}
[delete] https://crrev.com/8896b1cda304a7d7f8ba08dec69714243a1a7320/third_party/WebKit/LayoutTests/external/wpt/html/semantics/tabular-data/the-table-element/tHead-expected.txt
[modify] https://crrev.com/582bd4177742be6f5b06056170939778830d1f1f/third_party/WebKit/Source/core/html/HTMLTableElement.cpp
[modify] https://crrev.com/582bd4177742be6f5b06056170939778830d1f1f/third_party/WebKit/Source/core/html/HTMLTableElement.idl

I've created a new spreadsheet from the same data for a different purpose:
https://docs.google.com/spreadsheets/d/1u6fBSy3SXJBRG57NKyu0f2M8PTitZdetbZzDiw57eFE/edit?usp=sharing

Here's the query I used:
.mode csv
.output wpt.csv
select test.test, test.subtest, chrome.status, fx.status, edge.status
  from test
  join (select test.id as test_id, status.name as status
          from test
          join result on result.test_id = test.id
          join run on run.id = result.run_id
          join status on result.status_id = status.id
          where run.name = "firefox") as fx
    on fx.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "chrome") as chrome
     on chrome.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "edge") as edge
     on edge.test_id = test.id
  where fx.status not in ("PASS", "OK") and
        edge.status not in ("PASS", "OK") and
        chrome.status not in ("PASS", "OK")
  order by test.test;

(Same as original but all of them failing.)

https://gist.github.com/foolip/bb5dc2b02cc87ac4412dcd348868edd1 is an up-to-date list of tests that fail in Chrome but pass in Edge and Firefox.

I've gone through the remaining open issues and added them to one of two hotlists:

https://bugs.chromium.org/u/foolip@chromium.org/hotlists/WPT-Chrome-Fail-Edge-Firefox-Safari-Pass for tests that also pass in Safari, i.e. Chrome-only failures.

https://bugs.chromium.org/u/foolip@chromium.org/hotlists/WPT-Chrome-Safari-Fail-Edge-Firefox-Pass for what this issue was originally about, but only two bugs remain in that category.

I'll close this issue, since tracking with labels or hotlists works better, and there won't be a point in time when this issue is totally fixed.

A subset of this tracking bug (Chrome-only failures) is continued in issue 896242. Issue 651768 and issue 651774 remain blocked on this bug for now, but might later be moved to a "Chrome+Safari failures" tracking issue.