New issue
Advanced search Search tips
Starred by 3 users

Issue metadata

Status: Closed
Owner:
Closed: Oct 12
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug


Sign in to add a comment
link

Issue 651572: Investigate web-platform-tests that fail in Chrome but pass in Firefox and Edge

Reported by foolip@chromium.org, Sep 29 2016 Project Member

Issue description

https://docs.google.com/spreadsheets/d/1dIBAZwkuGy-H1AEk9pT1xF-pLb2Du8zZUnSRmrNdtj0/edit?usp=sharing

Data from http://hoppipolla.co.uk/410/results.db.gz

CSV filed produced like this:
$ sqlite results.db
.mode csv
.output wpt.cvs
select test.test, test.subtest, chrome.status, fx.status, edge.status
  from test
  join (select test.id as test_id, status.name as status
          from test
          join result on result.test_id = test.id
          join run on run.id = result.run_id
          join status on result.status_id = status.id
          where run.name = "firefox") as fx
    on fx.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "chrome") as chrome
     on chrome.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "edge") as edge
     on edge.test_id = test.id
  where fx.status in ("PASS", "OK") and
        edge.status in ("PASS", "OK") and
        chrome.status not in ("PASS", "OK")
  order by test.test;
 

Comment 1 by foolip@chromium.org, Sep 29 2016

Cc: tkent@chromium.org

Comment 2 by foolip@chromium.org, Sep 30 2016

Blockedon: 651741

Comment 3 by foolip@chromium.org, Sep 30 2016

Blockedon: 651742

Comment 4 by foolip@chromium.org, Sep 30 2016

Blockedon: 651743

Comment 5 by foolip@chromium.org, Sep 30 2016

Blockedon: 651744

Comment 6 by foolip@chromium.org, Sep 30 2016

Blockedon: 651746

Comment 7 by foolip@chromium.org, Sep 30 2016

Blockedon: 651748

Comment 8 by foolip@chromium.org, Sep 30 2016

Blockedon: 651749

Comment 9 by foolip@chromium.org, Sep 30 2016

Blockedon: 651750

Comment 10 by foolip@chromium.org, Sep 30 2016

Blockedon: 651752

Comment 11 by foolip@chromium.org, Sep 30 2016

Blockedon: 651753

Comment 12 by foolip@chromium.org, Sep 30 2016

Blocking: 651755

Comment 13 by foolip@chromium.org, Sep 30 2016

Blockedon: 651757

Comment 14 by foolip@chromium.org, Sep 30 2016

Blocking: -651755

Comment 15 by foolip@chromium.org, Sep 30 2016

Blockedon: 651755

Comment 16 by foolip@chromium.org, Sep 30 2016

Blockedon: 651759

Comment 17 by foolip@chromium.org, Sep 30 2016

Blockedon: 651761

Comment 18 by foolip@chromium.org, Sep 30 2016

Blockedon: 651762

Comment 19 by foolip@chromium.org, Sep 30 2016

Blockedon: 651764

Comment 20 by foolip@chromium.org, Sep 30 2016

Blockedon: 651765

Comment 21 by foolip@chromium.org, Sep 30 2016

Filed https://github.com/w3c/web-platform-tests/issues/3850 to move some tests in old-tests/submission/Opera/script_scheduling

Comment 22 by foolip@chromium.org, Sep 30 2016

Blockedon: 651767

Comment 23 by foolip@chromium.org, Sep 30 2016

Blockedon: 651768

Comment 24 by foolip@chromium.org, Sep 30 2016

Blockedon: 651769

Comment 25 by foolip@chromium.org, Sep 30 2016

Blockedon: v8:5450

Comment 26 by foolip@chromium.org, Sep 30 2016

Blockedon: 651770

Comment 27 by foolip@chromium.org, Sep 30 2016

Blockedon: 651771

Comment 28 by foolip@chromium.org, Sep 30 2016

Blockedon: 651772

Comment 29 by foolip@chromium.org, Sep 30 2016

Blockedon: 651773

Comment 30 by foolip@chromium.org, Sep 30 2016

Blockedon: 651774

Comment 31 by foolip@chromium.org, Sep 30 2016

Blockedon: 651777

Comment 32 by foolip@chromium.org, Sep 30 2016

Blockedon: 651778

Comment 33 by foolip@chromium.org, Sep 30 2016

Blockedon: 651781

Comment 34 by foolip@chromium.org, Sep 30 2016

Comment 35 by schenney@chromium.org, Sep 30 2016

Components: -Blink Blink>HTML
Adding Blink>HTML component because we need something to avoid constant triage and there is no good component for cross-area meta-bugs.

Comment 36 by jsb...@chromium.org, Sep 30 2016

Thanks for doing this, foolip@ !

From conversation w/ jgraham on #whatwg, the dataset was likely testing something before Chrome 53, so we'll have made several fixes since.

Comment 37 by rbyers@chromium.org, Sep 30 2016

Labels: Hotlist-PredictabilityInfra
Owner: foolip@chromium.org
Status: Assigned (was: Available)
Yeah first step for any of these is probably to verify that the test is indeed still failing (with a current chrome and current WPT checkout).

For people who aren't familiar, individual web-platform-tests can be easily run via https://w3c-test.org/, and there's a simplistic runner at https://w3c-test.org/tools/runner/index.html which can automate running most of the tests in-browser (relies on an iframe, so not perfect).

Assigning to foolip@ to track the meta-issue here, obviously the main work is being done in the blocking bugs.

Comment 38 by foolip@chromium.org, Sep 30 2016

I don't expect this to be a long-lived tracking effort, when the blocking bugs have been looked at or turned into more specific bugs, I'll close them and try to summarize how valuable this exercise was, if it's worth tracking 2 vs. 1 failures specifically in a future dashboard.

Comment 39 by tkent@chromium.org, Oct 5 2016

Blocking: 573309

Comment 40 by falken@chromium.org, Oct 7 2016

I guess these are all not in imported/ yet. Is the plan to put these tests in imported/ so they run on the CQ?

Comment 41 by tkent@chromium.org, Oct 7 2016

Cc: falken@chromium.org
> Is the plan to put these tests in imported/ so they run on the CQ?

It depends on a WPT directory.

For example, wpt/workers is not imported because no one owns it and haven't imported yet.  If someone owns it, it can be imported immediately.
wpt/html is owned and was imported, but many tests are skipped to be imported because of Blink infra issues. Importing such tests is not a easy task.

Comment 42 by falken@chromium.org, Oct 7 2016

ok worker team will own and import workers

Comment 43 by rbyers@chromium.org, Oct 7 2016

Great, thank you falken@!  In general it's up to each team to decide whether the cost of monitoring their relevant WPTs is worth the benefit they're getting (in some cases we know the test quality is low).  The idea of this exercise is to hopefully identify some tests whose results are valuable - pointing to real bugs in blink that we do actually care about.  Does that appear to be the case for workers?

Comment 44 by falken@chromium.org, Oct 11 2016

>  Does that appear to be the case for workers?

Makes sense. We'll investigate at Issue 651778.

Comment 45 by foolip@chromium.org, Oct 12 2016

Attaching results.db.gz in case http://hoppipolla.co.uk/410/results.db.gz goes away.
results.db.gz
23.3 MB Download

Comment 47 by tkent@chromium.org, Nov 21 2016

Blocking: -573309

Comment 48 by rbyers@chromium.org, Jan 3 2017

Components: Blink>Infra>Predictability
Labels: -Hotlist-PredictabilityInfra

Comment 49 by jeffcarp@chromium.org, Jan 20 2017

Cc: jeffcarp@chromium.org

Comment 50 by tkent@chromium.org, Mar 3 2017

Blockedon: 643712
Components: -Blink>HTML

Comment 51 by jeffcarp@chromium.org, Mar 22 2017

Wanted to update this bug since it's relevant: the Web Platform Tests Dashboard now gives you the ability to see which tests are least interoperable per (browser, spec) combination:

e.g. https://wptdashboard.appspot.com/interop/chrome/cssom

Comment 52 by yukishiino@chromium.org, May 9 2017

Blockedon: -643712

Comment 53 by foolip@chromium.org, Jul 3 2017

Components: Blink>Infra>Ecosystem

Comment 54 by foolip@chromium.org, Jul 3 2017

Components: -Blink>Infra>Predictability

Comment 55 by bugdroid1@chromium.org, Oct 17 2017

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/582bd4177742be6f5b06056170939778830d1f1f

commit 582bd4177742be6f5b06056170939778830d1f1f
Author: Rob Buis <rob.buis@samsung.com>
Date: Tue Oct 17 23:25:21 2017

Thead setter should throw if not handed a thead parameter

Thead setter should throw if not handed a thead parameter [1]:
"If the new value is neither null nor a thead element, then
a "HierarchyRequestError" DOMException must be thrown instead."

[1] https://html.spec.whatwg.org/#dom-table-thead

Bug:  651572 
Change-Id: Ia65a259b7bc6e9cca50d5491c497012149d1fa8e
Reviewed-on: https://chromium-review.googlesource.com/721860
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509588}
[delete] https://crrev.com/8896b1cda304a7d7f8ba08dec69714243a1a7320/third_party/WebKit/LayoutTests/external/wpt/html/semantics/tabular-data/the-table-element/tHead-expected.txt
[modify] https://crrev.com/582bd4177742be6f5b06056170939778830d1f1f/third_party/WebKit/Source/core/html/HTMLTableElement.cpp
[modify] https://crrev.com/582bd4177742be6f5b06056170939778830d1f1f/third_party/WebKit/Source/core/html/HTMLTableElement.idl

Comment 56 by foolip@chromium.org, Jan 12 2018

I've created a new spreadsheet from the same data for a different purpose:
https://docs.google.com/spreadsheets/d/1u6fBSy3SXJBRG57NKyu0f2M8PTitZdetbZzDiw57eFE/edit?usp=sharing

Here's the query I used:
.mode csv
.output wpt.csv
select test.test, test.subtest, chrome.status, fx.status, edge.status
  from test
  join (select test.id as test_id, status.name as status
          from test
          join result on result.test_id = test.id
          join run on run.id = result.run_id
          join status on result.status_id = status.id
          where run.name = "firefox") as fx
    on fx.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "chrome") as chrome
     on chrome.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "edge") as edge
     on edge.test_id = test.id
  where fx.status not in ("PASS", "OK") and
        edge.status not in ("PASS", "OK") and
        chrome.status not in ("PASS", "OK")
  order by test.test;

(Same as original but all of them failing.)

Comment 57 by foolip@chromium.org, Oct 12

https://gist.github.com/foolip/bb5dc2b02cc87ac4412dcd348868edd1 is an up-to-date list of tests that fail in Chrome but pass in Edge and Firefox.

Comment 58 by foolip@chromium.org, Oct 12

Status: Closed (was: Assigned)
I've gone through the remaining open issues and added them to one of two hotlists:

https://bugs.chromium.org/u/foolip@chromium.org/hotlists/WPT-Chrome-Fail-Edge-Firefox-Safari-Pass for tests that also pass in Safari, i.e. Chrome-only failures.

https://bugs.chromium.org/u/foolip@chromium.org/hotlists/WPT-Chrome-Safari-Fail-Edge-Firefox-Pass for what this issue was originally about, but only two bugs remain in that category.

I'll close this issue, since tracking with labels or hotlists works better, and there won't be a point in time when this issue is totally fixed.

Comment 59 by foolip@chromium.org, Oct 18

A subset of this tracking bug (Chrome-only failures) is continued in issue 896242. Issue 651768 and issue 651774 remain blocked on this bug for now, but might later be moved to a "Chrome+Safari failures" tracking issue.

Sign in to add a comment