New issue
Advanced search Search tips

Issue 651572 link

Starred by 3 users

Issue metadata

Status: Closed
Owner:
Closed: Oct 12
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug


Sign in to add a comment

Investigate web-platform-tests that fail in Chrome but pass in Firefox and Edge

Project Member Reported by foolip@chromium.org, Sep 29 2016

Issue description

https://docs.google.com/spreadsheets/d/1dIBAZwkuGy-H1AEk9pT1xF-pLb2Du8zZUnSRmrNdtj0/edit?usp=sharing

Data from http://hoppipolla.co.uk/410/results.db.gz

CSV filed produced like this:
$ sqlite results.db
.mode csv
.output wpt.cvs
select test.test, test.subtest, chrome.status, fx.status, edge.status
  from test
  join (select test.id as test_id, status.name as status
          from test
          join result on result.test_id = test.id
          join run on run.id = result.run_id
          join status on result.status_id = status.id
          where run.name = "firefox") as fx
    on fx.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "chrome") as chrome
     on chrome.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "edge") as edge
     on edge.test_id = test.id
  where fx.status in ("PASS", "OK") and
        edge.status in ("PASS", "OK") and
        chrome.status not in ("PASS", "OK")
  order by test.test;
 

Comment 1 by foolip@chromium.org, Sep 29 2016

Cc: tkent@chromium.org

Comment 2 by foolip@chromium.org, Sep 30 2016

Blockedon: 651741

Comment 3 by foolip@chromium.org, Sep 30 2016

Blockedon: 651742

Comment 4 by foolip@chromium.org, Sep 30 2016

Blockedon: 651743

Comment 5 by foolip@chromium.org, Sep 30 2016

Blockedon: 651744

Comment 6 by foolip@chromium.org, Sep 30 2016

Blockedon: 651746

Comment 7 by foolip@chromium.org, Sep 30 2016

Blockedon: 651748

Comment 8 by foolip@chromium.org, Sep 30 2016

Blockedon: 651749

Comment 9 by foolip@chromium.org, Sep 30 2016

Blockedon: 651750
Blockedon: 651752
Blockedon: 651753
Blocking: 651755
Blockedon: 651757
Blocking: -651755
Blockedon: 651755
Blockedon: 651759
Blockedon: 651761
Blockedon: 651762
Blockedon: 651764
Blockedon: 651765
Filed https://github.com/w3c/web-platform-tests/issues/3850 to move some tests in old-tests/submission/Opera/script_scheduling
Blockedon: 651767
Blockedon: 651768
Blockedon: 651769
Blockedon: v8:5450
Blockedon: 651770
Blockedon: 651771
Blockedon: 651772
Blockedon: 651773
Blockedon: 651774
Blockedon: 651777
Blockedon: 651778
Blockedon: 651781
I have finished filing bugs now, but not for these that are due to https://www.w3.org/Bugs/Public/show_bug.cgi?id=28244

These also seem to pass, so either the test or Blink changed since the run:
http://w3c-test.org/compat/webkit-text-fill-color-property-002.html
http://w3c-test.org/webgl/conformance-1.0.3/conformance/glsl/bugs/uniforms-should-not-lose-values.html
Components: -Blink Blink>HTML
Adding Blink>HTML component because we need something to avoid constant triage and there is no good component for cross-area meta-bugs.
Thanks for doing this, foolip@ !

From conversation w/ jgraham on #whatwg, the dataset was likely testing something before Chrome 53, so we'll have made several fixes since.


Labels: Hotlist-PredictabilityInfra
Owner: foolip@chromium.org
Status: Assigned (was: Available)
Yeah first step for any of these is probably to verify that the test is indeed still failing (with a current chrome and current WPT checkout).

For people who aren't familiar, individual web-platform-tests can be easily run via https://w3c-test.org/, and there's a simplistic runner at https://w3c-test.org/tools/runner/index.html which can automate running most of the tests in-browser (relies on an iframe, so not perfect).

Assigning to foolip@ to track the meta-issue here, obviously the main work is being done in the blocking bugs.

I don't expect this to be a long-lived tracking effort, when the blocking bugs have been looked at or turned into more specific bugs, I'll close them and try to summarize how valuable this exercise was, if it's worth tracking 2 vs. 1 failures specifically in a future dashboard.
Blocking: 573309
I guess these are all not in imported/ yet. Is the plan to put these tests in imported/ so they run on the CQ?
Cc: falken@chromium.org
> Is the plan to put these tests in imported/ so they run on the CQ?

It depends on a WPT directory.

For example, wpt/workers is not imported because no one owns it and haven't imported yet.  If someone owns it, it can be imported immediately.
wpt/html is owned and was imported, but many tests are skipped to be imported because of Blink infra issues. Importing such tests is not a easy task.


ok worker team will own and import workers
Great, thank you falken@!  In general it's up to each team to decide whether the cost of monitoring their relevant WPTs is worth the benefit they're getting (in some cases we know the test quality is low).  The idea of this exercise is to hopefully identify some tests whose results are valuable - pointing to real bugs in blink that we do actually care about.  Does that appear to be the case for workers?
>  Does that appear to be the case for workers?

Makes sense. We'll investigate at Issue 651778.
Attaching results.db.gz in case http://hoppipolla.co.uk/410/results.db.gz goes away.
results.db.gz
23.3 MB Download

Comment 47 by tkent@chromium.org, Nov 21 2016

Blocking: -573309
Components: Blink>Infra>Predictability
Labels: -Hotlist-PredictabilityInfra
Cc: jeffcarp@chromium.org
Blockedon: 643712
Components: -Blink>HTML
Wanted to update this bug since it's relevant: the Web Platform Tests Dashboard now gives you the ability to see which tests are least interoperable per (browser, spec) combination:

e.g. https://wptdashboard.appspot.com/interop/chrome/cssom
Blockedon: -643712
Components: Blink>Infra>Ecosystem
Components: -Blink>Infra>Predictability
Project Member

Comment 55 by bugdroid1@chromium.org, Oct 17 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/582bd4177742be6f5b06056170939778830d1f1f

commit 582bd4177742be6f5b06056170939778830d1f1f
Author: Rob Buis <rob.buis@samsung.com>
Date: Tue Oct 17 23:25:21 2017

Thead setter should throw if not handed a thead parameter

Thead setter should throw if not handed a thead parameter [1]:
"If the new value is neither null nor a thead element, then
a "HierarchyRequestError" DOMException must be thrown instead."

[1] https://html.spec.whatwg.org/#dom-table-thead

Bug:  651572 
Change-Id: Ia65a259b7bc6e9cca50d5491c497012149d1fa8e
Reviewed-on: https://chromium-review.googlesource.com/721860
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509588}
[delete] https://crrev.com/8896b1cda304a7d7f8ba08dec69714243a1a7320/third_party/WebKit/LayoutTests/external/wpt/html/semantics/tabular-data/the-table-element/tHead-expected.txt
[modify] https://crrev.com/582bd4177742be6f5b06056170939778830d1f1f/third_party/WebKit/Source/core/html/HTMLTableElement.cpp
[modify] https://crrev.com/582bd4177742be6f5b06056170939778830d1f1f/third_party/WebKit/Source/core/html/HTMLTableElement.idl

I've created a new spreadsheet from the same data for a different purpose:
https://docs.google.com/spreadsheets/d/1u6fBSy3SXJBRG57NKyu0f2M8PTitZdetbZzDiw57eFE/edit?usp=sharing

Here's the query I used:
.mode csv
.output wpt.csv
select test.test, test.subtest, chrome.status, fx.status, edge.status
  from test
  join (select test.id as test_id, status.name as status
          from test
          join result on result.test_id = test.id
          join run on run.id = result.run_id
          join status on result.status_id = status.id
          where run.name = "firefox") as fx
    on fx.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "chrome") as chrome
     on chrome.test_id = test.id
  join (select test.id as test_id, status.name as status
        from test
        join result on result.test_id = test.id
        join run on run.id = result.run_id
        join status on result.status_id = status.id
        where run.name = "edge") as edge
     on edge.test_id = test.id
  where fx.status not in ("PASS", "OK") and
        edge.status not in ("PASS", "OK") and
        chrome.status not in ("PASS", "OK")
  order by test.test;

(Same as original but all of them failing.)
https://gist.github.com/foolip/bb5dc2b02cc87ac4412dcd348868edd1 is an up-to-date list of tests that fail in Chrome but pass in Edge and Firefox.
Status: Closed (was: Assigned)
I've gone through the remaining open issues and added them to one of two hotlists:

https://bugs.chromium.org/u/foolip@chromium.org/hotlists/WPT-Chrome-Fail-Edge-Firefox-Safari-Pass for tests that also pass in Safari, i.e. Chrome-only failures.

https://bugs.chromium.org/u/foolip@chromium.org/hotlists/WPT-Chrome-Safari-Fail-Edge-Firefox-Pass for what this issue was originally about, but only two bugs remain in that category.

I'll close this issue, since tracking with labels or hotlists works better, and there won't be a point in time when this issue is totally fixed.
A subset of this tracking bug (Chrome-only failures) is continued in issue 896242. Issue 651768 and issue 651774 remain blocked on this bug for now, but might later be moved to a "Chrome+Safari failures" tracking issue.

Sign in to add a comment