New issue
Advanced search Search tips

Issue 839657 link

Starred by 2 users

Issue metadata

Status: Untriaged
Owner: ----
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Remove SLOW and NEEDSMANUALREBASELINE from expectation of WebKit layout tests

Project Member Reported by st...@chromium.org, May 4 2018

Issue description

I think we were talking past each other.

These entries certainly show up in the expectations files, because they're legal there.

They should *never* show up in the `actual` values. From re-reading your comment, it sounds like they don't, either. Is that correct?

They *might* show up in the `expected` values; if they do, that's probably at best confusing and makes it a bit harder to parse the results. In particular `SLOW` is completely independent of everything else. NEEDSMANUALREBASELINE is effectively equivalent to [ PASS FAIL ], and WONTFIX is effectively equivalent to [ SKIP ]. However, I don't think these should affect any of the primary queries. Does that also sound right?

Comment 2 by st...@chromium.org, May 4 2018

No, they don't show up in `actual` values. (Sorry, forgot to make it clear on that)
To avoid further confusion, I've rephrased my comments from the CL as below.

Based on the 120-day data below, SLOW, NEEDSMANUALREBASELINE and LEAK do show up in run.expected, while 'TEXT', 'AUDIO', 'IMAGE', 'IMAGE_TEXT' and 'MISSING' do not show up in run.expected.
Are all these legal? If yes, we could close this bug; otherwise, let's keep it open.

Either way, this does affect the query to find flaky test failures on CQ: https://chromium-review.googlesource.com/c/infra/infra/+/1031017/13/appengine/findit/services/flake_detection/flaky_tests.cq_false_rejection.sql#278

--------------- 120-day data in step '%webkit_layout_test%', values showed up in `actual` -----------------
1	PASS	 
2	MISSING	 
3	LEAK	 
4	TEXT	 
5	SKIP	 
6	IMAGE_TEXT	 
7	CRASH	 
8	IMAGE	 
9	TIMEOUT

--------------- 120-day data in step '%webkit_layout_test%', values showed up in `expected` -----------------
1       TIMEOUT	 
2	SLOW	 
3	PASS	 
4	CRASH	 
5	WONTFIX	 
6	LEAK	 
7	SKIP	 
8	FAIL	 
9	NEEDSMANUALREBASELINE

Sign in to add a comment