Reverting crrev.com/c/884081

Philip,

Please take a look.

It's not missing -expected.txt file. The file doesn't exist because it is a testharness.js test and is expected to pass all assertions.

Reverting the import wouldn't really help because the next import will bring back the files again.

I tried to look into the failure on Win7 dbg. The output is a weird one-byte whitespace: https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=695d99fa4c91466a4d2093af463d66ba66adca9e
And the error message "(-expected.txt was missing)" doesn't really make sense as explained above.

I'll try to figure out what's going on. If I can't find obvious reasons soon, I'll send a TestExpectations change to disable the test on Win 7 dbg.

Confirm the test is in fact flaky on all platforms. The flake is more noticeable on debug builds.

The weird thing is that it's a pass-failure flake instead of pass-timeout. When the test fails, the output is empty. Even the testharness text is absent, which confuses the test runner.

AFAICT, the test is written correctly. It could be slow, but I don't see why the output can be empty... 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ca6ef980a1aed413f3c7652f37eb8e38ce0576f9

commit ca6ef980a1aed413f3c7652f37eb8e38ce0576f9
Author: Robert Ma <robertma@chromium.org>
Date: Wed Jan 24 19:48:43 2018

Mark acid3/numbered-tests.html as flaky

No-Try because the test has not been imported yet.

TBR=foolip

No-Try: True
Bug: 805463
Change-Id: I3418bc42699e7f9aa7e8e501c53ab881b5933594
Reviewed-on: https://chromium-review.googlesource.com/883729
Reviewed-by: Robert Ma <robertma@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#531652}
[modify] https://crrev.com/ca6ef980a1aed413f3c7652f37eb8e38ce0576f9/third_party/WebKit/LayoutTests/TestExpectations

Expectation change has landed. Downgrading the issue to P3.

Philip, can you take a look at why the test may produce empty output sometimes? I feel like my testharness/JS knowledge isn't sufficient. Thanks!

Note: I haven't yet observed the same failure mode in wptrunner, so it might be related to content_shell and/or our custom testharnessreport.js.

The test is still reported as failing for some reason.
https://ci.chromium.org/buildbot/chromium.win/Win7%20Tests%20%28dbg%29%281%29/65991

I'll apply 'Missing' attribute on it.

Oops, sorry, looks like missing is different from failure. Yes please change the expectation. Thank you!

'Missing' also didn't work

third_party/WebKit/LayoutTests/TestExpectations:1758 "Missing" expectations are not allowed; download new baselines (see https://goo.gl/SHVYrZ), or as a fallback, use "NeedsManualRebaseline". external/wpt/acid/acid3/numbered-tests.html

Sigh... I guess NeedsManualRebaseline is the best option then, despite not its intended use case. I don't want to skip the test because I'd like to see some more stats how often and on what (other) platforms it fails.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/55f9e53e4f2f850e11802d5f09ab9df8d4008793

commit 55f9e53e4f2f850e11802d5f09ab9df8d4008793
Author: Vasilii Sukhanov <vasilii@chromium.org>
Date: Thu Jan 25 16:31:03 2018

Mark acid3/numbered-tests.html as skip

The test is missing expectations. 'Failure' doesn't suppress it amd 'Missing' isn't allowed.

TBR=foolip@chromium.org
NOTRY=true

Bug: 805463
Change-Id: I0aad9cd95f47ee3c5c967d817711330ed4c68761
Reviewed-on: https://chromium-review.googlesource.com/887061
Reviewed-by: Vasilii Sukhanov <vasilii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#531914}
[modify] https://crrev.com/55f9e53e4f2f850e11802d5f09ab9df8d4008793/third_party/WebKit/LayoutTests/TestExpectations

I just read your comment but the CL has landed :(

Oh no worries. Skipping is fine, too.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/12ac6f19e140b0cacf3c91464a8585e80bb750b8

commit 12ac6f19e140b0cacf3c91464a8585e80bb750b8
Author: Robert Ma <robertma@chromium.org>
Date: Fri Mar 09 22:46:25 2018

Unskip acid3/numbered-tests.html

Looks like crbug.com/807627 has been fixed by the recent roll of wpt tools,
in which case this test won't be flaky any more.

Bug: 805463
Change-Id: I184560ed406a082c661f313047830d79d5d152d1
Reviewed-on: https://chromium-review.googlesource.com/957223
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542263}
[modify] https://crrev.com/12ac6f19e140b0cacf3c91464a8585e80bb750b8/third_party/WebKit/LayoutTests/TestExpectations

Sadly the test is still flaky:
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%28dbg%29%281%29/builds/66845
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%28dbg%29%281%29/builds/66843
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%28dbg%29%281%29/builds/66842
https://uberchromegw.corp.google.com/i/chromium.win/builders/Win7%20Tests%20%28dbg%29%281%29/builds/66840

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e9f36c62a656b541be316a1208d9e023a76d13e0

commit e9f36c62a656b541be316a1208d9e023a76d13e0
Author: Christian Dullweber <dullweber@chromium.org>
Date: Mon Mar 12 14:46:10 2018

Revert "Unskip acid3/numbered-tests.html"

This reverts commit 12ac6f19e140b0cacf3c91464a8585e80bb750b8.

Reason for revert: Test is flaky

Original change's description:
> Unskip acid3/numbered-tests.html
>
> Looks like crbug.com/807627 has been fixed by the recent roll of wpt tools,
> in which case this test won't be flaky any more.
>
> Bug: 805463
> Change-Id: I184560ed406a082c661f313047830d79d5d152d1
> Reviewed-on: https://chromium-review.googlesource.com/957223
> Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
> Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#542263}

TBR=foolip@chromium.org,robertma@chromium.org

No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 805463
Change-Id: Iec79614a2dfa66353bf4900d95ac7defc3b7a0bf
Reviewed-on: https://chromium-review.googlesource.com/959062
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Christian Dullweber <dullweber@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542479}
[modify] https://crrev.com/e9f36c62a656b541be316a1208d9e023a76d13e0/third_party/WebKit/LayoutTests/TestExpectations