New issue
Advanced search Search tips

Issue 664268 link

Starred by 3 users
Status: Fixed
Owner:
qyears...@chromium.org
Closed: Dec 2016
Cc:
tkent@chromium.org
jeffcarp@chromium.org
jsb...@chromium.org
dpranke@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 640947



Sign in to add a comment

Layout test flakiness cause: run-webkit-tests timeout vs testharness.js timeout.

Project Member Reported by qyears...@chromium.org, Nov 10 2016 
For testharness.js tests, there is timer in the test itself and when the time taken passes that, the result is the text TIMEOUT in the actual result.

The test runner also has a timeout mechanism, but it can change, and the default varies between debug and release builds.

So, for some tests, depending on the default timeout for testharness.js, and the timeout for the layout test runner, and whether the build is debug or release, the test result can be either failure (text diff) or timeout.

This is something that's frequently hit by w3c-test-autoroller, since if the test times out on the try bots, it may fail with the text "TIMEOUT" on debug builds.

Comment 1 by tkent@chromium.org, Nov 11 2016

Cc: tkent@chromium.org
How about just disabling the timeout detection of testharness.js?
Project Member

Comment 2 by bugdroid1@chromium.org, Nov 11 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/668b72eeb4c7e68e918c6a9f9f0be45b57db4330

commit 668b72eeb4c7e68e918c6a9f9f0be45b57db4330
Author: tkent <tkent@chromium.org>
Date: Fri Nov 11 01:39:45 2016

Gardening: Mark reload_post_1.html Failure too with --enable-wptserve.

BUG= 664268 
TBR=qyearsley@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2492093002
Cr-Commit-Position: refs/heads/master@{#431447}

[modify] https://crrev.com/668b72eeb4c7e68e918c6a9f9f0be45b57db4330/third_party/WebKit/LayoutTests/WPTServeExpectations

Comment 3 by tkent@chromium.org, Nov 11 2016 

We can disable testharness.js timeout by a testharnessreport.js change.

diff --git a/third_party/WebKit/LayoutTests/imported/wpt/resources/testharnessreport.js b/third_party/WebKit/LayoutTests/imported/wpt/resources/testharnessreport.js
index da58d62..3304461 100644
--- a/third_party/WebKit/LayoutTests/imported/wpt/resources/testharnessreport.js
+++ b/third_party/WebKit/LayoutTests/imported/wpt/resources/testharnessreport.js
@@ -26,7 +26,7 @@
     // test results into an HTML table.  When that table is dumped as text, no
     // spacing between cells is preserved, and it is therefore not readable. By
     // setting output to false, the HTML table will not be created.
-    setup({"output":false});
+    setup({"output":false, "explicit_timeout":true});
 
     // Function used to convert the test status code into the corresponding
     // string

Comment 4 by mek@chromium.org, Nov 11 2016

Blocking: 640947

Comment 5 by mek@chromium.org, Nov 11 2016 

For what it's worth, before we began importing testharness.js regularly from web-platform-tests our local copy already had a modified timeout to make sure the testharness.js timeout would never be hit before the test runner timeout (https://chromium.googlesource.com/chromium/src/+/e637932966ffda86c10fd345a6e5d8392ccfd716). So going back to that situation (but this time without modifying testharness.js) sounds like a very sensible thing to do.

Comment 6 by qyears...@chromium.org, Nov 11 2016 

Awesome, that sounds like the right solution to me too :-)
Project Member

Comment 7 by bugdroid1@chromium.org, Nov 15 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1a8851a5030c4e6dbb51c98db5c3600d742985be

commit 1a8851a5030c4e6dbb51c98db5c3600d742985be
Author: qyearsley <qyearsley@chromium.org>
Date: Tue Nov 15 22:29:53 2016

Disable testharness.js timeout.

Reason: Our test runner has a separate timeout mechanism; currently whether the test result is Failure or Timeout depends on which happens first: hitting the testharness.js timeout or the run-webkit-tests timeout.

Disabling the testharness.js timeout by default by setting explicit_timeout was suggested by tkent@.

In some cases, this means that some tests that were hitting the testharness.js timeout in Debug builds will now be continuing to run longer, and either timing out or failing in a different way.

BUG= 664268 

Review-Url: https://codereview.chromium.org/2500053002
Cr-Commit-Position: refs/heads/master@{#432271}

[modify] https://crrev.com/1a8851a5030c4e6dbb51c98db5c3600d742985be/third_party/WebKit/LayoutTests/SlowTests
[modify] https://crrev.com/1a8851a5030c4e6dbb51c98db5c3600d742985be/third_party/WebKit/LayoutTests/TestExpectations
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/html/semantics/embedded-content/media-elements/playing-the-media-resource/pause-remove-from-document-networkState-expected.txt
[modify] https://crrev.com/1a8851a5030c4e6dbb51c98db5c3600d742985be/third_party/WebKit/LayoutTests/imported/wpt/resources/testharnessreport.js
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/client-navigate.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/clients-get-cross-origin.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/clients-matchall-client-types.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-canvas-tainting-cache.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-canvas-tainting.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-cors-xhr.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-csp.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-event-async-respond-with.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-event-respond-with-stops-propagation.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-request-css-base-url.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-request-xhr.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/fetch-response-xhr.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/getregistrations.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/invalid-blobtype.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/invalid-header.https-expected.txt
[delete] https://crrev.com/e29f8ab494ddbb5c85407d544204c5168e5028d8/third_party/WebKit/LayoutTests/imported/wpt/service-workers/service-worker/referer.https-expected.txt
[modify] https://crrev.com/1a8851a5030c4e6dbb51c98db5c3600d742985be/third_party/WebKit/LayoutTests/resources/testharnessreport.js

Comment 8 by falken@chromium.org, Nov 16 2016 

 Issue 446352  has been merged into this issue.

Comment 9 by qyears...@chromium.org, Dec 9 2016

Status: done (was: Available)

Comment 10 by qyears...@chromium.org, Dec 9 2016

Owner: qyears...@chromium.org
Status: Fixed (was: Done)
►

Sign in to add a comment