New issue
Advanced search Search tips

Issue 663860 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocking:
issue 601332



Sign in to add a comment

4 tests in http/tests/appcache/ are flaky when run in random order

Project Member Reported by jeffcarp@chromium.org, Nov 9 2016

Issue description

The tests:

http/tests/appcache/main-resource-redirect.html
http/tests/appcache/non-html.xhtml
http/tests/appcache/obsolete-error-events.html
http/tests/appcache/remove-cache.html

appear to be flaky when run in random order. Steps to reproduce:

This should pass consistently:
Tools/Scripts/run-webkit-tests http/tests/appcache/

This should fail intermittently:
Tools/Scripts/run-webkit-tests http/tests/appcache/ --order=random

We're working on removing order dependency from Blink layout tests. More context here:
https://docs.google.com/document/d/11-628Ny6hY__kfIv__zVD0m3rA0OB9tQLLC_gJI_PNg/edit

 
Description: Show this description

Comment 2 by jsb...@chromium.org, Nov 29 2016

Able to repro. Output locally and on the bots is just a spurious extra (or sometimes missing):

CONSOLE WARNING: line 22: Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.

I see that when running w/o random order, so it appears to just be general flakiness. http/tests/appcache/obsolete-error-events.html on the dashboard shows as generally flaky anyway.

Comment 3 by jsb...@chromium.org, Nov 29 2016

Many of the appcache tests use a trick where the manifest is a PHP file which can be accessed with ?command=XXX to control server behavior (e.g. "connect" or "disconnect" to simulate online/offline behavior).

These 4 in particular have code like:

// If the manifest script happened to be in a wrong state, reset it.
var timeoutId = setTimeout(resetManifest, 100);

(where resetManifest does a sync XHR)

... which explains the random order flakiness: if a previous run left the server in a bad state, the test needs to do an extra XHR to reset it, causing the extra logging.

Comment 4 by jsb...@chromium.org, Nov 29 2016

Owner: jsb...@chromium.org
Status: Started (was: Untriaged)
Project Member

Comment 5 by bugdroid1@chromium.org, Nov 30 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8

commit 1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8
Author: jsbell <jsbell@chromium.org>
Date: Wed Nov 30 22:31:15 2016

Deflake tests in http/tests/appcache/ when run in random order

The tests use sync XHR to poke the server to tweak the AppCache manifest
state. Since the manifest is PHP server is re-used across tests the
initial state could be unexpected. The tests handled this as necessary,
but meant that a variable number of sync XHRs would be issued. Since
sync XHRs log a deprecation warning the output was flaky. Fix this
by switching server control to async XHR.

Additionally, obsolete-error-events.html would time out if no update
was needed, so handle the 'noupdate' event too.

BUG= 663860 
R=jeffcarp@chromium.org

Review-Url: https://codereview.chromium.org/2540833002
Cr-Commit-Position: refs/heads/master@{#435450}

[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/RandomOrderExpectations
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/main-resource-redirect-expected.txt
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/main-resource-redirect.html
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/non-html-expected.txt
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/non-html.xhtml
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/obsolete-error-events-expected.txt
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/obsolete-error-events.html
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/remove-cache-expected.txt
[modify] https://crrev.com/1b6aabf1422f519dfe93fde0ba65b04c3a7a60a8/third_party/WebKit/LayoutTests/http/tests/appcache/remove-cache.html

Comment 6 by jsb...@chromium.org, Nov 30 2016

Status: fixed (was: Started)

Sign in to add a comment