Deterministic webkit_layout_test failures when using custom test ordering. |
|||||
Issue descriptionOn a local, tip of tree build of blink_tests, I've got a large swath of tests that deterministically fail. [macOS, but I expect this to apply to other platforms as well]. """ $ cat ./test_list html/details_summary/details-add-child-1.html external/wpt/cookies/samesite/form-post-blank.html $ third_party/blink/tools/run_web_tests.py -t gn --driver-logging --test-list=./test_list -j 1 --order=none """ The following tests all deterministically fail when substituted for the second test entry in "test_list". * external/wpt/cookies/prefix/__secure.header.https.html * external/wpt/cookies/samesite/fetch.html * external/wpt/cookies/samesite/form-get-blank.html * external/wpt/cookies/samesite/form-post-blank.html * external/wpt/cookies/samesite/iframe-reload.html * external/wpt/cookies/samesite/iframe.html * external/wpt/cookies/samesite/img.html * external/wpt/cookies/samesite/window-open.html * external/wpt/webxr/xrSession_cancelAnimationFrame.https.html * external/wpt/webxr/xrSession_cancelAnimationFrame_invalidhandle.https.html * external/wpt/webxr/xrSession_requestAnimationFrame_callback_calls.https.html
,
Sep 26
Raw test output: """ IN: 'http://web-platform.test:8001/cookies/samesite/form-post-blank.html\n OUT: Content-Type: text/plain\n OUT: This is a testharness.js-based test.\n OUT: PASS Same-host top-level form POSTs are strictly same-site\n OUT: PASS Subdomain top-level form POSTs are strictly same-site\n OUT: FAIL Cross-site top-level form POSTs are cross-site assert_equals: Non-SameSite cookies are always sent. expected (string) "0.369367262517307" but got (undefined) undefined\n OUT: PASS Same-host redirecting to same-host top-level form POSTs are strictly same-site\n OUT: PASS Subdomain redirecting to same-host top-level form POSTs are strictly same-site\n OUT: PASS Cross-site redirecting to same-host top-level form POSTs are strictly same-site\n OUT: PASS Same-host redirecting to subdomain top-level form POSTs are strictly same-site\n OUT: PASS Subdomain redirecting to subdomain top-level form POSTs are strictly same-site\n OUT: PASS Cross-site redirecting to subdomain top-level form POSTs are strictly same-site\n OUT: FAIL Same-host redirecting to cross-site top-level form POSTs are cross-site assert_equals: Non-SameSite cookies are always sent. expected (string) "0.6804303742279576" but got (undefined) undefined\n OUT: FAIL Subdomain redirecting to cross-site top-level form POSTs are cross-site assert_equals: Non-SameSite cookies are always sent. expected (string) "0.053074137036093205" but got (undefined) undefined\n OUT: FAIL Cross-site redirecting to cross-site top-level form POSTs are cross-site assert_equals: Non-SameSite cookies are always sent. expected (string) "0.2240315205355896" but got (undefined) undefined\n OUT: Harness: the test ran to completion.\n """
,
Sep 26
I've confirmed that these repro steps also result in deterministic failure on Linux.
,
Sep 26
I can reproduce the cookies test failures locally on Linux, but cannot reproduce the last three webxr failures. I compared the request logs of wptserve, and they are the same regardless of the execution order. It looks like content_shell fails to set Non-SameSite cookies when the cookies tests run after some html/ tests (including but not limited to html/details_summary/details-add-child-1.html) in the same process. (FWIW, the html/ tests in question never touch cookies; AFAICT, they are not leaking web-exposed states.) mkwst@ could you triage/delegate?
,
Sep 26
+andypaicu since mkwst is OOO
,
Sep 28
andypaicu -- ping. This is one of the blockers to deflaking webkit layout tests.
,
Sep 28
@erikchen - you should feel fully empowered to disable these tests if they're blocking the more general effort.
,
Dec 4
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by erikc...@chromium.org
, Sep 26