Many *idlharness* WPT tests fail on chromium.webkit/WebKit Linux Trusty MSAN |
||||||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of grt@chromium.org webkit_layout_tests failing on chromium.webkit/WebKit Linux Trusty MSAN Builders failed on: - WebKit Linux Trusty MSAN: https://ci.chromium.org/buildbot/chromium.webkit/WebKit%20Linux%20Trusty%20MSAN
,
Jun 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/06e083bda350bb8c39dd2d54e46748db0154ce07 commit 06e083bda350bb8c39dd2d54e46748db0154ce07 Author: Greg Thompson <grt@chromium.org> Date: Tue Jun 26 13:30:55 2018 Mark various layout tests as timing out on Linux. BUG=856601 TBR=petermayo@chromium.org Change-Id: I82d31d27ab3acecca5dbd345127bd0290e54f933 Reviewed-on: https://chromium-review.googlesource.com/1114841 Commit-Queue: Greg Thompson <grt@chromium.org> Reviewed-by: Peter Mayo <petermayo@chromium.org> Reviewed-by: Greg Thompson <grt@chromium.org> Cr-Commit-Position: refs/heads/master@{#570393} [modify] https://crrev.com/06e083bda350bb8c39dd2d54e46748db0154ce07/third_party/WebKit/LayoutTests/TestExpectations
,
Jun 28 2018
,
Jul 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e3589ef074fe15fbdd4310ac39788502ada70aa7 commit e3589ef074fe15fbdd4310ac39788502ada70aa7 Author: Takashi Sakamoto <tasak@google.com> Date: Mon Jul 09 11:41:26 2018 [Gardening] mark various layout tests as timed out on Linux. WebKit Linux Trusty MSAN Build: * external/wpt/FileAPI/idlharness.worker.html * external/wpt/bluetooth/idl/idlharness.tentative.window.html * external/wpt/css/cssom-view/interfaces.html * external/wpt/mediacapture-streams/MediaDevices-IDL-enumerateDevices.html * external/wpt/page-visibility/idlharness.html * external/wpt/touch-events/idlharness.window.html * external/wpt/web-animations/interfaces/Animation/idlharness.html * http/tests/devtools/network/network-filters.js TBR=pmonette@chromium.org Bug: 856601 Change-Id: I8f3fb802b935fa1c321a89c46b7d63a801e8c2c4 No-Try: true Reviewed-on: https://chromium-review.googlesource.com/1128836 Reviewed-by: Takashi Sakamoto <tasak@google.com> Commit-Queue: Takashi Sakamoto <tasak@google.com> Cr-Commit-Position: refs/heads/master@{#573273} [modify] https://crrev.com/e3589ef074fe15fbdd4310ac39788502ada70aa7/third_party/WebKit/LayoutTests/TestExpectations
,
Jul 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8a49ff6a3074257bdf00a33468d909ce0007bbff commit 8a49ff6a3074257bdf00a33468d909ce0007bbff Author: Vasilii Sukhanov <vasilii@chromium.org> Date: Mon Jul 09 13:59:38 2018 Mark http/tests/event-timing/event-timing-retrievability.html as timing out TBR=pmonette@chromium.org Bug: 856601 Change-Id: Idac2114ffb81e7e02acf75b8b30f194cfa51c6a1 Reviewed-on: https://chromium-review.googlesource.com/1128967 Reviewed-by: Vasilii Sukhanov <vasilii@chromium.org> Cr-Commit-Position: refs/heads/master@{#573294} [modify] https://crrev.com/8a49ff6a3074257bdf00a33468d909ce0007bbff/third_party/WebKit/LayoutTests/TestExpectations
,
Jul 12
Revising summary. Most of the failing tests are *idlharness* or *interface* tests. If you search TestExpectations you see many entries, starting around 2018-06-26. It's hard to tell when these started because MSAN isn't on the Flakiness Dashboard. I don't see changes upstream to WPT that could have caused the regression. +foolip: any ideas?
,
Jul 12
Issue 862588 has been merged into this issue.
,
Jul 12
Sorry, my WPT checkout just wasn't updated. There are a lot of WPT IDL changes from Luke in recent days that align with these test failures. Luke: Can you take a look please?
,
Jul 12
Surprised to see these timing out; starting with FileAPI, we can at least parallelize the fetches and clean up the tests a little; https://github.com/web-platform-tests/wpt/pull/11957 We'll see if that improves things. Otherwise it might just need to be a long timeout, if `new File` and `new Blob` are just dang slow. Will look at the others soon.
,
Jul 12
Chiming in to add some more context:
The MSAN/ASAN/dbg bots are indeed much slower than release bots. If a test takes multiple seconds (very close to timeout) on normal bots, the test is likely to time out on MSAN/ASAN/dbg bots, in which case the recommended solution is to first speed up the test and perhaps give it a long timeout (by adding a meta timeout tag), which will also reduce the potential flakiness of the test on release bots.
In rare cases, a test might run very fast (<1s) on release bots but still time out on slower bots. We can then add specific expectations for the test in {ASAN,MSAN}Expectations.
,
Jul 12
Seems like a common denominator here is that these files are pulling in the `html.idl` dependency, which is a whopping 2500 lines or so. Might need to flag those tests as long timeout, or come up with a fancy ".idl" test type that generates a test with the timeout META tag, but we'll try with parallel fetches first.
,
Jul 12
The complication here is that we can't at run time increase the timeout, so it's not as simple as adding an if to idlharness.js. Besides, there are a LOT of idlharness tests that depend on dom/html. This wasn't an issue before because many of these tests added ad-hoc (usually empty) dependent interfaces instead of pulling in the dependencies properly. We suspect parsing dom.idl and html.idl is computationally expensive and might suffer from MSAN/ASAN overhead more than other tests, but we'd like to get the low-hanging fruits first (making fetches of IDL files parallel) and see if it helps.
,
Jul 12
Update: I've sifted through the expectations (https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/TestExpectations?type=cs&q=file:testexpectations&sq=package:chromium&g=0&l=1) to ensure that all of the tests tagged with Timeout have been modified to have parallel fetches already, or in a pending PR. filtered list: /bluetooth/idl/idlharness.tentative.window.html /cookie-store/idlharness.tentative.https.html /css/cssom-view/interfaces.html /dom/nodes/Element-classlist.html /fetch/cors-rfc1918/idlharness.tentative.any.html /FileAPI/idlharness.html /FileAPI/idlharness.worker.html /gamepad/idlharness.html /IndexedDB/interfaces.any.html /mediacapture-record/idlharness.html /mediacapture-streams/idlharness.https.window.html /page-visibility/idlharness.html /payment-request/idlharness.https.window.html /presentation-api/controlling-ua/idlharness.https.html /push-api/idlharness.https.any.sharedworker.html /push-api/idlharness.https.any.worker.html /resource-timing/idlharness.any.html /resource-timing/idlharness.any.worker.html /selection/interfaces.html /service-workers/service-worker/interfaces-sw.https.html /touch-events/idlharness.window.html /wpt/mediacapture-streams/MediaDevices-IDL-enumerateDevices.html
,
Jul 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6f03c4c03992a2a99f1ef3f829bafc6ed4ea1572 commit 6f03c4c03992a2a99f1ef3f829bafc6ed4ea1572 Author: Robert Ma <robertma@chromium.org> Date: Fri Jul 27 22:28:25 2018 Move MSAN-specific expectations to MSANExpectations They don't need to (and shouldn't) be in the bloated TestExpectations. Most of these are related to idlharness.js and their corresponding bugs have been merged to crbug.com/856601, so update the bug links accordingly. Besides, two expectations were incorrectly removed in an import https://crrev.com/c/1152529; they are also added back in this CL. Bug: 856601 Change-Id: Idf3a58cd3a4b5ae9f149bbff7b5150bfda3ed645 Reviewed-on: https://chromium-review.googlesource.com/1153615 Commit-Queue: Philip Jägenstedt <foolip@chromium.org> Reviewed-by: Philip Jägenstedt <foolip@chromium.org> Cr-Commit-Position: refs/heads/master@{#578831} [modify] https://crrev.com/6f03c4c03992a2a99f1ef3f829bafc6ed4ea1572/third_party/WebKit/LayoutTests/MSANExpectations [modify] https://crrev.com/6f03c4c03992a2a99f1ef3f829bafc6ed4ea1572/third_party/WebKit/LayoutTests/TestExpectations
,
Jul 30
What is the recommended way to fix these? It looks like we need to merge more bugs/update more expectations: issue 866965 issue 810963 issue 819778
,
Jul 30
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a6a03569dea7b9baeb4ea58895ca57e7e7811aa4 commit a6a03569dea7b9baeb4ea58895ca57e7e7811aa4 Author: Kenichi Ishibashi <bashi@chromium.org> Date: Mon Jul 30 02:14:59 2018 Move some S13nServiceWorker related test expectations to MSANExpectations These tests are timing out only on msan bots. Move these entries to MSANExpectations so that we can make sure these tests pass on linux bots. Bug: 866965 , 856601 Change-Id: I48d15a00a508c989865ebba2fa88987ddcd46c80 Reviewed-on: https://chromium-review.googlesource.com/1154609 Reviewed-by: Matt Falkenhagen <falken@chromium.org> Commit-Queue: Kenichi Ishibashi <bashi@chromium.org> Cr-Commit-Position: refs/heads/master@{#578964} [modify] https://crrev.com/a6a03569dea7b9baeb4ea58895ca57e7e7811aa4/third_party/WebKit/LayoutTests/MSANExpectations [modify] https://crrev.com/a6a03569dea7b9baeb4ea58895ca57e7e7811aa4/third_party/WebKit/LayoutTests/TestExpectations
,
Aug 1
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f3d5d112714f7b12bad5dc23820d89b6cfc141e1 commit f3d5d112714f7b12bad5dc23820d89b6cfc141e1 Author: Christos Froussios <cfroussios@chromium.org> Date: Wed Aug 01 07:34:44 2018 Disable external/wpt/IndexedDB/idlharness.any.sharedworker.html on MSAN It is mostly failing (timeout) Bug: 856601 No-try: true No-Presubmit: true Change-Id: Ic8785e8bf5237d4e252c94a48a00302ce6e0303a Reviewed-on: https://chromium-review.googlesource.com/1158144 Commit-Queue: Christos Froussios <cfroussios@chromium.org> Reviewed-by: Christos Froussios <cfroussios@chromium.org> Cr-Commit-Position: refs/heads/master@{#579719} [modify] https://crrev.com/f3d5d112714f7b12bad5dc23820d89b6cfc141e1/third_party/WebKit/LayoutTests/MSANExpectations
,
Aug 1
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d9ce625fa18367381d40a7c876fb1b7acd96ad7a commit d9ce625fa18367381d40a7c876fb1b7acd96ad7a Author: Christos Froussios <cfroussios@chromium.org> Date: Wed Aug 01 08:07:38 2018 Disable several *idlharness* tests They are flaking with timeout Bug: 856601 No-try: true No-Presubmit: true Change-Id: I40e09e6b8aed3cb2014a284d012abc8122d614b7 Reviewed-on: https://chromium-review.googlesource.com/1158228 Reviewed-by: Christos Froussios <cfroussios@chromium.org> Commit-Queue: Christos Froussios <cfroussios@chromium.org> Cr-Commit-Position: refs/heads/master@{#579723} [modify] https://crrev.com/d9ce625fa18367381d40a7c876fb1b7acd96ad7a/third_party/WebKit/LayoutTests/MSANExpectations
,
Aug 1
OK we need to take this more seriously. Does anyone know what kind of workload would be affected the most on MSAN builds? CPU-bound or IO-bound? And how do we enable MSAN locally to reproduce the flakes?
,
Aug 10
,
Aug 10
Issue 872731 has been merged into this issue.
,
Aug 10
I've detected 5 new tests that have timed out multiple times in the past 14 runs: * external/wpt/css/css-typed-om/interfaces.html [x11] * external/wpt/mediacapture-streams/idlharness.https.window.html [x5] * external/wpt/notifications/idlharness.https.any.html [x3] * external/wpt/push-api/idlharness.https.any.html [x2] * external/wpt/speech-api/idlharness.window.html [x6] Adding disables for these 5 (see Issue 873045 ).
,
Aug 10
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/67c441d2c33570fc281cef35a61169ad41a1a76b commit 67c441d2c33570fc281cef35a61169ad41a1a76b Author: Matt Giuca <mgiuca@chromium.org> Date: Fri Aug 10 05:42:52 2018 WebKit Layout Tests: Disable a lot of flaky tests on MSAN. Adds a bunch of tests with high-probability flake due to timeout or crash, to existing lists of very similar tests that were already disabled. Tbr: mgiuca@chromium.org Bug: 873045 , 838057 , 856601, 869818 Change-Id: I66e14994ef4551d5690d3a66d51fce84a1eaf456 No-Presubmit: true No-Tree-Checks: true No-Try: true Reviewed-on: https://chromium-review.googlesource.com/1170450 Reviewed-by: Matt Giuca <mgiuca@chromium.org> Commit-Queue: Matt Giuca <mgiuca@chromium.org> Cr-Commit-Position: refs/heads/master@{#582068} [modify] https://crrev.com/67c441d2c33570fc281cef35a61169ad41a1a76b/third_party/WebKit/LayoutTests/MSANExpectations
,
Aug 13
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/67d786b7e1c30e47ce0c0093c60709abdd6e75f2 commit 67d786b7e1c30e47ce0c0093c60709abdd6e75f2 Author: Makoto Shimazu <shimazu@chromium.org> Date: Mon Aug 13 07:08:11 2018 Mark more idlharness.js tests on MSan as slow TBR=robertma@chromium.org NOTRY=true Bug: 856601 Change-Id: I8527c9d7c1df382f08620378ac9931634cfdd5aa Reviewed-on: https://chromium-review.googlesource.com/1172226 Reviewed-by: Makoto Shimazu <shimazu@chromium.org> Commit-Queue: Makoto Shimazu <shimazu@chromium.org> Cr-Commit-Position: refs/heads/master@{#582519} [modify] https://crrev.com/67d786b7e1c30e47ce0c0093c60709abdd6e75f2/third_party/WebKit/LayoutTests/MSANExpectations
,
Aug 14
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a1a1d10c0e71ad349a47c3eeea15fe7ff603a30e commit a1a1d10c0e71ad349a47c3eeea15fe7ff603a30e Author: Makoto Shimazu <shimazu@chromium.org> Date: Tue Aug 14 07:33:28 2018 Mark more idlharness tests as [ Timeout Pass ] on MSan bots TBR=robertma@chromium.org NOTRY=true Bug: 856601 Change-Id: Iad0f3b236494cc6d1f14af0fc03518ebf3e0671a Reviewed-on: https://chromium-review.googlesource.com/1174144 Reviewed-by: Makoto Shimazu <shimazu@chromium.org> Commit-Queue: Makoto Shimazu <shimazu@chromium.org> Cr-Commit-Position: refs/heads/master@{#582856} [modify] https://crrev.com/a1a1d10c0e71ad349a47c3eeea15fe7ff603a30e/third_party/WebKit/LayoutTests/MSANExpectations
,
Aug 17
What's a progress on this bug? So many tests have been timing out, although they are non-deterministic (i.e., the tests are just slow). It doesn't really look productive to mark tests as [Timeout] one by one every time we find them... :/
,
Sep 19
,
Sep 19
Pawning this off to Robert who had some thoughts about working out what's slow (in an MSAN build).
,
Sep 21
,
Sep 21
,
Sep 28
We're still seeing timeouts on several tests. Especially the first one: * external/wpt/push-api/idlharness.https.any.serviceworker.html * external/wpt/BackgroundSync/interfaces.https.any.html * external/wpt/IndexedDB/idlharness.any.worker.html * external/wpt/notifications/idlharness.https.any.serviceworker.html * external/wpt/BackgroundSync/interfaces.https.any.html * external/wpt/IndexedDB/idlharness.any.worker.html * external/wpt/gamepad/idlharness.window.html https://ci.chromium.org/buildbot/chromium.webkit/WebKit%20Linux%20Trusty%20MSAN/
,
Oct 3
,
Oct 5
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f8b83bcb698c1fbd01e784b11198e3dcc6cefa64 commit f8b83bcb698c1fbd01e784b11198e3dcc6cefa64 Author: Marijn Kruisselbrink <mek@chromium.org> Date: Fri Oct 05 21:48:28 2018 Mark more idlharness tests as [ Timeout Pass ] on MSan bots. Tbr: robertma@chromium.org Bug: 856601 Change-Id: Ib6b0b9c13c89de9637222601d0830cef7bcaea8f Reviewed-on: https://chromium-review.googlesource.com/c/1265561 Reviewed-by: Marijn Kruisselbrink <mek@chromium.org> Commit-Queue: Marijn Kruisselbrink <mek@chromium.org> Cr-Commit-Position: refs/heads/master@{#597320} [modify] https://crrev.com/f8b83bcb698c1fbd01e784b11198e3dcc6cefa64/third_party/WebKit/LayoutTests/MSANExpectations
,
Nov 6
Ping robertma@chromium.org, from your friendly neighbourhood ecosystem infra rotation :-)
,
Nov 15
wpt/css/filter-effects/interfaces.any.html is also timeout on Linux Trusty MSAN https://ci.chromium.org/buildbot/chromium.webkit/WebKit%20Linux%20Trusty%20MSAN/10984
,
Nov 15
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a4b02559ebf271f64a4d26c4e41765f657eac2d6 commit a4b02559ebf271f64a4d26c4e41765f657eac2d6 Author: Ella Ge <eirage@chromium.org> Date: Thu Nov 15 20:08:55 2018 filter-effects/interfaces.any.html failed on MSAN add MSAN expectation TBR=robertma@chromium.org Bug: 856601 Change-Id: Idb002c64071199228cb2b5881e1e773de0325fbc Reviewed-on: https://chromium-review.googlesource.com/c/1338299 Reviewed-by: Ella Ge <eirage@chromium.org> Reviewed-by: Robert Ma <robertma@chromium.org> Commit-Queue: Ella Ge <eirage@chromium.org> Cr-Commit-Position: refs/heads/master@{#608485} [modify] https://crrev.com/a4b02559ebf271f64a4d26c4e41765f657eac2d6/third_party/WebKit/LayoutTests/MSANExpectations
,
Nov 15
Kyle will take over the investigation.
,
Nov 16
Issue 906211 has been merged into this issue.
,
Nov 22
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5a2a7a91adc9e7c97d867a098fa315d2b2bef7b1 commit 5a2a7a91adc9e7c97d867a098fa315d2b2bef7b1 Author: Anders Hartvoll Ruud <andruud@chromium.org> Date: Thu Nov 22 11:06:48 2018 Flaky on MSAN: http/tests/devtools/elements/elements-save-to-temp-var.js. TBR=kyleju@chromium.org Bug: 856601 Change-Id: I4ab0da613ba282fab128206a91805593952ba0ff Reviewed-on: https://chromium-review.googlesource.com/c/1347362 Reviewed-by: Anders Hartvoll Ruud <andruud@chromium.org> Commit-Queue: Anders Hartvoll Ruud <andruud@chromium.org> Cr-Commit-Position: refs/heads/master@{#610375} [modify] https://crrev.com/5a2a7a91adc9e7c97d867a098fa315d2b2bef7b1/third_party/WebKit/LayoutTests/MSANExpectations
,
Dec 14
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by grt@chromium.org
, Jun 26 2018Labels: Stability-Memory-MemorySanitizer OS-Linux Type-Bug