testharness.js update broke many tests (assert_throws became stricter) |
||
Issue descriptionThe test-harness update at https://codereview.chromium.org/2891463002 has updated many tests with expectations of failure such as: crash-expected.txt >>>> FAIL Do not crash when exceptions are thrown during keyframe iteration. Test bug: need to pass exception to assert_throws() <<<< These tests are no longer being run properly, and no longer test what they are supposed to. We should fix them. The root cause is that test-harness.js has changed the way assert_throws works. assert_throws(code, func, description) Passing "null" for a code used to be acceptable. Now the code must be "the expected exception". Fixing each test involves running the test, figuring out what the thrown exception is, and fixing the assert_throws call. For example: assert_throws(null, ..... becomes: assert_throws({name: "TypeError"}, ....... The tests affected are: third_party/WebKit/LayoutTests/animations/keyframe-iteration-exception-crash-expected.txt third_party/WebKit/LayoutTests/bindings/dictionary-iterator-expected.txt third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothDevice-expected.txt third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothGATTRemoteServer-expected.txt third_party/WebKit/LayoutTests/custom-elements/constructor-context-dies-cross-context-call-expected.txt third_party/WebKit/LayoutTests/custom-elements/v0-v1-interop-expected.txt third_party/WebKit/LayoutTests/external/wpt/content-security-policy/securitypolicyviolation/idl-expected.txt third_party/WebKit/LayoutTests/external/wpt/credential-management/idl.https-expected.txt third_party/WebKit/LayoutTests/external/wpt/cssom/interfaces-expected.txt third_party/WebKit/LayoutTests/external/wpt/dom/interfaces-expected.txt third_party/WebKit/LayoutTests/external/wpt/domxpath/interfaces-expected.txt third_party/WebKit/LayoutTests/external/wpt/fetch/api/headers/headers-idl-expected.txt third_party/WebKit/LayoutTests/external/wpt/fetch/api/request/request-idl-expected.txt third_party/WebKit/LayoutTests/external/wpt/fetch/api/response/response-idl-expected.txt third_party/WebKit/LayoutTests/external/wpt/IndexedDB/interleaved-cursors-expected.txt third_party/WebKit/LayoutTests/external/wpt/media-source/interfaces-expected.txt third_party/WebKit/LayoutTests/external/wpt/mediacapture-fromelement/idlharness-expected.txt third_party/WebKit/LayoutTests/external/wpt/mediacapture-record/idlharness-expected.txt third_party/WebKit/LayoutTests/external/wpt/mediacapture-streams/MediaStreamTrack-init.https-expected.txt third_party/WebKit/LayoutTests/external/wpt/mediasession/idlharness-expected.txt third_party/WebKit/LayoutTests/external/wpt/orientation-event/idlharness-expected.txt third_party/WebKit/LayoutTests/external/wpt/payment-request/payment-request-show-method.https-expected.txt third_party/WebKit/LayoutTests/external/wpt/resources/idlharness.js third_party/WebKit/LayoutTests/external/wpt/resources/testharness.js third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/fetch-request-fallback.https-expected.txt third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/register-link-element.https-expected.txt third_party/WebKit/LayoutTests/external/wpt/svg/interfaces-expected.txt third_party/WebKit/LayoutTests/external/wpt/url/interfaces-expected.txt third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/Animation/idlharness-expected.txt third_party/WebKit/LayoutTests/external/wpt/webrtc/datachannel-idlharness-expected.txt third_party/WebKit/LayoutTests/external/wpt/webrtc/interfaces-expected.txt third_party/WebKit/LayoutTests/external/wpt/webrtc/RTCPeerConnection-idl-expected.txt third_party/WebKit/LayoutTests/external/wpt/webstorage/idlharness-expected.txt third_party/WebKit/LayoutTests/external/wpt/webvtt/api/interfaces-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-2d-imageData-create-nonfinite-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-arc-negative-radius-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-currentTransform-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-ellipse-negative-radius-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-getContext-invalid-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-getImageData-invalid-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-hit-regions-fill-rule-test-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-ImageData-constructor-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-isPointInPath-winding-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-lineDash-input-sequence-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-lineDash-invalid-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-overloads-drawImage-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-overloads-fillText-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-overloads-strokeText-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-path-addpath-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-path-context-clip-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-path-context-fill-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-path-context-stroke-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-pattern-behaviour-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-putImageData-expected.txt third_party/WebKit/LayoutTests/fast/canvas/canvas-with-incorrect-args-expected.txt third_party/WebKit/LayoutTests/fast/canvas/draw-focus-if-needed-expected.txt third_party/WebKit/LayoutTests/fast/canvas/draw-focus-if-needed-with-path2d-expected.txt third_party/WebKit/LayoutTests/fast/canvas/drawImage-with-broken-image-expected.txt third_party/WebKit/LayoutTests/fast/canvas/drawImage-with-valid-image-expected.txt third_party/WebKit/LayoutTests/fast/canvas/gradient-addColorStop-with-invalid-color-expected.txt third_party/WebKit/LayoutTests/fast/canvas/gradient-addColorStop-with-invalid-offset-expected.txt third_party/WebKit/LayoutTests/fast/canvas/linearGradient-infinite-values-expected.txt third_party/WebKit/LayoutTests/fast/canvas/radialGradient-infinite-values-expected.txt third_party/WebKit/LayoutTests/fast/dom/custom/custom-elements-registry-expected.txt third_party/WebKit/LayoutTests/fast/dom/dataset-expected.txt third_party/WebKit/LayoutTests/fast/dom/lenient-this-expected.txt third_party/WebKit/LayoutTests/fast/dom/Window/window-constructor-expected.txt third_party/WebKit/LayoutTests/fast/dom/Window/window-methods-construct-expected.txt third_party/WebKit/LayoutTests/fast/encoding/api/sharedarraybuffer-expected.txt third_party/WebKit/LayoutTests/fast/js/put-forwards-expected.txt third_party/WebKit/LayoutTests/fast/mediarecorder/BlobEvent-basic-expected.txt third_party/WebKit/LayoutTests/fast/serviceworker/serviceworker-interface-expected.txt third_party/WebKit/LayoutTests/fast/serviceworker/serviceworkercontainer-interface-expected.txt third_party/WebKit/LayoutTests/fast/xmlhttprequest/xmlhttprequest-send-sharedarraybuffer-expected.txt third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetch-click-event.https-expected.txt third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetch-event.https-expected.txt third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetch-fail-event.https-expected.txt third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetched-event.https-expected.txt third_party/WebKit/LayoutTests/http/tests/notifications/notification-properties-expected.txt third_party/WebKit/LayoutTests/http/tests/notifications/serviceworker-notification-properties-expected.txt third_party/WebKit/LayoutTests/http/tests/push_messaging/pushmessagedata-expected.txt third_party/WebKit/LayoutTests/http/tests/w3c/webperf/approved/navigation-timing/html/idlharness-expected.txt third_party/WebKit/LayoutTests/mojo/codec-expected.txt third_party/WebKit/LayoutTests/mojo/union-expected.txt third_party/WebKit/LayoutTests/nfc/idl-NFC-expected.txt third_party/WebKit/LayoutTests/payments/payment-request-interface-expected.txt third_party/WebKit/LayoutTests/platform/mac-retina/external/wpt/IndexedDB/interleaved-cursors-expected.txt third_party/WebKit/LayoutTests/platform/mac/http/tests/security/w3c/cross-origin-objects-expected.txt third_party/WebKit/LayoutTests/platform/win7/http/tests/security/w3c/cross-origin-objects-expected.txt third_party/WebKit/LayoutTests/platform/win7/virtual/mojo-loading/http/tests/security/w3c/cross-origin-objects-expected.txt third_party/WebKit/LayoutTests/resize-observer/observe-expected.txt third_party/WebKit/LayoutTests/resources/idlharness.js third_party/WebKit/LayoutTests/resources/testharness.js third_party/WebKit/LayoutTests/storage/indexeddb/observer-expected.txt third_party/WebKit/LayoutTests/storage/indexeddb/observer-worker-expected.txt third_party/WebKit/LayoutTests/svg/dom/SVGLength-calc-in-attr-expected.txt third_party/WebKit/LayoutTests/svg/dom/svglength-units-expected.txt third_party/WebKit/LayoutTests/svg/dom/SVGLength-viewport-units-expected.txt third_party/WebKit/LayoutTests/TestExpectations third_party/WebKit/LayoutTests/typedcssom/computedstyle/computedStylePropertyMap-expected.txt third_party/WebKit/LayoutTests/typedcssom/cssCalcLength-expected.txt third_party/WebKit/LayoutTests/typedcssom/cssMatrixComponent-expected.txt third_party/WebKit/LayoutTests/typedcssom/cssPerspective-expected.txt third_party/WebKit/LayoutTests/typedcssom/cssScale-expected.txt third_party/WebKit/LayoutTests/typedcssom/cssSimpleLength-expected.txt third_party/WebKit/LayoutTests/typedcssom/cssSkew-expected.txt third_party/WebKit/LayoutTests/typedcssom/cssTranslation-expected.txt third_party/WebKit/LayoutTests/virtual/mojo-loading/http/tests/security/w3c/cross-origin-objects-expected.txt
,
May 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/18a8183507dabb63503b4ef20f7c41d3042dfc83 commit 18a8183507dabb63503b4ef20f7c41d3042dfc83 Author: Chris Nardi <cnardi@chromium.org> Date: Mon May 14 21:42:38 2018 Add expected exceptions to assert_throws statements Some assert_throws statements did not include expected exceptions, which broke their respective tests following a testharness.js update. Add expected exceptions for all assert_throws statements. The only functional change was to fast/serviceworker/serviceworker-interface.html as per the spec ServiceWorker extends EventTarget and not Worker. Bug: 754365 Change-Id: I733605d721085788d43e179ca6f9a7c7b25da995 Reviewed-on: https://chromium-review.googlesource.com/1056253 Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Commit-Queue: Chris Nardi <cnardi@chromium.org> Cr-Commit-Position: refs/heads/master@{#558476} [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/bindings/dictionary-iterator-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/bindings/dictionary-iterator.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothGATTRemoteServer-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothGATTRemoteServer.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/custom-elements/constructor-context-dies-cross-context-call-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/custom-elements/constructor-context-dies-cross-context-call.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/custom-elements/v0-v1-interop-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/custom-elements/v0-v1-interop.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/canvas/canvas-putImageData-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/canvas/canvas-putImageData.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/Window/window-constructor-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/Window/window-constructor.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/Window/window-methods-construct-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/Window/window-methods-construct.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/custom/custom-elements-registry-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/custom/custom-elements-registry.html [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/dataset-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/dataset.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/lenient-this-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/lenient-this.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/mediarecorder/BlobEvent-basic-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/mediarecorder/BlobEvent-basic.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworker-interface-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworker-interface.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworkercontainer-interface-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworkercontainer-interface.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/http/tests/notifications/notification-properties-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/http/tests/notifications/notification-properties.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/http/tests/notifications/serviceworker-notification-properties-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/http/tests/notifications/serviceworker-notification-properties.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/http/tests/push_messaging/pushmessagedata-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/http/tests/push_messaging/resources/pushmessagedata-worker.js [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/mojo/codec-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/mojo/codec.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/mojo/union-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/mojo/union.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/nfc/idl-NFC-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/nfc/idl-NFC.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/platform/linux/http/tests/security/w3c/cross-origin-objects-expected.txt [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/platform/mac/http/tests/security/w3c/cross-origin-objects-expected.txt [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/platform/win7/http/tests/security/w3c/cross-origin-objects-expected.txt [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/storage/indexeddb/observer-expected.txt [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/storage/indexeddb/observer-worker-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/storage/indexeddb/resources/observer.js [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/svg/dom/SVGLength-calc-in-attr-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/svg/dom/SVGLength-calc-in-attr.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/svg/dom/SVGLength-viewport-units-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/svg/dom/SVGLength-viewport-units.html [delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/svg/dom/svglength-units-expected.txt [modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/svg/dom/svglength-units.html
,
May 14 2018
There should no longer be any of these left, verified through searching for " Test bug: need to pass exception to assert_throws()" in -expected.txt files. |
||
►
Sign in to add a comment |
||
Comment 1 by qyears...@chromium.org
, Aug 10 2017Components: Blink>Infra
Owner: ----
Status: Available (was: Untriaged)
Summary: testharness.js update broke many tests (assert_throws became stricter) (was: test-harness update has broken many tests)