New issue
Advanced search Search tips

Issue 754365 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

testharness.js update broke many tests (assert_throws became stricter)

Project Member Reported by atotic@chromium.org, Aug 10 2017

Issue description

The test-harness update at https://codereview.chromium.org/2891463002
has updated many tests with expectations of failure such as:

crash-expected.txt
>>>> 
FAIL Do not crash when exceptions are thrown during keyframe iteration. Test bug: need to pass exception to assert_throws()
<<<<

These tests are no longer being run properly, and no longer test what they are supposed to. We should fix them.

The root cause is that test-harness.js has changed the way 
assert_throws works.
 
assert_throws(code, func, description)

Passing "null" for a code used to be acceptable. Now the code must be "the expected exception".

Fixing each test involves running the test, figuring out what the
thrown exception is, and fixing the assert_throws call. For example:

assert_throws(null, .....

becomes:

assert_throws({name: "TypeError"}, .......

The tests affected are:

third_party/WebKit/LayoutTests/animations/keyframe-iteration-exception-crash-expected.txt
third_party/WebKit/LayoutTests/bindings/dictionary-iterator-expected.txt
third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothDevice-expected.txt
third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothGATTRemoteServer-expected.txt
third_party/WebKit/LayoutTests/custom-elements/constructor-context-dies-cross-context-call-expected.txt
third_party/WebKit/LayoutTests/custom-elements/v0-v1-interop-expected.txt
third_party/WebKit/LayoutTests/external/wpt/content-security-policy/securitypolicyviolation/idl-expected.txt
third_party/WebKit/LayoutTests/external/wpt/credential-management/idl.https-expected.txt
third_party/WebKit/LayoutTests/external/wpt/cssom/interfaces-expected.txt
third_party/WebKit/LayoutTests/external/wpt/dom/interfaces-expected.txt
third_party/WebKit/LayoutTests/external/wpt/domxpath/interfaces-expected.txt
third_party/WebKit/LayoutTests/external/wpt/fetch/api/headers/headers-idl-expected.txt
third_party/WebKit/LayoutTests/external/wpt/fetch/api/request/request-idl-expected.txt
third_party/WebKit/LayoutTests/external/wpt/fetch/api/response/response-idl-expected.txt
third_party/WebKit/LayoutTests/external/wpt/IndexedDB/interleaved-cursors-expected.txt
third_party/WebKit/LayoutTests/external/wpt/media-source/interfaces-expected.txt
third_party/WebKit/LayoutTests/external/wpt/mediacapture-fromelement/idlharness-expected.txt
third_party/WebKit/LayoutTests/external/wpt/mediacapture-record/idlharness-expected.txt
third_party/WebKit/LayoutTests/external/wpt/mediacapture-streams/MediaStreamTrack-init.https-expected.txt
third_party/WebKit/LayoutTests/external/wpt/mediasession/idlharness-expected.txt
third_party/WebKit/LayoutTests/external/wpt/orientation-event/idlharness-expected.txt
third_party/WebKit/LayoutTests/external/wpt/payment-request/payment-request-show-method.https-expected.txt
third_party/WebKit/LayoutTests/external/wpt/resources/idlharness.js
third_party/WebKit/LayoutTests/external/wpt/resources/testharness.js
third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/fetch-request-fallback.https-expected.txt
third_party/WebKit/LayoutTests/external/wpt/service-workers/service-worker/register-link-element.https-expected.txt
third_party/WebKit/LayoutTests/external/wpt/svg/interfaces-expected.txt
third_party/WebKit/LayoutTests/external/wpt/url/interfaces-expected.txt
third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/Animation/idlharness-expected.txt
third_party/WebKit/LayoutTests/external/wpt/webrtc/datachannel-idlharness-expected.txt
third_party/WebKit/LayoutTests/external/wpt/webrtc/interfaces-expected.txt
third_party/WebKit/LayoutTests/external/wpt/webrtc/RTCPeerConnection-idl-expected.txt
third_party/WebKit/LayoutTests/external/wpt/webstorage/idlharness-expected.txt
third_party/WebKit/LayoutTests/external/wpt/webvtt/api/interfaces-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-2d-imageData-create-nonfinite-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-arc-negative-radius-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-currentTransform-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-ellipse-negative-radius-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-getContext-invalid-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-getImageData-invalid-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-hit-regions-fill-rule-test-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-ImageData-constructor-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-isPointInPath-winding-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-lineDash-input-sequence-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-lineDash-invalid-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-overloads-drawImage-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-overloads-fillText-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-overloads-strokeText-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-path-addpath-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-path-context-clip-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-path-context-fill-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-path-context-stroke-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-pattern-behaviour-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-putImageData-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/canvas-with-incorrect-args-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/draw-focus-if-needed-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/draw-focus-if-needed-with-path2d-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/drawImage-with-broken-image-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/drawImage-with-valid-image-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/gradient-addColorStop-with-invalid-color-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/gradient-addColorStop-with-invalid-offset-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/linearGradient-infinite-values-expected.txt
third_party/WebKit/LayoutTests/fast/canvas/radialGradient-infinite-values-expected.txt
third_party/WebKit/LayoutTests/fast/dom/custom/custom-elements-registry-expected.txt
third_party/WebKit/LayoutTests/fast/dom/dataset-expected.txt
third_party/WebKit/LayoutTests/fast/dom/lenient-this-expected.txt
third_party/WebKit/LayoutTests/fast/dom/Window/window-constructor-expected.txt
third_party/WebKit/LayoutTests/fast/dom/Window/window-methods-construct-expected.txt
third_party/WebKit/LayoutTests/fast/encoding/api/sharedarraybuffer-expected.txt
third_party/WebKit/LayoutTests/fast/js/put-forwards-expected.txt
third_party/WebKit/LayoutTests/fast/mediarecorder/BlobEvent-basic-expected.txt
third_party/WebKit/LayoutTests/fast/serviceworker/serviceworker-interface-expected.txt
third_party/WebKit/LayoutTests/fast/serviceworker/serviceworkercontainer-interface-expected.txt
third_party/WebKit/LayoutTests/fast/xmlhttprequest/xmlhttprequest-send-sharedarraybuffer-expected.txt
third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetch-click-event.https-expected.txt
third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetch-event.https-expected.txt
third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetch-fail-event.https-expected.txt
third_party/WebKit/LayoutTests/http/tests/background_fetch/background-fetched-event.https-expected.txt
third_party/WebKit/LayoutTests/http/tests/notifications/notification-properties-expected.txt
third_party/WebKit/LayoutTests/http/tests/notifications/serviceworker-notification-properties-expected.txt
third_party/WebKit/LayoutTests/http/tests/push_messaging/pushmessagedata-expected.txt
third_party/WebKit/LayoutTests/http/tests/w3c/webperf/approved/navigation-timing/html/idlharness-expected.txt
third_party/WebKit/LayoutTests/mojo/codec-expected.txt
third_party/WebKit/LayoutTests/mojo/union-expected.txt
third_party/WebKit/LayoutTests/nfc/idl-NFC-expected.txt
third_party/WebKit/LayoutTests/payments/payment-request-interface-expected.txt
third_party/WebKit/LayoutTests/platform/mac-retina/external/wpt/IndexedDB/interleaved-cursors-expected.txt
third_party/WebKit/LayoutTests/platform/mac/http/tests/security/w3c/cross-origin-objects-expected.txt
third_party/WebKit/LayoutTests/platform/win7/http/tests/security/w3c/cross-origin-objects-expected.txt
third_party/WebKit/LayoutTests/platform/win7/virtual/mojo-loading/http/tests/security/w3c/cross-origin-objects-expected.txt
third_party/WebKit/LayoutTests/resize-observer/observe-expected.txt
third_party/WebKit/LayoutTests/resources/idlharness.js
third_party/WebKit/LayoutTests/resources/testharness.js
third_party/WebKit/LayoutTests/storage/indexeddb/observer-expected.txt
third_party/WebKit/LayoutTests/storage/indexeddb/observer-worker-expected.txt
third_party/WebKit/LayoutTests/svg/dom/SVGLength-calc-in-attr-expected.txt
third_party/WebKit/LayoutTests/svg/dom/svglength-units-expected.txt
third_party/WebKit/LayoutTests/svg/dom/SVGLength-viewport-units-expected.txt
third_party/WebKit/LayoutTests/TestExpectations
third_party/WebKit/LayoutTests/typedcssom/computedstyle/computedStylePropertyMap-expected.txt
third_party/WebKit/LayoutTests/typedcssom/cssCalcLength-expected.txt
third_party/WebKit/LayoutTests/typedcssom/cssMatrixComponent-expected.txt
third_party/WebKit/LayoutTests/typedcssom/cssPerspective-expected.txt
third_party/WebKit/LayoutTests/typedcssom/cssScale-expected.txt
third_party/WebKit/LayoutTests/typedcssom/cssSimpleLength-expected.txt
third_party/WebKit/LayoutTests/typedcssom/cssSkew-expected.txt
third_party/WebKit/LayoutTests/typedcssom/cssTranslation-expected.txt
third_party/WebKit/LayoutTests/virtual/mojo-loading/http/tests/security/w3c/cross-origin-objects-expected.txt


 
Cc: qyears...@chromium.org
Components: Blink>Infra
Owner: ----
Status: Available (was: Untriaged)
Summary: testharness.js update broke many tests (assert_throws became stricter) (was: test-harness update has broken many tests)
Thanks for filing this!

Burn down list:

https://docs.google.com/spreadsheets/d/1N3el2Xx2aL-qL6dMojPRlV-2qGY0fcrJIshzSoHBcxw/edit#gid=0
Project Member

Comment 2 by bugdroid1@chromium.org, May 14 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/18a8183507dabb63503b4ef20f7c41d3042dfc83

commit 18a8183507dabb63503b4ef20f7c41d3042dfc83
Author: Chris Nardi <cnardi@chromium.org>
Date: Mon May 14 21:42:38 2018

Add expected exceptions to assert_throws statements

Some assert_throws statements did not include expected exceptions, which
broke their respective tests following a testharness.js update. Add
expected exceptions for all assert_throws statements. The only
functional change was to fast/serviceworker/serviceworker-interface.html
as per the spec ServiceWorker extends EventTarget and not Worker.

Bug:  754365 
Change-Id: I733605d721085788d43e179ca6f9a7c7b25da995
Reviewed-on: https://chromium-review.googlesource.com/1056253
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Chris Nardi <cnardi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558476}
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/bindings/dictionary-iterator-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/bindings/dictionary-iterator.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothGATTRemoteServer-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/bluetooth/idl/idl-BluetoothGATTRemoteServer.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/custom-elements/constructor-context-dies-cross-context-call-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/custom-elements/constructor-context-dies-cross-context-call.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/custom-elements/v0-v1-interop-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/custom-elements/v0-v1-interop.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/canvas/canvas-putImageData-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/canvas/canvas-putImageData.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/Window/window-constructor-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/Window/window-constructor.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/Window/window-methods-construct-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/Window/window-methods-construct.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/custom/custom-elements-registry-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/custom/custom-elements-registry.html
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/dataset-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/dataset.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/dom/lenient-this-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/dom/lenient-this.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/mediarecorder/BlobEvent-basic-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/mediarecorder/BlobEvent-basic.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworker-interface-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworker-interface.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworkercontainer-interface-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/fast/serviceworker/serviceworkercontainer-interface.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/http/tests/notifications/notification-properties-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/http/tests/notifications/notification-properties.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/http/tests/notifications/serviceworker-notification-properties-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/http/tests/notifications/serviceworker-notification-properties.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/http/tests/push_messaging/pushmessagedata-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/http/tests/push_messaging/resources/pushmessagedata-worker.js
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/mojo/codec-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/mojo/codec.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/mojo/union-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/mojo/union.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/nfc/idl-NFC-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/nfc/idl-NFC.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/platform/linux/http/tests/security/w3c/cross-origin-objects-expected.txt
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/platform/mac/http/tests/security/w3c/cross-origin-objects-expected.txt
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/platform/win7/http/tests/security/w3c/cross-origin-objects-expected.txt
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/storage/indexeddb/observer-expected.txt
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/storage/indexeddb/observer-worker-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/storage/indexeddb/resources/observer.js
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/svg/dom/SVGLength-calc-in-attr-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/svg/dom/SVGLength-calc-in-attr.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/svg/dom/SVGLength-viewport-units-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/svg/dom/SVGLength-viewport-units.html
[delete] https://crrev.com/ea78046d0b4669359d4830b5920be8d9fa7baf8c/third_party/WebKit/LayoutTests/svg/dom/svglength-units-expected.txt
[modify] https://crrev.com/18a8183507dabb63503b4ef20f7c41d3042dfc83/third_party/WebKit/LayoutTests/svg/dom/svglength-units.html

Comment 3 by cnardi@chromium.org, May 14 2018

Owner: cnardi@chromium.org
Status: Fixed (was: Available)
There should no longer be any of these left, verified through searching for " Test bug: need to pass exception to assert_throws()" in -expected.txt files.

Sign in to add a comment