The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/44c2aec5bbd487a98728635f32e8f3c3d6935768

commit 44c2aec5bbd487a98728635f32e8f3c3d6935768
Author: Tim 'mithro' Ansell <tansell@chromium.org>
Date: Tue Aug 22 03:36:15 2017

LayoutTests: Disabling WebCryptoAPI tests imported for wpt.

These tests are flaky on Mac / Windows and are failing everywhere.

BUG=655458, 721814 , 757698 ,709227
TBR=qyearsley@chromium.org,dpranke@chromium.org,jeffcarp@chromium.org
NOTRY=true

Change-Id: Ia0ae3d6ee71751bf667c110e732c557309c21737
Reviewed-on: https://chromium-review.googlesource.com/624770
Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org>
Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#496200}
[modify] https://crrev.com/44c2aec5bbd487a98728635f32e8f3c3d6935768/third_party/WebKit/LayoutTests/TestExpectations

Do you have a link for where someone can view why they're flaky? That is, whether it's a timeout or something else?

These tests shouldn't be flaky, and my gut feels that it'd be something with the harness, but I don't want to claim that without additional data :)

Well, the first step would be to make them passing. They are flakily going from failing to timing out / crashing.

Try looking at https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=webkit_layout_tests&tests=WebCryptoAPI and https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=webkit_tests&tests=WebCryptoAPI

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/48bdfc57f7a71076ae35788dd18b27d3c8439a9f

commit 48bdfc57f7a71076ae35788dd18b27d3c8439a9f
Author: Tim 'mithro' Ansell <tansell@chromium.org>
Date: Tue Aug 22 08:32:46 2017

LayoutTests: Disable WebCryptoAPI missed from last CL.

XXX.worker.html verse XXX.https.html

Failures -> https://build.chromium.org/p/chromium.linux/builders/Linux%20Tests/builds/61159

BUG=655458, 721814 , 757698 ,709227
TBR=qyearsley@chromium.org,dpranke@chromium.org,jeffcarp@chromium.org
NOTRY=true

Change-Id: I3cf8864d8a5c8af379322b25298e0e7ebdc6b356
Reviewed-on: https://chromium-review.googlesource.com/625638
Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org>
Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#496251}
[modify] https://crrev.com/48bdfc57f7a71076ae35788dd18b27d3c8439a9f/third_party/WebKit/LayoutTests/TestExpectations

Upping the priority on this one, lots of tests are disabled, this needs to be triaged. Adding sheriff label until we see it's green, since last change(s) to TestExpectations contained errors.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a581405cdb71ba301a6694c03c90ba730541816d

commit a581405cdb71ba301a6694c03c90ba730541816d
Author: Henrik Boström <hbos@chromium.org>
Date: Tue Aug 22 10:31:40 2017

Fix incorrect paths in WebCrpytoAPI TestExpectations

BUG=655458, 721814 , 757698 ,709227
TBR=qyearsley@chromium.org,dpranke@chromium.org,jeffcarp@chromium.org
NOTRY=true

Change-Id: I05c9db91a8e104d871c1f82aec6129066518259a
Reviewed-on: https://chromium-review.googlesource.com/625517
Commit-Queue: Henrik Boström <hbos@chromium.org>
Reviewed-by: Henrik Boström <hbos@chromium.org>
Cr-Commit-Position: refs/heads/master@{#496271}
[modify] https://crrev.com/a581405cdb71ba301a6694c03c90ba730541816d/third_party/WebKit/LayoutTests/TestExpectations

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b9e0702f4f6561a8fd1eec1684c8cda906576091

commit b9e0702f4f6561a8fd1eec1684c8cda906576091
Author: Henrik Boström <hbos@chromium.org>
Date: Tue Aug 22 11:20:48 2017

Revert "Fix incorrect paths in WebCrpytoAPI TestExpectations"

This reverts commit a581405cdb71ba301a6694c03c90ba730541816d.

Reason for revert: Might have accidentally added/removed tests, and we want to Skip some of these, not Pass/Failure.

Original change's description:
> Fix incorrect paths in WebCrpytoAPI TestExpectations
> 
> BUG=655458, 721814 , 757698 ,709227
> TBR=qyearsley@chromium.org,dpranke@chromium.org,jeffcarp@chromium.org
> NOTRY=true
> 
> Change-Id: I05c9db91a8e104d871c1f82aec6129066518259a
> Reviewed-on: https://chromium-review.googlesource.com/625517
> Commit-Queue: Henrik Boström <hbos@chromium.org>
> Reviewed-by: Henrik Boström <hbos@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#496271}

TBR=hbos@chromium.org

Change-Id: Id3c86fc509f644fdc3bc6a196650c1c3cd157fa9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 655458,  721814 ,  757698 , 709227
Reviewed-on: https://chromium-review.googlesource.com/625996
Reviewed-by: Henrik Boström <hbos@chromium.org>
Reviewed-by: Tim 'mithro' Ansell <tansell@chromium.org>
Commit-Queue: Henrik Boström <hbos@chromium.org>
Cr-Commit-Position: refs/heads/master@{#496279}
[modify] https://crrev.com/b9e0702f4f6561a8fd1eec1684c8cda906576091/third_party/WebKit/LayoutTests/TestExpectations

I verified locally that the tests pass after reverting the above CL. We should remove the Sheriff-Chromium label once all bots are green.

Clarification: the run passes because it skips the failing tests, but the tests still needs to be fixed and re-enabled.

Removing Sheriff label; bots are all green now (or at least the webkit_tests step is passing on all of them)

Issue still needs an owner -- rsleevi, can you triage?

From https://storage.googleapis.com/chromium-layout-test-archives/WebKit_Win7__dbg_/10592/layout-test-results/results.html, and failures.js, it looks like crypto.subtle is undefined, but I don't know enough about the state of the API to know if that's an issue with the WPT test harness, a change to the spec, or a bug in the implementation.

Removing from Owner and moving back to triage queue.

We previously had issues with WPT because WebCrypto is secure-origin restricted.

From looking at https://storage.googleapis.com/chromium-layout-test-archives/WebKit_Win7__dbg_/10592/layout-test-results/results.html , these are all running on HTTP.

From looking at WPT master, these have been set to run on HTTPS for the past four months - https://github.com/w3c/web-platform-tests/tree/master/WebCryptoAPI/generateKey

Do you have any idea why our import might not be running them on HTTPS? I would expect crypto.subtle to be missing in this case, but surprised it's limited to just generateKey, apparently.

I've start to investigate how can we execute WebCrypto test in https.

Hyungwook told me about this bug during TPAC. I believe WPT HTTPS tests are supported in Blink. And from wpt.fyi, Chrome runs the tests fine (there are some failures in some tests cases, which is not a problem). Perhaps we just need to rebaseline.

rsleevi@ & andypaicu@:

I saw the change from HTTP to HTTPS happened at https://codereview.chromium.org/2838603002 (andypaicu was the author). However, that CL only renamed a bunch of html tests and did not touch the worker tests (*.worker.js, wpt runner automatically generates the corresponding *.html boilerplate for these). It's the worker tests that are failing because of undefined crypto.subtle.

I'm not familiar with the background, but I assume service workers also need to be served via HTTPS to be considered as secure context, which means we should also rename *.worker.js to *.https.worker.js. Is that correct?

Thanks for looking Robert!

Comment #14 sounds correct to me:
*.worker.js should be renamed to *.https.worker.js

Thanks for the confirmation!

Decided to fix the issue (along with some other minor issues in the WebCryptoAPI folder) in the upstream: https://github.com/w3c/web-platform-tests/pull/8284

I'll re-enable these tests once the upstream PR is merged and imported to Chromium.

The PR has landed.

Here's the CL to re-enable the test: https://chromium-review.googlesource.com/c/chromium/src/+/775146

I'm not sure who should review it, requesting review from eroman@ for now. Everyone is welcome to jump in. :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/212d1bc46c89111f15b4e373f4257f6356a4784f

commit 212d1bc46c89111f15b4e373f4257f6356a4784f
Author: Robert Ma <robertma@chromium.org>
Date: Mon Nov 20 21:58:50 2017

Re-enable WebCryptoAPI WPT (generateKey)

They are fixed in the upstream:
https://github.com/w3c/web-platform-tests/pull/8284

Despite a few flaky timeouts, most tests should now pass.

Bug:  757698 
Change-Id: I453c4b06db36526067003fecbdbf2949f3a42e59
Reviewed-on: https://chromium-review.googlesource.com/775146
Reviewed-by: Eric Roman <eroman@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517941}
[modify] https://crrev.com/212d1bc46c89111f15b4e373f4257f6356a4784f/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/212d1bc46c89111f15b4e373f4257f6356a4784f/third_party/WebKit/LayoutTests/external/wpt/WebCryptoAPI/idlharness-expected.txt
[delete] https://crrev.com/5ef1225bd132d42f2a7be4126fe215fd7ecc447d/third_party/WebKit/LayoutTests/external/wpt/WebCryptoAPI/idlharness.https-expected.txt

Hi WebCrypto folks,

I've fixed the infra issue in the CL above. Now there are currently four remaining tests that are flaky (pass / timeout):

TestExpectations:

 crbug.com/757698  external/wpt/WebCryptoAPI/generateKey/successes_RSA-OAEP.https.worker.html [ Pass Timeout ]
 crbug.com/757698  external/wpt/WebCryptoAPI/generateKey/successes_RSA-PSS.https.worker.html [ Pass Timeout ]
 crbug.com/757698  external/wpt/WebCryptoAPI/generateKey/successes_RSASSA-PKCS1-v1_5.https.worker.html [ Pass Timeout ]
 crbug.com/757698  external/wpt/WebCryptoAPI/generateKey/test_successes_RSA-OAEP.https.html [ Pass Timeout ]

Note (to sheriffs): other test_successes_RSA*.https.html tests might also be slow. If they turn out flaky, please add flaky expectations similar to the ones above.


From what I understand, these cryptographic operations are slow by their nature. Not sure if it's possible to adjust/split the tests to make them pass reliably, or if we have to increase the timeout. I'll leave this question to crypto experts :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bc247f7b0fb5439c9494ef1a21312731a76e5a7e

commit bc247f7b0fb5439c9494ef1a21312731a76e5a7e
Author: Robert Ma <robertma@chromium.org>
Date: Wed Nov 22 23:14:57 2017

Mark all RSA key generation tests as flaky timeout

And use a new dedicated issue for them.

TBR=eroman@chromium.org

Bug:  757698 ,  787971 
Change-Id: If51835eb88cf12ba2a16fe655aa2b54d2df97f2e
Reviewed-on: https://chromium-review.googlesource.com/786261
Reviewed-by: Robert Ma <robertma@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#518791}
[modify] https://crrev.com/bc247f7b0fb5439c9494ef1a21312731a76e5a7e/third_party/WebKit/LayoutTests/TestExpectations

Actually, I opened a separate  bug 787971  to track the slowness/timeout issue of RSA key generation, and changed TestExpectations accordingly (see above). Let's close this bug.