Attaching flakliness dashboard and failure log for future reference.

Deleted: Screen Shot 2018-07-02 at 11.29.12 AM.png 667 KB

	Screen Shot 2018-07-02 at 11.29.12 AM.png 667 KB View Download

Deleted: failure_Win.log 2.9 KB

failure_Win.log 2.9 KB View Download

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/db5145c73cb09ab9348aadf424dd86c2dfbb5245

commit db5145c73cb09ab9348aadf424dd86c2dfbb5245
Author: Luciano Pacheco <lucmult@chromium.org>
Date: Mon Jul 02 02:24:33 2018

Disable flaky test IntermediateFromAia404

Test has been flaky on Win builders.

Tbr: eroman@chromium.org
Bug:  859387 
Change-Id: I57bdda3903c30e06f05fe89d367f94429bb4e31a
Reviewed-on: https://chromium-review.googlesource.com/1121917
Reviewed-by: Luciano Pacheco <lucmult@chromium.org>
Commit-Queue: Luciano Pacheco <lucmult@chromium.org>
Cr-Commit-Position: refs/heads/master@{#571806}
[modify] https://crrev.com/db5145c73cb09ab9348aadf424dd86c2dfbb5245/net/cert/cert_verify_proc_unittest.cc

 Issue 859295  has been merged into this issue.

Summary: 

The test "CertVerifyProcInternalWithNetFetchingTest.IntermediateFromAia404/CertVerifyProcWin" is flakin (on Windows).

The test verifies a leaf certificate, without an intermediate having been given. The leaf includes an AIA URL for the intermediate, however fetching this URL gives a 404.

The test is expected to fail with ERR_CERT_AUTHORITY_INVALID, however sometimes fails with ERR_CERT_INVALID instead.

The failure log shows that an attempt to retrieve the intermediate was made to the EmbeddedTestServer.

Interestingly, the related test "IntermediateFromAia200" is passing on Windows. IntermediateFromAia200 is the same as IntermediateFromAia404, however instead of returning a 404 for the AIA fetch, it returns the expected intermediate certificate's DER.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/876f79ed479cfcb8be6441fd017a8b54a4b465d8

commit 876f79ed479cfcb8be6441fd017a8b54a4b465d8
Author: Olivier Robin <olivierrobin@chromium.org>
Date: Thu Jul 05 08:18:44 2018

Disable IntermediateFromAia404

Test is flaky on iOS

Bug:  859387 , 860189
Tbr: eroman@chromium.org
Change-Id: Ibc2dbd55dff9b0bfa02c7bd99c97aa9d9693dbd2
Reviewed-on: https://chromium-review.googlesource.com/1126762
Reviewed-by: Olivier Robin <olivierrobin@chromium.org>
Commit-Queue: Olivier Robin <olivierrobin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#572735}
[modify] https://crrev.com/876f79ed479cfcb8be6441fd017a8b54a4b465d8/net/cert/cert_verify_proc_unittest.cc

Regarding the Windows flakes:

I haven't been able to reproduce the Windows flakes locally.
I can reproduce flakiness using the trybots, so was able to run some experiments. However those did not pinpoint why the flakes are happening:

  * Using an AIA URL that is a bogus scheme and path (file://xxx) flakes between ERR_CERT_INVALID and ERR_CERT_AUTHORITY_INVALID on Windows trybots

  * Using an AIA URL that is a consistently unreachable http://0.0.0.0:9/xxx URL (flakes between ERR_CERT_INVALID and ERR_CERT_AUTHORITY_INVALID on Windows.

  * Using an AIA URL that responds consistently with invalid DER flakes between ERR_CERT_INVALID and ERR_CERT_AUTHORITY_INVALID on Windows.


The only (apparently) non-flaky test parameters I have been able to craft was:

  * Using an AIA URL that responds with the correct certificate, HOWEVER is served with a 404 (not found) HTTP status code consistently gives ERR_CERT_INVALID on Windows.

I can't explain why this particular combination would give a non-flaky results whereas the other variations flake. It could be that this is also flaky, but simply did not flake for my number of test runs.

Also amusingly, this last combination passes on macOS, which apparently accepts certificates from 404 responses....

My ability to trace the code from trybot results is limited. But I can say that the Windows code does an initial attempt with strong signing enable and it fails with:

CERT_TRUST_IS_OFFLINE_REVOCATION | CERT_TRUST_REVOCATION_STATUS_UNKNOWN | CERT_TRUST_IS_NOT_SIGNATURE_VALID

CertVerifyProcWin considers the presence of either CERT_TRUST_IS_NOT_SIGNATURE_VALID or CERT_TRUST_HAS_WEAK_SIGNATURE as sufficient to retry without strong signing. It does that, but the results is the same.

When mapping errors, CERT_TRUST_IS_NOT_SIGNATURE_VALID without CERT_TRUST_HAS_WEAK_SIGNATURE is mapped to ERR_CERT_INVALID.




A final note on testing process: in the checked in code, there is some non-determinism in the certificate chains used - they use randomized subjects, serial numbers, AIA urls, and SPKIs (to thwart other caching issues). I accounted for this in my experiments, and verified that it is not these randomized parameters causing the flakiness.


Debugging this further doesn't seem like time well spent - I will resolve this by changing the test and/or expectations on Windows to account for this flakiness.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3eb19bed582661462758b60a7a084737688247aa

commit 3eb19bed582661462758b60a7a084737688247aa
Author: Eric Roman <eroman@chromium.org>
Date: Fri Jul 06 22:52:27 2018

Re-enable IntermediateFromAia404 on Windows.

The cause of the flakiness is not completely understood, however test expectations have been changed to accept both ways in which the verification may fail.

Bug:  859387 
Change-Id: I585d1e4721e311b88888b34ac9fa7ce1ba8169be
Reviewed-on: https://chromium-review.googlesource.com/1128261
Reviewed-by: Matt Mueller <mattm@chromium.org>
Commit-Queue: Eric Roman <eroman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#573104}
[modify] https://crrev.com/3eb19bed582661462758b60a7a084737688247aa/net/cert/cert_verify_proc_unittest.cc