mattm: could your change have increased this test's flakiness?

https://codereview.chromium.org/2838243002

I'm unable to reproduce the flake locally, including trying on an asan build :-\

No, I don't think the test actually touches any code that I changed.

Digging in a bit..
The test is failing on the results of X509Certificate::IsIssuedByEncoded calls. Those checks are supposed to just be verifying that the input data is as expected before running the actual test. They somehow say that client_1's issuer matches both kAuthority1DN and kAuthority2DN, and the same for client_2.pem.

But despite those failing, the actual meat of the test succeeds, with SelectClientCerts returning only the 1 expected certificate.

This is quite curious.

it starts spreading accross bots and becomes constantly failing

hmm, looks like it is still flaky, not constantly.

Linux ChromiumOS Tests (dbg)(1) is red but then green.

Also I overlooked #2, "I'm unable to reproduce the flake locally, including trying on an asan build :-\"

Let us leave it P1 though.

 Issue 716630  has been merged into this issue.

Note that this is failing on the linux_chromium_tsan_rel_ng and linux_chromium_asan_rel_ng tryservers, e.g.:

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_tsan_rel_ng/builds/63849
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_asan_rel_ng/builds/359830

See recent history:

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_tsan_rel_ng?numbuilds=200
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_asan_rel_ng?numbuilds=200

This is blocking CLs from going through the CQ. Upgrading to P0. Must be addressed urgently.

OK, it's not failing reliably on the ASAN/TSAN bots, only intermittently. Downgrading to P1 again.

I reverted https://codereview.chromium.org/2844963005 . So far so good on linux and linux dbg, but it's only been a cycle or too, which isn't enough that I'm confident. For the record I tried running the test locally, and couldn't get it to fail on tip of tree after 250 tries!

The revert did in make the tree go green.

I am not sure https://codereview.chromium.org/2844963005 is the root cause. Note that it is committed 2017-04-28 12:54:33 UTC. But I have already seen the same test failure before at Fri Apr 28 08:57:27 2017 here https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_asan_rel_ng/builds/359432. (If the time is in UTC(?))

Besides, the change https://codereview.chromium.org/2844963005 is only turned on in google3, and I believe the only test in chrome exercises the code should be net/spdy/core/lifo_write_scheduler_test.cc

I am not very familiar with how to run chrome test with asan and tsan. If you could point me the instructions, I will try to reproduce it locally, and debug the root cause. Thanks!

yasong, you are indeed right. I spoke too soon. net_unittests is still flakey. I will reland your patch and continue looking.

Moving discussion from 716630 to here:

> sky: how did you get findit to identify https://codereview.chromium.org/2844963005?
> I can't find a set of inputs to "Analyze flake" at http://findit-for-me.appspot.com/ that doesn't produce
> an error.

I didn't do anything special to trigger find-it. I believe find-it runs automatically. See comment #14, https://codereview.chromium.org/2844963005#msg14 .

> I also tried with a waterfall link, which suspected a different CL with a score of "N/A":
> https://findit-for-me.appspot.com/waterfall/failure?url=https%3A%2F%2Fbuild.chromium.org%2Fp%2Fchromium.chromiumos%2Fbuilders%2FLinux%2520ChromiumOS%2520Tests%2520%2528dbg%2529%25281%2529%2Fbuilds%2F25440

I created a revert of said patch here: https://codereview.chromium.org/2849193002/ . Lets hope that is the one.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da1b049fa7cad45f62965d461c5119a48775ab38

commit da1b049fa7cad45f62965d461c5119a48775ab38
Author: sky <sky@chromium.org>
Date: Mon May 01 18:04:02 2017

Revert "Do not abort redirect responses with unadvertised encoding."

This reverts commit 14d9d00fcdd545dc47387443e17c3d9fbb41e5fb.

Reverting in hopes of fixing 716594. See it for details.
I had to do this manually as codereview complained about file being
too big.

NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
R=eustas@chromium.org,holte@chromium.org,rdsmith@chromium.org,holte@chromium.org,mmenke@chromium.org
TBR=eustas@chromium.org,holte@chromium.org,rdsmith@chromium.org,holte@chromium.org,mmenke@chromium.org
BUG=716594

Review-Url: https://codereview.chromium.org/2849193002
Cr-Commit-Position: refs/heads/master@{#468353}

[modify] https://crrev.com/da1b049fa7cad45f62965d461c5119a48775ab38/net/http/http_network_transaction.cc
[modify] https://crrev.com/da1b049fa7cad45f62965d461c5119a48775ab38/net/http/http_network_transaction_unittest.cc
[modify] https://crrev.com/da1b049fa7cad45f62965d461c5119a48775ab38/tools/metrics/histograms/histograms.xml

 Issue 716730  has been merged into this issue.

Looking at the flakiness dashboard AFAICT this is the first failure: https://build.chromium.org/p/client.v8.fyi/builders/Linux%20Tests%20(dbg)(1)/builds/2946 . Which corresponds to
df319447d45c6280e70dc38ba916946ddc96a144, 44983. That's an FYI bot though and the revision doesn't line up, so I'm ignoring it.

Looking at the output of the linux test bots this seems to be the first failure: https://build.chromium.org/p/chromium.linux/builders/Linux%20Tests%20%28dbg%29%281%29%2832%29/builds/40596 . Which corresponds to c443c00c0c9e10b8ced1064d7da18c3a9668cac2 or 467971 . The patch I just reverted is after that, so I'll re-revert it again.

Matt, I think I'm going to try https://codereview.chromium.org/2838243002 next. Any other ideas?

I'm now trying a revert of https://codereview.chromium.org/2838243002 . Will hope that is it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/50576f3cd7be64cff426b733c39535cdf0379eb9

commit 50576f3cd7be64cff426b733c39535cdf0379eb9
Author: sky <sky@chromium.org>
Date: Mon May 01 19:28:03 2017

Revert "Revert "Do not abort redirect responses with unadvertised encoding.""

This reverts commit da1b049fa7cad45f62965d461c5119a48775ab38.

The original patch landed before the first flake, so I don't think
this is it. Reverting revert.

OPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
R=eustas@chromium.org,holte@chromium.org,rdsmith@chromium.org,holte@chromium.org,mmenke@chromium.org
TBR=eustas@chromium.org,holte@chromium.org,rdsmith@chromium.org,holte@chromium.org,mmenke@chromium.org
BUG=716594

Review-Url: https://codereview.chromium.org/2852123002
Cr-Commit-Position: refs/heads/master@{#468385}

[modify] https://crrev.com/50576f3cd7be64cff426b733c39535cdf0379eb9/net/http/http_network_transaction.cc
[modify] https://crrev.com/50576f3cd7be64cff426b733c39535cdf0379eb9/net/http/http_network_transaction_unittest.cc
[modify] https://crrev.com/50576f3cd7be64cff426b733c39535cdf0379eb9/tools/metrics/histograms/histograms.xml

https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Ozone%20Tests%20(1)/builds/46345

has the same failure at revision 467879, which is before  https://codereview.chromium.org/2838243002 (467901)

I wonder if infra updated NSS on the bots? I get a similar failure if I run the test against a locally built trunk version of NSS. I'll dig in more..

Matt, you are indeed right. It looks like the revert of https://codereview.chromium.org/2838243002 didn't land, so I don't need to revert it. I closed https://codereview.chromium.org/283824300 .

Thanks for investigating!

thestig disabled the test here: https://codereview.chromium.org/2850083002 .

Removing Sheriff-Chromium as the test was disabled so it should no longer show up on the waterfall/cq.

This bug's status is started, but it has no owner.  Assigning to sky who last updated the status

I'm not a good owner for this. Passing to mattm.

Hi,all
I have met this problems too, I have found that this unittest failed in ubuntu which is virtual machine but success in the real ubuntu.
I have no idea why this problem happened, it is some solutions to solve it?