Your first list of failures looks wrong - in particular, the two SystemNetworkContext/NetworkContextConfigurationFtpPacBrowserTest.FtpPac tests failed on the second run, but not on the first.

It looks to me like a lot of those tests are using the Python SpawnedTestServer instead of the EmbeddedTestServer.

All FTP tests use it (PDFExtensionTest.OpenFromFTP, SystemNetworkContext/NetworkContextConfigurationFtpPacBrowserTest.FtpPac/*, CrossOriginXHR.ContentScript), ComponentCloudPolicyTest.FetchExtensionPolicy looks to use it, DevToolsSanityTest.TestNetworkSize looks to use it.

Not going to look at the other tests, but given that, in general, most tests are using the embedded test server (Or should be), it seems likely the issue is the Python test server.  Repeatedly running into flake with starting the server is one of the reasons we switched to a C++ test server in most tests.

Thanks for triaging mmenke@. Who can own this? Is it straightforward to switch these tests to use the embedded test server rather than the Python based one? I think I've run into this elsewhere while trying to change the virtual Python environment in Issue 804174, so if we could just get rid of the spawned Python server that would be awesome.

Often no, the embedded test server doesn't support FTP, websockets, OCSP - and I believe there are other features as well.  Tests that just need HTTP/HTTPS should pretty much all be using the EmbeddedTestServer already.  Fixing this would be a pretty major undertaking.  A quarter?  Two?  No idea.

We can't really have someone tackle this as a one-off, would need management buy-in.

These tests seem like low flakiness at 0.1% to 1%.
One interesting observation: in most of the cases these tests are either still FAIL after 3 retries or PASS at 1st retry.

7-day data on only win7_chromium_rel_ng:
https://screenshot.googleplex.com/zfrYNzjJWtO.png
https://screenshot.googleplex.com/9TBKZcQBSpA.png

7-day data on ALL CQ builders:
https://screenshot.googleplex.com/LKLeZf8cFjS.png
https://screenshot.googleplex.com/LF4Qjqj8Bcn.png

Looks like some of the DevToolsExtensionTests subclass a test framework that starts the spawned test server, but don't actually use it (i.e., these are timing out due to running something they don't seem to even use), though a lot of other tests sharing the same framework do use the spawned test server.  Those might be able to be moved off of it.

Wonder if the second failure rate depends on how many tests failed in the first try.  Do we retry tests in parallel or in sequence?  Could imagine if we make certain assumptions about browser test CPU performance, we could not have enough resources to start the python test server in parallel X times at once.

For gtest, retry is in sequence https://chromium.googlesource.com/chromium/src/+/2d00d828c596faaef0eab092764c1ef9f69b1219/base/test/launcher/unit_test_launcher.cc#719

This is likely a duplicate of issue 804174.  We can reopen it if needed, but don't think we need to investigate further unless it continues to appear.