Not much to go on here.  I agree that we need more logging, probably log each config that it's checking.

Yeah, after this happening like 5 times in a row I was unable to reproduce it in a test CL in about 6-7 tries. Let's wait for a bit and see if it happens again. Might be something that happens where there's more load.

Looks like this isn't just end2end tests.
Look at this WebGL failure, for example:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28Intel%20HD%20630%29/2450

No tests were run, browser failed to start 3 times.

Maybe the new driver / OS version we've upgraded to has problems?

CC'ing more people.

Unless there's a deeper problem with the browser starting reliably on Windows (which is possible), my first guess would be that it's a machine misconfiguration, like the IP KVM dongle becoming detached. In this case though:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28Intel%20HD%20630%29/2450

for the failing shard:

https://chromium-swarm.appspot.com/task?id=40a424330838e910&refresh=10&show_raw=1

I don't see a widespread problem on the affected bot:

https://chromium-swarm.appspot.com/bot?id=build144-a9&sort_stats=total%3Adesc

but maybe problems like this are being auto-detected and fixed by the Labs team?

It's unclear whether the failure mode of angle_end2end_tests is the same. More logging inside that test harness would help.

Assigning to this week's pixel wrangler to keep an eye on this. jmadill@, as this week's ANGLE wrangler, your help reporting any more flakes on the ANGLE CQ is appreciated.

Found one more case:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28Intel%20HD%20630%29/2345

2 cases in 200 builds is not so bad, actually.

Found one more case:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28Intel%20HD%20630%29/2345

2 cases in 200 builds is not so bad, actually.

OK. That shard's from October 10 (12 days ago) and the bot's healthy at this point:

https://chromium-swarm.appspot.com/bot?id=build133-a9&sort_stats=total%3Adesc

Popped up again here:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win_angle_rel_ng/3856

Thanks for the report. Here's the failing shard:
https://chromium-swarm.appspot.com/task?id=40cc50abc3cf9a10&refresh=10&show_raw=1

It doesn't look like anything's obviously wrong with that bot:
https://chromium-swarm.appspot.com/bot?id=build158-a9&sort_stats=total%3Adesc

although I didn't dig back far enough in its log to see the run that failed, and whether the auto-reboot that happens after it failed cleared up the failures.

Ria: as current pixel wrangler could you please reach out to jmadill@ and see whether you can figure out whether this is a transient failure that was fixed by the bot's rebooting? Try clicking "Show More Tasks" repeatedly on build158-a9's Swarming page. Thanks.

passing this to this week's pixel wrangler Wei

Happened again here:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win_angle_rel_ng/4274

Jamie, could you try to add more logging to ANGLE's tests to help diagnose this? The general pixel wranglers aren't having any success with this problem.

I didn't see this last week, but perhaps I wasn't looking hard enough. Jamie, is this still a problem?

There's a different failure mode now but it happens all the time.

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win-angle-rel/124
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win-angle-rel/122

A bunch more at https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win-angle-rel?limit=200 . TBH I'm not sure if this is the same root cause or the same as issue 923198 .

Doesn't look like issue 923198 to me.
No capacity problems, bots were assigned to tasks.

Looking at https://chromium-swarm.appspot.com/task?id=426b4cae91846b10&refresh=10&show_raw=1, what I see is no output, but:
Created	1/15/2019, 12:07:09 PM (Eastern Standard Time)
Started	1/15/2019, 12:25:38 PM (Eastern Standard Time)
Expires	1/15/2019, 1:07:09 PM (Eastern Standard Time)
Completed	1/15/2019, 12:46:57 PM (Eastern Standard Time)
Pending Time	18m 29s
Running Time	21m 17s

So, the task was doing something for 21 minutes and finished before expiry. However, we don't see any output.
maruel@, are there more logs somewhere which could bring more light to what's going on?

Have you tried running the task locally? See the instructions on the task's page.

I don't have a machine with same config readily available, but retrying the same task on Swarming succeeds without problem.
https://chromium-swarm.appspot.com/task?id=428f64aecc344e10&refresh=10&show_raw=1

Are there any logs of communication between the swarmed bot and MILO (or whatever is the thing that launches the swarmed tasks)?

I recommend a combination of 'Debug task' and http://go/swarming-ssh. You may want to try to pinpoint if this only happens to a subset of the bots, albeit both look fine:
https://chromium-swarm.appspot.com/bot?id=build118-a9
https://chromium-swarm.appspot.com/bot?id=build159-a9