Here are the earliest examples I could find: https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty/builds/28853
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20Leak/builds/6606

My suspicion is that https://chromium-review.googlesource.com/461722 or https://chromium-review.googlesource.com/564681 is the cause of the flake.

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20ASAN/builds/3284 has the failure and is at revision 485433. Going to try to revert 485419.

Hard to revert. PTAL ASAP. Thanks!

Created revert of 485419
https://chromium-review.googlesource.com/c/566980/

I think we'll want to add more entries to TestExpectations. First thing to do is to list which tests are flaky/failing on which bots now.

Oh, wait, I didn't see that https://chromium-review.googlesource.com/c/566980/ was indeed successfully committed, which means "WPTServe: Run .any.js and .worker.js variations" is reverted.

Current failures may be due to other things. Either way, next step is still to find out what's failing where. At first glance, it looks like since 6:00 this morning PST hundreds of tests have been crashing, unrelated to the .any.js change.

Meanwhile, when we reland "WPTServe: Run .any.js and .worker.js variations" we may have to add some more test expectations.

Looking at recent layout test logs:

https://chromium-swarm.appspot.com/task?id=374aa1f7acbd9710&refresh=10&show_raw=1

Excerpt:

06:23:36.193 15400 Failed to start the content_shell process: 
[15409:15409:0711/062336.177935:5143837134:ERROR:fontconfig_util_linux.cc(88)] You are missing /usr/share/fonts/truetype/msttcorefonts/Arial.ttf. Try re-running build/install-build-deps.sh. Also see https://chromium.googlesource.com/chromium/src/+/master/docs/layout_tests_linux.md

So... since this morning, it appears that the linux machines that are running layout test swarming tasks are not configured to have all of the required fonts.

Update from discussion in chat:

The msttcorefonts are only installed if the Debian contrib package repo is enabled: https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh#482

This may be the case for some bots and not others; some shards are passing tests, whereas others are not: Example of passing shard: https://chromium-swarm.appspot.com/task?id=374b4b1c94feec10&refresh=10&show_raw=1

Is this perhaps related to https://chrome-internal-review.googlesource.com/c/407412/  ?

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/6497809a7da7836587f7b25b6d8e78b95d3df9e7

commit 6497809a7da7836587f7b25b6d8e78b95d3df9e7
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Tue Jul 11 19:07:45 2017

 Issue 741025  has been merged into this issue.

 Issue 741039  has been merged into this issue.

per chromium-dev, might have same cause as  bug 740995 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/291d972a020826050086e58b6080cf92c29f3ad6

commit 291d972a020826050086e58b6080cf92c29f3ad6
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Tue Jul 11 20:38:34 2017

It looks like we pushed a bad OS image to GCE last night, that didn't contain the needed fonts. images take 24 hours to roll out, so we wouldn't have seen failures everywhere yet, but the problem would get steadily worse.

We've reverted the bad image and are working to repair the existing machines now.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/a966d231b3eec4d042713327796843292fe973a8

commit a966d231b3eec4d042713327796843292fe973a8
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Tue Jul 11 20:55:19 2017

The ability to quarantine bots is very neat!

Are we expecting that when the new OS image rolls out, then those GCE instances/bots will be automatically unquarantined and will be used again?

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/de3d4643644095e5b84a69292a24bddd01d659eb

commit de3d4643644095e5b84a69292a24bddd01d659eb
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Tue Jul 11 21:21:02 2017

The linux builders on chromium.webkit are all green now (https://build.chromium.org/p/chromium.webkit/builders) and I believe that we're now no longer seeing this on other builders that run layout tests.

Thanks maruel@ and dpranke@ :-)

I think I am seeing similar tests failing again
https://uberchromegw.corp.google.com/i/chromium.webkit/builders/WebKit%20Linux%20Trusty%20Leak

Could it be related?

Maybe it could. Also, "avaialble" with owner doesn't make any sense.

Assigning back to maruel@ AND downgrading priority because I see green builds too.

There are still failures, for example:

Linux Trusty:
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty/builds/28944

Linux Trusty Leak:
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20Leak/builds/6669

Linux MSAN:
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20MSAN/builds/1923

Linux Debug:
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20%28dbg%29/builds/2826

 Issue 741577  has been merged into this issue.

 Issue 741563  has been merged into this issue.

 Issue 741253  has been merged into this issue.

 Issue 741187  has been merged into this issue.

I am still seeing a bunch of flaky tests from gfx_unittest due to the missing font.

I just saw another issue related to missing files from WebKit Linux Trusty MSAN. Some of media-related layout tests are crashing:

media/video-controls-visible-audio-only.html
media/controls-drag-timebar.html
media/W3C/video/events/event_playing_manual.html
media/W3C/audio/paused/paused_true_during_pause.html
media/media-document-audio-repaint.html
(...more)

---
07:42:08.322 29683   ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
07:42:08.322 29683   ALSA lib conf.c:4248:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
07:42:08.322 29683   ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
07:42:08.322 29683   ALSA lib conf.c:4248:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
07:42:08.322 29683   ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
07:42:08.322 29683   ALSA lib conf.c:4248:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
07:42:08.322 29683   ALSA lib conf.c:4727:(snd_config_expand) Evaluate error: No such file or directory
07:42:08.322 29683   ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default
07:42:08.322 29683   [10687:10720:0712/074207.157113:10197023447:WARNING:alsa_util.cc(24)] PcmOpen: default,No such file or directory
07:42:08.322 29683   ALSA lib confmisc.c:768:(parse_card) cannot find card '0'
07:42:08.322 29683   ALSA lib conf.c:4248:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
07:42:08.322 29683   ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
07:42:08.322 29683   ALSA lib conf.c:4248:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
07:42:08.322 29683   ALSA lib confmisc.c:1251:(snd_func_refer) error evaluating name
07:42:08.322 29683   ALSA lib conf.c:4248:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
07:42:08.322 29683   ALSA lib conf.c:4727:(snd_config_expand) Evaluate error: No such file or directory
07:42:08.322 29683   ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM default
---

Could it be a symptom of this issue?

I think this is something different.   My understanding there is no support for audio on the linux gce vm's.

So the failures in media tests on WebKit Linux Trusty MSAN could be a separate issue, and a separate bug could be filed for that.

Meanwhile, some of the failures noted by xidachen@ in #25 do look like the same issue. For example:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty/builds/28952
webkit_layout_tests shard #2:
04:07:00.811 17402 Failed to start the content_shell process: 
[17411:17411:0712/040700.796327:347357494:ERROR:fontconfig_util_linux.cc(88)] You are missing /usr/share/fonts/truetype/msttcorefonts/Arial.ttf. Try re-running build/install-build-deps.sh. Also see https://chromium.googlesource.com/chromium/src/+/master/docs/layout_tests_linux.md

Same failure as before.

So, some swarming bots are still affected. That failure was from about 5 hours ago.


Are we expecting all of the affected bots to be quarantined by now?

Note though that there are no failures since about 5 hours ago, so this might actually be OK now.

Relatedly I did see one other separate failure:  bug 741693 

The image that infra rolled back to takes ~24hrs to fully propagate, so lingering failures were to be expected.  If there are still further failures, then please reopen.

Awesome, thanks

 Issue 741766  has been merged into this issue.