It looks like the failed run has to do with the failed bot lacking libwayland-egl.so.1 in its system image.

Possibly related to https://chromium-review.googlesource.com/c/chromium/src/+/1005567 . Will look more deeply.

Actually, this seems more like a bot issue that it doesn't have wayland, and I think the first failure is before my patch landed. Also, my patch didn't actually change any required libraries. I don't see wayland in the list of packages in build/install-build-deps.sh. Maybe that's the issue? I've no idea why this would suddenly be an issue.

 Issue 831699  has been merged into this issue.

Assigning to oncall infra trooper hzl@

Actually, ChromeOS shouldn't currently depend upon wayland-client. (it likely will soon, but not at this time). Will keep poking.

Would you please take a look at this issue related to build/install-build-deps.sh?

I'll take this.

I went back a month and ChromeOS had dependencies on this:
//content/test:content_browsertests --[private]-->
//content/test:test_support --[private]-->
//ui/ozone:ozone --[public]-->
//ui/ozone:platform --[private]-->
//ui/ozone/platform/wayland:wayland

This is because of this line: https://chromium.googlesource.com/chromium/src/+/master/ui/ozone/ozone.gni#57

So does anybody own the action item of ensuring that bots actually have Wayland library installed?

hzl: did you find something that indicated that libwayland is supposed to be installed by install-build-deps?

As far as I can tell it's missing from the sysroot image?

+hzl re #14

https://chromium-review.googlesource.com/c/chromium/src/+/1008420 adds wayland to the install-build-deps. If the bots automatically pick this up, then we should be good. I'm not familiar with the sysroot images.

> As far as I can tell it's missing from the sysroot image?

The library has been in the sysroot images for a while but apparently not on the swarming images.  Those must be regenerated and pushed to all the bots.  +friedman could you update the images once https://chromium-review.googlesource.com/c/chromium/src/+/1008420 lands?

Also I think we should revert the cl that's causing this because linux-chromeos-rel is gray on all my CLs and I can't land anything.

re #14, I got that inspiration from #6. 

+friedman

Including @friedman into the loop, re #18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dd7e237ba9b69855adf60e8141b7dc45c613b6a7

commit dd7e237ba9b69855adf60e8141b7dc45c613b6a7
Author: Scott Violet <sky@chromium.org>
Date: Thu Apr 12 01:11:26 2018

Adds wayland-egl to the list of required libraries

Apparently this has been required for a while now on ChromeOS. It's
not really required on non-ChromeOS, but no doubt it will soon, so
adding to both.

BUG= 831700 
TEST=none

Change-Id: I6f859ff4767582c6c2a7155c995347d1963ceebd
Reviewed-on: https://chromium-review.googlesource.com/1008420
Reviewed-by: David Reveman <reveman@chromium.org>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Scott Violet <sky@chromium.org>
Cr-Commit-Position: refs/heads/master@{#549987}
[modify] https://crrev.com/dd7e237ba9b69855adf60e8141b7dc45c613b6a7/build/install-build-deps.sh

Kicking image creation now.

thomasanderson, I'm not sure which patch triggered this failure to start happening. I couldn't find anything that changed recently with regards to new deps on wayland.

The image is ready to be canaried, but jbudorick's done some investigation into the existing failing images that suggests that something else might be more likely as the root cause. We're still looking into it.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/05aa89df2d88781bff56cbc80c2551a53143ab93

commit 05aa89df2d88781bff56cbc80c2551a53143ab93
Author: smut <smut@google.com>
Date: Thu Apr 12 03:33:40 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/ba63c7280e01e959f6856af2969ff4c0fae8c65d

commit ba63c7280e01e959f6856af2969ff4c0fae8c65d
Author: Jao-ke Chin-Lee <jchinlee@google.com>
Date: Thu Apr 12 03:53:58 2018

This is blocking the cq.

#27 reverted our MP image. I'm in the middle of tearing down all of the MP machines on chromium-swarm using the problematic image.

all MP machines on chromium-swarm using the problematic image have been terminated, and most have been reaped. linux-chromeos-rel should be able to chew through its queue overnight.

The bot in the original report (https://build.chromium.org/p/chromium.memory/builders/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29) is now green, so looking good.

And CQ seems to be fine too. Leaving open for jbudorick@ to verify all is as expected, but lowering prio since the issue seems fixed.

I think we're all set here. Thanks everyone for your help here. Postmortem to follow.

Issue 831959 has been merged into this issue.

Does this still need to be pushed out to non-gce/mp hosts?

#36: it seemed to only be an issue for MP because we only run the cros tests that were affected there, so I don't think so.

Postmortem here: https://docs.google.com/document/d/1RpQyLQ2ncSZckY57wFFuHzWJt2r0cdOpMSi1avNDvMo/edit#