An example of failing Luci bot due to this issue: https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8951515267836408512%2F%2B%2Fsteps%2Fcompile%2F0%2Fstdout

Loading the libraries from /usr/lib is correct -- binaries should not run against the sysroot libraries.

The i386 version of libX11-xcb.so.1 is necessary on the Linux builders.

On Buildbot machine (build45-m1) libX11-xcb.so.1 comes from package libx11-xcb1:i386 (discovered by running dpkg -S /usr/lib/i386-linux-gnu/libX11-xcb.so.1). It is also a symlink to libX11-xcb.so.1.0.0.

On Swarming machine (build16-m1) /usr/lib/i386-linux-gnu/libX11-xcb.so.1 exists, but it's not part of any debian package (how?..), and it is also not a symlink. In fact it is empty. I don't understand how it could happen.

Regarless, I think the package in question should be installed by 'install-build-deps.sh --lib32'. If not, it looks like a bug in install-build-deps.sh. 

Note that the problem of running install-build-deps.sh automatically by Puppet is unsolved, so we'll have to do it manually...

Ahh, we ran install-build-deps.sh on the afflicted machine, but didn't realize there was an option to explicitly select 32 bit packages.

Zhiling, could you try running the script again, but this time add the --lib32 option?

(And the empty lib is prob hzl manually touching/experimenting on the bot.)

> could you try running the script again, but this time add the --lib32 option?

You must wait for https://chromium-review.googlesource.com/c/chromium/src/+/972590 to land first.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dd47ad310164aa2160129d98bd8be7b7357ddc51

commit dd47ad310164aa2160129d98bd8be7b7357ddc51
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Wed Mar 21 19:30:19 2018

Add libx11-xcb1:i386 as a 32-bit build dep

libX11-xcb.so.1 is required to generate the v8 context shapshot.  This was
causing the LUCI version of Linux Build (dbg)(32) to fail [1].

Patch adapted from [2] from timbrown@.

[1] https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8951515267836408512%2F%2B%2Fsteps%2Fcompile%2F0%2Fstdout
[2] https://chromium-review.googlesource.com/c/chromium/src/+/940196

BUG= 823962 
R=dpranke
CC=timbrown

Change-Id: I8fa8f8d8281613193bc9faf2f2c0c86d900ac401
Reviewed-on: https://chromium-review.googlesource.com/972590
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#544802}
[modify] https://crrev.com/dd47ad310164aa2160129d98bd8be7b7357ddc51/build/install-build-deps.sh

Back over to hzl@ to rerun install-build-deps

Thanks for the quick fix. In /usr/lib/i386-linux-gnu/libX11, new files are created.

chrome-bot@build16-m1:(Linux 14.04):~$ ls /usr/lib/i386-linux-gnu/libX11<TAB>
libX11.so.6          libX11.so.6.3.0      libX11-xcb.so.1.0.0 

As you can see, the created file is "libX11-xcb.so.1.0.0", not "libX11-xcb.so.1".
The same thing happened on the buildbot linux builder (dbg)(32). The way people solved it 5 years ago (that file was created 5 years ago) is to create a sym link from "libX11-xcb.so.1" to "libX11-xcb.so.1.0.0".

The following is what I see on buildbot linux builder dbg 32.
ls -la /usr/lib/i386-linux-gnu/libX11-xcb.so.1
lrwxrwxrwx 1 root root 19 Dec  4  2013 /usr/lib/i386-linux-gnu/libX11-xcb.so.1 -> libX11-xcb.so.1.0.0

I will do the same thing to fix it.

And now, the failing command works! Thanks all!