$ find /b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/|grep X11|head
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/XF86keysym.h
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/X.h
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/Xwinsock.h
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/Xpoll.h
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/ap_keysym.h
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/XWDFile.h
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/extensions
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/extensions/XKBsrv.h
/b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/X11/extensions/XKBproto.h

$ find /b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk/tarballs/|grep X11|grep Xlib
(shows nothing.)

Maybe:
1. not included in the original tar ball
2. something bad happened on extracting tar ball and Xlib.h has gone.

Let me try to clear /b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk to fix the issue if this is caused by 2.

removed:
[1] chrome-bot@slave9-c1:(Linux 14.04):~$ rm -rf /b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/.cros_cache/chrome-sdk

I am not sure who I need to notify but I guess amd64-generic+10110.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz broken.  It might not have Xlib.h.

Luis and Caroline, do you know who I need to notify?

I would start by checking with the ChromeOS build/infra team.

we don't include X packages anymore because CrOS doesn't use X.  so the sysroot itself omitting Xlib.h doesn't sound like a bug to me.  looking at older recent sysroots suggests this isn't a new thing and Chrome has been building fine.

it becomes a question of why the Chrome build is trying to include X components.  maybe something on the Simple Chrome side (with the recent chrome-sdk changes) broke, or the Chrome side itself broke by incorrectly including X things, or Chrome started building components it shouldn't for a CrOS target.  should compare the last passing run with the first failing run and see if anything on the GN args have changed.

either way, fairly certain not a bug in the CrOS sysroot or toolchain, so i'll drop those peeps from the bug.

Is this something I can try building on a Linux box? If so, please let me know what the GN configuration should be and I'll take a look.

Looking at the first line of the build output, 6590 is building "chromiumos_preflight" and 6591 (where the failure starts) is building "All all".

The build target is passed to "cros chrome-sdk", so while this may be due to a chromite change, it does not seem to be simple chrome per-se.

Unfortunately our build output does not make it particularly clear what target is being build, but x11_util.cc is failing to build and appears to be part of remoting/host/linux.

the Simple Chrome workflow can run on any Linux system w/out need for a CrOS SDK.  but i'm not sure how to reproduce this particular bot config, and i don't think the CrOS bots are showing this issue (yet?).

+dgarrett@chromium.org, +akeshet@chromium.org - Has anything changed that would cause this builder to build "All all" instead of "chromiumos_preflight" that you are aware of? Any idea where this command is coming from exactly?

For referenc, this is the last succeeding build command:

/b/rr/tmpSLUZNR/rw/scripts/slave/.recipe_deps/depot_tools/cros chrome-sdk '--board=amd64-generic' --nocolor '--log-level=debug' --use-external-config --gomadir /b/c/goma_client --nostart-goma -- /b/rr/tmpSLUZNR/rw/scripts/slave/.recipe_deps/depot_tools/ninja -w 'dupbuild=err' -C /b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/src/out_amd64-generic/Release -d stats -j 50 chromiumos_preflight


And this one fails:

/b/rr/tmp2s9ghw/rw/scripts/slave/.recipe_deps/depot_tools/cros chrome-sdk '--board=amd64-generic' --nocolor '--log-level=debug' --use-external-config --gomadir /b/c/goma_client --nostart-goma -- /b/rr/tmp2s9ghw/rw/scripts/slave/.recipe_deps/depot_tools/ninja -w 'dupbuild=err' -C /b/c/builder/ChromeOS_amd64_Chromium_Goma_Canary/src/out_amd64-generic/Release -d stats -j 50 All all

The problem is not in chrome, or 'cros chrome-sdk', it's how we are invoking it.

We have never tried to make building "all" work in Simple Chrome. Here is what chromiumos_preflight includes:
https://cs.chromium.org/chromium/src/BUILD.gn?type=cs&q=chromiumos_preflight&sq=package:chromium&l=828

I think that this is where we are setting up the ninja command passed to simple chrome:

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium/api.py?type=cs&q=_run_ninja&sq=package:chromium&l=424

I have no clue how to determine where that is getting called from though.

the chromium.fyi waterfall isn't managed by the CrOS team.  that's a Chrome waterfall.

re: comment #12: I don't really understand how that is relevant? Aren't we all just trying to track down the problem? If there is someone more likely to understand the failure cause here, please cc them.

i dropped CrOS infra from the bug earlier because, afaik, Aviv & Don don't monitor any Chromium waterfalls.  if you have questions about those, you'll want to ping Chromium people.

'All' and 'all' is come from
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium/config.py?type=cs&sq=package:chromium&l=532
?

Looks configuration issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f4bfda2512e446f2236ff5c3d75706b0dbdad604

commit f4bfda2512e446f2236ff5c3d75706b0dbdad604
Author: Takuto Ikuta <tikuta@google.com>
Date: Thu Nov 16 07:33:23 2017

Specify build target for 'ChromeOS amd64 Chromium Goma Canary'

Without target specification, the builder tries to build 'All', 'all' target.
But some targets are unsupported in simple chrome.

Let me fix the issue by using the same target specified in chromeos-amd64-generic-rel.

TBR=dpranke@chromium.org

Bug:  785167 
Change-Id: I66a5e99ec80a0df7524db199f206eb9cf22c3cd8
Reviewed-on: https://chromium-review.googlesource.com/773921
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Reviewed-by: Shinya Kawanaka <shinyak@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517015}
[modify] https://crrev.com/f4bfda2512e446f2236ff5c3d75706b0dbdad604/testing/buildbot/chromium.fyi.json

Thank you for your investigation!