Note: I sure wish we could restore monotonic increasing build numbers to make it easier to refer to, talk about & reason about the sequence of builds for a given builder (b/78123581), rather than having to resort to obscure and unweildy "CIDB build ID" or "Buildbucket ID" or "MILO" numbers.
The most recent grunt-release build failed in the SimpleChromeArtifacts and TestSimpleChromeWorkflow stages:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=2823400
There doesn't seem to be anything wrong in the "SimpleChromeArtifacts" logs:
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8938799046846097504/+/steps/SimpleChromeArtifacts/0/stdout
However, according to the time-chart that stage appears to have stopped at exactly the same time as TestSimpleChromeWorkflow, which does show errors:
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8938799046846097504/+/steps/TestSimpleChromeWorkflow/0/stdout
[154/45361] CXX obj/base/base_static/base_switches.o
FAILED: obj/base/base_static/base_switches.o
x86_64-cros-linux-gnu-clang++ -B../../../../../../../../../../tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache/chrome-sdk/tarballs/grunt+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__grunt-release__R70-10949.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/binutils-bin/2.27.0-gold -Wno-unknown-warning-option -MMD -MF obj/base/base_static/base_switches.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DOFFICIAL_BUILD -DGOOGLE_CHROME_BUILD -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -DCR_CLANG_REVISION=\"338452-1\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -DCR_SYSROOT_HASH=4e7db513b0faeea8fb410f70c9909e8736f5c0ab -DOS_CHROMEOS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -fno-strict-aliasing -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -B../../third_party/binutils/Linux_x64/Release/bin -pthread -fcolor-diagnostics -fmerge-all-constants -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -m64 -march=x86-64 -Wall -Wextra -Wimplicit-fallthrough -Wthread-safety -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-user-defined-warnings -Wno-unused-lambda-capture -Wno-null-pointer-arithmetic -Wno-enum-compare-switch -Wno-ignored-pragma-optimize -fno-omit-frame-pointer -g2 -gsplit-dwarf -ggnu-pubnames -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -O2 -fno-ident -fdata-sections -ffunction-sections -std=gnu++14 -fno-exceptions -fno-rtti --sysroot=../../../../../../../../../../tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache/chrome-sdk/tarballs/grunt+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__grunt-release__R70-10949.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz -fvisibility-inlines-hidden -pipe -pipe -pipe -march=amdfam10 -fno-split-dwarf-inlining -fdebug-info-for-profiling -D__google_stl_debug_vector=1 -Wno-unknown-warning-option -stdlib=libc++ -Wno-inline-asm -c ../../base/base_switches.cc -o obj/base/base_static/base_switches.o
realpath: unrecognized option '--relative-to=/b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_grunt/Release'
Usage:
realpath [-s|--strip] [-z|--zero] filename ...
realpath -h|--help
realpath -v|--version
/b/swarming/w/ir/tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache/chrome-sdk/tarballs/grunt+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__grunt-release__R70-10949.0.0+target_toolchain/usr/bin/clang++: 16: exec: /../../lib/ld-linux-x86-64.so.2: not found
... [and lots similar to the above]
[653/45361] ACTION //extensions/browser/api:api_registration_bundle_generator_registration(//build/toolchain/cros:target)
ninja: build stopped: subcommand failed.
22:16:43: INFO: RunCommand: python /b/swarming/w/ir/cache/goma/client/goma_ctl.py stop
using /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/goma_tmp_dir.ZUJBOx as tmpdir
Killing compiler proxy.
compiler proxy status: http://127.0.0.1:8088 quit!
[1;31m22:16:45: ERROR: Traceback (most recent call last):
File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 682, in Run
self.PerformStage()
File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 328, in PerformStage
self._BuildChrome(sdk_cmd, goma)
File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 275, in _BuildChrome
result = sdk_cmd.Ninja(run_args={'extra_env': extra_env})
File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/commands.py", line 3063, in Ninja
return self.Run(cmd, run_args=run_args)
File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/commands.py", line 3048, in Run
return cros_build_lib.RunCommand(cros_cmd, cwd=self.cwd, **run_args)
File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/cros_build_lib.py", line 647, in RunCommand
raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: cros --log-level debug --cache-dir /b/swarming/w/ir/tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache chrome-sdk --board grunt --cwd /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src --sdk-path /b/swarming/w/ir/cache/cbuild/repository/buildbot_archive/grunt-release/R70-10949.0.0 --nogn-gen --nostart-goma --gomadir /b/swarming/w/ir/cache/goma/client --internal --chrome-src /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src -- ninja -C out_grunt/Release -j 500 chrome chrome_sandbox nacl_helper
...
22:14:35: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/.cache/common/gsutil_4.33.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/swarming/w/ir/cache/cbuild/repository/buildbot_archive/grunt-release/R70-10949.0.0/UPLOADED gs://chromeos-moblab-dev-arc-bld/grunt-release/R70-10949.0.0/UPLOADED
[1;31m22:16:47: ERROR: [0m
22:16:47: INFO: Translating result to fail.
************************************************************
** Finished Stage SimpleChromeArtifacts - Tue, 07 Aug 2018 22:16:47 -0700 (PDT)
************************************************************
CC'ing Build & CI folks initially to help route to someone who may know more about this stage in the build process and can help triage.
Comment 1 by djkurtz@google.com
, Aug 8