New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 872329 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner: ----
Closed: Aug 8
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

grunt-release: SimpleChromeArtifacts/TestSimpleChromeWorkflow stages failed for CIDB build ID = 2823400

Project Member Reported by djkurtz@google.com, Aug 8

Issue description

Note: I sure wish we could restore monotonic increasing build numbers to make it easier to refer to, talk about & reason about the sequence of builds for a given builder (b/78123581), rather than having to resort to obscure and unweildy "CIDB build ID" or "Buildbucket ID" or "MILO" numbers.

The most recent grunt-release build failed in the SimpleChromeArtifacts and TestSimpleChromeWorkflow stages:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=2823400

There doesn't seem to be anything wrong in the "SimpleChromeArtifacts" logs:
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8938799046846097504/+/steps/SimpleChromeArtifacts/0/stdout

However, according to the time-chart that stage appears to have stopped at exactly the same time as TestSimpleChromeWorkflow, which does show errors:
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8938799046846097504/+/steps/TestSimpleChromeWorkflow/0/stdout

[154/45361] CXX obj/base/base_static/base_switches.o
FAILED: obj/base/base_static/base_switches.o 
x86_64-cros-linux-gnu-clang++ -B../../../../../../../../../../tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache/chrome-sdk/tarballs/grunt+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__grunt-release__R70-10949.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/binutils-bin/2.27.0-gold -Wno-unknown-warning-option -MMD -MF obj/base/base_static/base_switches.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DOFFICIAL_BUILD -DGOOGLE_CHROME_BUILD -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -DCR_CLANG_REVISION=\"338452-1\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -DCR_SYSROOT_HASH=4e7db513b0faeea8fb410f70c9909e8736f5c0ab -DOS_CHROMEOS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -fno-strict-aliasing -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -B../../third_party/binutils/Linux_x64/Release/bin -pthread -fcolor-diagnostics -fmerge-all-constants -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -m64 -march=x86-64 -Wall -Wextra -Wimplicit-fallthrough -Wthread-safety -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-user-defined-warnings -Wno-unused-lambda-capture -Wno-null-pointer-arithmetic -Wno-enum-compare-switch -Wno-ignored-pragma-optimize -fno-omit-frame-pointer -g2 -gsplit-dwarf -ggnu-pubnames -fvisibility=hidden -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -O2 -fno-ident -fdata-sections -ffunction-sections -std=gnu++14 -fno-exceptions -fno-rtti --sysroot=../../../../../../../../../../tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache/chrome-sdk/tarballs/grunt+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__grunt-release__R70-10949.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz -fvisibility-inlines-hidden -pipe -pipe -pipe -march=amdfam10 -fno-split-dwarf-inlining -fdebug-info-for-profiling -D__google_stl_debug_vector=1 -Wno-unknown-warning-option -stdlib=libc++ -Wno-inline-asm -c ../../base/base_switches.cc -o obj/base/base_static/base_switches.o
realpath: unrecognized option '--relative-to=/b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src/out_grunt/Release'
Usage:
 realpath [-s|--strip] [-z|--zero] filename ...
 realpath -h|--help
 realpath -v|--version
/b/swarming/w/ir/tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache/chrome-sdk/tarballs/grunt+__b__swarming__w__ir__cache__cbuild__repository__buildbot_archive__grunt-release__R70-10949.0.0+target_toolchain/usr/bin/clang++: 16: exec: /../../lib/ld-linux-x86-64.so.2: not found

... [and lots similar to the above]

[653/45361] ACTION //extensions/browser/api:api_registration_bundle_generator_registration(//build/toolchain/cros:target)
ninja: build stopped: subcommand failed.
22:16:43: INFO: RunCommand: python /b/swarming/w/ir/cache/goma/client/goma_ctl.py stop
using /b/swarming/w/ir/cache/cbuild/repository/chroot/tmp/goma_tmp_dir.ZUJBOx as tmpdir
Killing compiler proxy.
compiler proxy status: http://127.0.0.1:8088 quit!
22:16:45: ERROR: Traceback (most recent call last):
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 682, in Run
    self.PerformStage()
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 328, in PerformStage
    self._BuildChrome(sdk_cmd, goma)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 275, in _BuildChrome
    result = sdk_cmd.Ninja(run_args={'extra_env': extra_env})
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/commands.py", line 3063, in Ninja
    return self.Run(cmd, run_args=run_args)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/commands.py", line 3048, in Run
    return cros_build_lib.RunCommand(cros_cmd, cwd=self.cwd, **run_args)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: cros --log-level debug --cache-dir /b/swarming/w/ir/tmp/t/cbuildbot-tmpsOVs7N/chrome-sdk-cache72pmW5/cache chrome-sdk --board grunt --cwd /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src --sdk-path /b/swarming/w/ir/cache/cbuild/repository/buildbot_archive/grunt-release/R70-10949.0.0 --nogn-gen --nostart-goma --gomadir /b/swarming/w/ir/cache/goma/client --internal --chrome-src /b/swarming/w/ir/cache/cbuild/repository/.cache/distfiles/target-master/chrome-src-internal/src -- ninja -C out_grunt/Release -j 500 chrome chrome_sandbox nacl_helper
...
22:14:35: INFO: RunCommand: /b/swarming/w/ir/cache/cbuild/repository/.cache/common/gsutil_4.33.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/swarming/w/ir/cache/cbuild/repository/buildbot_archive/grunt-release/R70-10949.0.0/UPLOADED gs://chromeos-moblab-dev-arc-bld/grunt-release/R70-10949.0.0/UPLOADED
22:16:47: ERROR: 
22:16:47: INFO: Translating result  to fail.
************************************************************
** Finished Stage SimpleChromeArtifacts - Tue, 07 Aug 2018 22:16:47 -0700 (PDT)
************************************************************



CC'ing Build & CI folks initially to help route to someone who may know more about this stage in the build process and can help triage.
 
Same issue occurred on the previous build 2821542.
But, it passed on the one before that, build 2820703:

I don't see any difference between the good & bad "SimpleChromeArtifacts" output.
But, interestingly, the passing "TestSimpleChromeWorkflow" log seems to actually contain the tail of the "SimpleChromeArtifacts" log, since it ends like this:
https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8938858906199830000/+/steps/TestSimpleChromeWorkflow/0/stdout

************************************************************
** Finished Stage SimpleChromeArtifacts - Tue, 07 Aug 2018 07:34:10 -0700 (PDT)
************************************************************

Mergedinto: 871966
Status: Duplicate (was: Available)
Looks like this failure is common across boards and already has an open bug. 

Sign in to add a comment