Bot "Android MIPS Builder" is broken, apparently due to a change in the /build repo |
|||||||||
Issue description
First breakage happened in build #4711 [1]. The error happens at the compile step:
(view as text)
Traceback (most recent call last):
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/run.py", line 397, in run
recipe_result = recipe_script.run(api, api._engine.properties)
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 57, in run
self.RunSteps, properties, self.PROPERTIES, api=api)
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 512, in invoke_with_properties
**additional_args)
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 473, in _invoke_with_properties
return callable_obj(*props, **additional_args)
File "/b/build/scripts/slave/recipes/chromium.py", line 64, in RunSteps
compile_targets, tests_including_triggered)
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
return func(*a, **kw)
File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 374, in compile_specific_targets
self.transient_check(update_step, lambda transform_name:
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
return func(*a, **kw)
File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 329, in transient_check
command(lambda name: name)
File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 378, in <lambda>
mb_buildername=mb_buildername))
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
return func(*a, **kw)
File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 466, in run_mb_and_compile
self.m.chromium.compile(compile_targets, name='compile%s' % name_suffix)
File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
return func(*a, **kw)
File "/b/build/scripts/slave/recipe_modules/chromium/api.py", line 236, in compile
if 'goma' in self.c.compile_py.compiler:
TypeError: argument of type 'NoneType' is not iterable
I see some recent changes in the api.py file [2] from the last line of the stack trace so I'm suspicious one of them could have caused it.
[1] https://uberchromegw.corp.google.com/i/chromium.android/builders/Android%20MIPS%20Builder%20%28dbg%29/builds/4711
[2] https://chromium.googlesource.com/chromium/tools/build/+log/master/scripts/slave/recipe_modules/chromium/api.py
,
Apr 27 2016
I am seeing that gomacc is not found FAILED: /home/chrome-bot/goma/gomacc ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF clang_x64/obj/base/allocator/tcmalloc/debugallocation_shim.o.d -DNO_HEAP_CHECK -DV8_DEPRECATION_WARNINGS -DENABLE_MDNS=1 -DENABLE_NOTIFICATIONS -DENABLE_PEPPER_CDMS -DENABLE_PLUGINS=1 -DENABLE_PDF=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DUSE_UDEV -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_CLIPBOARD_AURAX11=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_X11=1 -DENABLE_WEBRTC=1 -DENABLE_EXTENSIONS=1 -DENABLE_TASK_MANAGER=1 -DENABLE_THEMES=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_APP_LIST=1 -DENABLE_SETTINGS_APP=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_SERVICE_DISCOVERY=1 -DENABLE_AUTOFILL_DIALOG=1 -DENABLE_TOPCHROME_MD=1 -DUSE_PROPRIETARY_CODECS -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DENABLE_MEDIA_ROUTER=1 -DFIELDTRIAL_TESTING_ENABLED -DCR_CLANG_REVISION=267383-1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -D_GLIBCXX_DEBUG=1 -DTCMALLOC_FOR_DEBUGALLOCATION -DTCMALLOC_DONT_REPLACE_SYSTEM_ALLOC -I../../base/allocator -I../../third_party/tcmalloc/chromium/src/base -I../../third_party/tcmalloc/chromium/src -I../.. -Iclang_x64/gen -Wno-sign-compare -Wno-unused-result -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -fcolor-diagnostics -fdebug-prefix-map=/b/build/slave/android_build/build/src=. -pthread -m64 -march=x86-64 -O0 -g1 -gsplit-dwarf --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libFindBadConstructs.so -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang check-templates -Xclang -plugin-arg-find-bad-constructs -Xclang follow-macro-expansion -Xclang -plugin-arg-find-bad-constructs -Xclang check-implicit-copy-ctors -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Wheader-hygiene -Wstring-conversion -Werror -Wall -Wno-unused-variable -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-deprecated-register -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-shift-negative-value -Wno-undefined-var-template -Wno-reorder -Wno-unused-function -Wno-unused-local-typedefs -Wno-unused-private-field -fno-threadsafe-statics -fvisibility-inlines-hidden -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare -std=gnu++11 -fno-rtti -fno-exceptions -Wno-deprecated -c ../../base/allocator/debugallocation_shim.cc -o clang_x64/obj/base/allocator/tcmalloc/debugallocation_shim.o /bin/sh: 1: /home/chrome-bot/goma/gomacc: not found ninja: build stopped: subcommand failed.
,
Apr 27 2016
This MIPS bot has been failing for quite some time, and build success happens for a very short period of time assigning to jbudorick@ to triage
,
Apr 27 2016
upstream bot, so removing RVG The ensure_goma step disappeared in 4711 for reasons I don't know immediately. Investigating.
,
Apr 27 2016
,
Apr 27 2016
Might be https://chromium.googlesource.com/chromium/tools/build/+/5147bf9ce85ed6220182453a5e1e491b67b5a3ad
,
Apr 27 2016
,
Apr 27 2016
,
Apr 27 2016
,
Apr 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d9fde20a08da03c5cc357449e1ddff789caa3785 commit d9fde20a08da03c5cc357449e1ddff789caa3785 Author: jbudorick <jbudorick@chromium.org> Date: Wed Apr 27 19:55:53 2016 [Android] Stop using goma on chromium.android:Android MIPS Builder (dbg) BUG= 607201 TBR=dpranke@chromium.org Review-Url: https://codereview.chromium.org/1923143004 Cr-Commit-Position: refs/heads/master@{#390155} [modify] https://crrev.com/d9fde20a08da03c5cc357449e1ddff789caa3785/tools/mb/mb_config.pyl
,
Apr 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/e998885b560ab33e9f5ac8bc335c07e9511a1d27 commit e998885b560ab33e9f5ac8bc335c07e9511a1d27 Author: jbudorick@chromium.org <jbudorick@chromium.org> Date: Wed Apr 27 23:47:56 2016 [Android] Assign a dedicated machine for the mips builder on chromium.android. BUG= 607201 Review-Url: https://codereview.chromium.org/1926953002 git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@300253 0039d316-1c4b-4281-b951-d872f2087c98 [modify] https://crrev.com/e998885b560ab33e9f5ac8bc335c07e9511a1d27/masters/master.chromium.android/builders.pyl
,
Apr 27 2016
master restart for #11 here: https://chromereviews.googleplex.com/410577013/
,
Apr 28 2016
,
Apr 28 2016
only passing builds % the two that were killed by the master restart.
,
May 6 2016
Just FYI. I assume that you failed to use goma because the necessary option is missing in args.gn. In followings, there is no goma_dir option, and ninja try to use gomacc in the default location, which is wrong gomacc location on bots. https://uberchromegw.corp.google.com/i/chromium.android/builders/Android%20MIPS%20Builder%20%28dbg%29/builds/4711/steps/generate_build_files/logs/stdio Please try to set goma_dir like this. https://build.chromium.org/p/tryserver.chromium.android/builders/android_chromium_gn_compile_rel/builds/62284/steps/generate_build_files%20%28with%20patch%29/logs/stdio It sets goma_dir to where ensure_goma installed gomacc. i.e. make the recipe have ensure_goma step to install the goma, and make goma_dir option in args.gn to point the directory. If my explanation is unclear, please feel free to say so. I will try to implement recipe for you. Also, please feel free to ask any Goma-related questions to user's mailing list. We are willing to answer any questions we get. If you are too shy to ask questions in such a mailing list, please feel free to send email to our team's mailing list. Thanks. |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by carlosk@chromium.org
, Apr 27 2016I didn't have enough time to investigate the issue so I'm leaving it to the next sheriff. One note though: I did try to run the recipe locally using the command line suggested in setup_build > run_recipe. As it was taking a while I started multi-tasking. When I came back to the terminal window the build had failed because of a input timeout: (...) Do you accept the license? [y/n]: > Traceback (most recent call last): File "src/build/android/play_services/update.py", line 515, in <module> sys.exit(main(sys.argv[1:])) File "src/build/android/play_services/update.py", line 99, in main return args.func(args) File "src/build/android/play_services/update.py", line 193, in Download config.version_number)): File "src/build/android/play_services/update.py", line 405, in _CheckLicenseAgreement return raw_input('> ') in ('Y', 'y') EOFError: EOF when reading a line Error: Command '/usr/bin/python src/build/android/play_services/update.py download' returned non-zero exit status 1 in /usr/local/ssd/clankium/build/slave/fake_slave/build step returned non-zero exit code: 2 @@@STEP_FAILURE@@@ @@@STEP_CLOSED@@@ I have no idea if I can resume this but if not this timeout becomes very annoying.