New issue
Advanced search Search tips

Issue 607201 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: Bug



Sign in to add a comment

Bot "Android MIPS Builder" is broken, apparently due to a change in the /build repo

Project Member Reported by carlosk@chromium.org, Apr 27 2016

Issue description

First breakage happened in build #4711 [1]. The error happens at the compile step:

(view as text)
Traceback (most recent call last):
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/run.py", line 397, in run
    recipe_result = recipe_script.run(api, api._engine.properties)
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 57, in run
    self.RunSteps, properties, self.PROPERTIES, api=api)
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 512, in invoke_with_properties
    **additional_args)
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 473, in _invoke_with_properties
    return callable_obj(*props, **additional_args)
  File "/b/build/scripts/slave/recipes/chromium.py", line 64, in RunSteps
    compile_targets, tests_including_triggered)
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
    return func(*a, **kw)
  File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 374, in compile_specific_targets
    self.transient_check(update_step, lambda transform_name:
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
    return func(*a, **kw)
  File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 329, in transient_check
    command(lambda name: name)
  File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 378, in <lambda>
    mb_buildername=mb_buildername))
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
    return func(*a, **kw)
  File "/b/build/scripts/slave/recipe_modules/chromium_tests/api.py", line 466, in run_mb_and_compile
    self.m.chromium.compile(compile_targets, name='compile%s' % name_suffix)
  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 220, in _inner
    return func(*a, **kw)
  File "/b/build/scripts/slave/recipe_modules/chromium/api.py", line 236, in compile
    if 'goma' in self.c.compile_py.compiler:
TypeError: argument of type 'NoneType' is not iterable

I see some recent changes in the api.py file [2] from the last line of the stack trace so I'm suspicious one of them could have caused it.


[1] https://uberchromegw.corp.google.com/i/chromium.android/builders/Android%20MIPS%20Builder%20%28dbg%29/builds/4711

[2] https://chromium.googlesource.com/chromium/tools/build/+log/master/scripts/slave/recipe_modules/chromium/api.py
 
I didn't have enough time to investigate the issue so I'm leaving it to the next sheriff.

One note though: I did try to run the recipe locally using the command line suggested in setup_build > run_recipe. As it was taking a while I started multi-tasking. When I came back to the terminal window the build had failed because of a input timeout:

(...)
Do you accept the license? [y/n]: 
> Traceback (most recent call last):
  File "src/build/android/play_services/update.py", line 515, in <module>
    sys.exit(main(sys.argv[1:]))
  File "src/build/android/play_services/update.py", line 99, in main
    return args.func(args)
  File "src/build/android/play_services/update.py", line 193, in Download
    config.version_number)):
  File "src/build/android/play_services/update.py", line 405, in _CheckLicenseAgreement
    return raw_input('> ') in ('Y', 'y')
EOFError: EOF when reading a line
Error: Command '/usr/bin/python src/build/android/play_services/update.py download' returned non-zero exit status 1 in /usr/local/ssd/clankium/build/slave/fake_slave/build
step returned non-zero exit code: 2
@@@STEP_FAILURE@@@
@@@STEP_CLOSED@@@

I have no idea if I can resume this but if not this timeout becomes very annoying.

Comment 2 by qin...@chromium.org, Apr 27 2016

I am seeing that gomacc is not found

FAILED: /home/chrome-bot/goma/gomacc ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF clang_x64/obj/base/allocator/tcmalloc/debugallocation_shim.o.d -DNO_HEAP_CHECK -DV8_DEPRECATION_WARNINGS -DENABLE_MDNS=1 -DENABLE_NOTIFICATIONS -DENABLE_PEPPER_CDMS -DENABLE_PLUGINS=1 -DENABLE_PDF=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DUSE_UDEV -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_CLIPBOARD_AURAX11=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -DUSE_X11=1 -DENABLE_WEBRTC=1 -DENABLE_EXTENSIONS=1 -DENABLE_TASK_MANAGER=1 -DENABLE_THEMES=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_APP_LIST=1 -DENABLE_SETTINGS_APP=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_SERVICE_DISCOVERY=1 -DENABLE_AUTOFILL_DIALOG=1 -DENABLE_TOPCHROME_MD=1 -DUSE_PROPRIETARY_CODECS -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DENABLE_MEDIA_ROUTER=1 -DFIELDTRIAL_TESTING_ENABLED -DCR_CLANG_REVISION=267383-1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -D_GLIBCXX_DEBUG=1 -DTCMALLOC_FOR_DEBUGALLOCATION -DTCMALLOC_DONT_REPLACE_SYSTEM_ALLOC -I../../base/allocator -I../../third_party/tcmalloc/chromium/src/base -I../../third_party/tcmalloc/chromium/src -I../.. -Iclang_x64/gen -Wno-sign-compare -Wno-unused-result -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -fcolor-diagnostics -fdebug-prefix-map=/b/build/slave/android_build/build/src=. -pthread -m64 -march=x86-64 -O0 -g1 -gsplit-dwarf --sysroot=../../build/linux/debian_wheezy_amd64-sysroot -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libFindBadConstructs.so -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang check-templates -Xclang -plugin-arg-find-bad-constructs -Xclang follow-macro-expansion -Xclang -plugin-arg-find-bad-constructs -Xclang check-implicit-copy-ctors -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Wheader-hygiene -Wstring-conversion -Werror -Wall -Wno-unused-variable -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-deprecated-register -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-shift-negative-value -Wno-undefined-var-template -Wno-reorder -Wno-unused-function -Wno-unused-local-typedefs -Wno-unused-private-field -fno-threadsafe-statics -fvisibility-inlines-hidden -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare -std=gnu++11 -fno-rtti -fno-exceptions -Wno-deprecated -c ../../base/allocator/debugallocation_shim.cc -o clang_x64/obj/base/allocator/tcmalloc/debugallocation_shim.o
/bin/sh: 1: /home/chrome-bot/goma/gomacc: not found
ninja: build stopped: subcommand failed.


Comment 3 by qin...@chromium.org, Apr 27 2016

Owner: jbudorick@chromium.org
This MIPS bot has been failing for quite some time, and build success happens for a very short period of time

assigning to jbudorick@ to triage
Components: -Infra Infra>Client>Android
Labels: -Restrict-View-Google Infra-Android OS-Android Pri-1 Type-Bug
upstream bot, so removing RVG

The ensure_goma step disappeared in 4711 for reasons I don't know immediately. Investigating.
Cc: jbudorick@chromium.org
Owner: ----
Status: Available (was: Untriaged)
Cc: -jbudorick@chromium.org
Owner: jbudorick@chromium.org
Status: Started (was: Available)

Comment 9 by benhenry@google.com, Apr 27 2016

Labels: -Infra-Android
Project Member

Comment 10 by bugdroid1@chromium.org, Apr 27 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d9fde20a08da03c5cc357449e1ddff789caa3785

commit d9fde20a08da03c5cc357449e1ddff789caa3785
Author: jbudorick <jbudorick@chromium.org>
Date: Wed Apr 27 19:55:53 2016

[Android] Stop using goma on chromium.android:Android MIPS Builder (dbg)

BUG= 607201 
TBR=dpranke@chromium.org

Review-Url: https://codereview.chromium.org/1923143004
Cr-Commit-Position: refs/heads/master@{#390155}

[modify] https://crrev.com/d9fde20a08da03c5cc357449e1ddff789caa3785/tools/mb/mb_config.pyl

Project Member

Comment 11 by bugdroid1@chromium.org, Apr 27 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/e998885b560ab33e9f5ac8bc335c07e9511a1d27

commit e998885b560ab33e9f5ac8bc335c07e9511a1d27
Author: jbudorick@chromium.org <jbudorick@chromium.org>
Date: Wed Apr 27 23:47:56 2016

[Android] Assign a dedicated machine for the mips builder on chromium.android.

BUG= 607201 

Review-Url: https://codereview.chromium.org/1926953002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@300253 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/e998885b560ab33e9f5ac8bc335c07e9511a1d27/masters/master.chromium.android/builders.pyl

master restart for #11 here: https://chromereviews.googleplex.com/410577013/

Comment 13 by ukai@chromium.org, Apr 28 2016

Cc: ukai@chromium.org yyanagisawa@chromium.org tikuta@chromium.org shinyak@chromium.org
Status: Fixed (was: Started)
only passing builds % the two that were killed by the master restart.
Just FYI.

I assume that you failed to use goma because the necessary option is missing in args.gn.
In followings, there is no goma_dir option, and ninja try to use gomacc in the default location, which is wrong gomacc location on bots.
https://uberchromegw.corp.google.com/i/chromium.android/builders/Android%20MIPS%20Builder%20%28dbg%29/builds/4711/steps/generate_build_files/logs/stdio

Please try to set goma_dir like this.
https://build.chromium.org/p/tryserver.chromium.android/builders/android_chromium_gn_compile_rel/builds/62284/steps/generate_build_files%20%28with%20patch%29/logs/stdio
It sets goma_dir to where ensure_goma installed gomacc.
i.e. make the recipe have ensure_goma step to install the goma, and make goma_dir option in args.gn to point the directory.
If my explanation is unclear, please feel free to say so.  I will try to implement recipe for you.

Also, please feel free to ask any Goma-related questions to user's mailing list.  We are willing to answer any questions we get.  If you are too shy to ask questions in such a mailing list, please feel free to send email to our team's mailing list.

Thanks.

Sign in to add a comment