New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 706012 link

Starred by 2 users
Status: Archived
Owner:
dgarr...@chromium.org
Closed: Apr 2017
Cc:
d...@chromium.org
█ bhthompson@chromium.org
saintlou@google.com
josa...@google.com
puneetster@google.com
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

Branch builders checking out chromite on branch breaks cbuildbot_launch.

Project Member Reported by bhthompson@google.com, Mar 28 2017 
The master and pre-flight builders on the release branch waterfall are dying just after being started, for example:

https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R58-9334.B/builds/24

@@@SEED_STEP@cbuildbot [master-release]@@@
@@@STEP_CURSOR@cbuildbot [master-release]@@@
@@@STEP_STARTED@@@
@@@STEP_LINK@stdout-->stdio@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fmaster-release_release-R58-9334.B%2F24%2F%2B%2Frecipes%2Fsteps%2Fcbuildbot__master-release_%2F0%2Fstdout@@@
@@@STEP_LOG_LINE@exception@Traceback (most recent call last):@@@
@@@STEP_LINK@logdog-->exception@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fmaster-release_release-R58-9334.B%2F24%2F%2B%2Frecipes%2Fsteps%2Fcbuildbot__master-release_%2F0%2Flogs%2Fexception%2F0@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 204, in run@@@
@@@STEP_LOG_LINE@exception@    env=step_env, cwd=step_config.cwd)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 323, in _run_cmd@@@
@@@STEP_LOG_LINE@exception@    **fhandles)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/third_party/subprocess42.py", line 249, in __init__@@@
@@@STEP_LOG_LINE@exception@    super(Popen, self).__init__(args, **kwargs)@@@
@@@STEP_LOG_LINE@exception@  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__@@@
@@@STEP_LOG_LINE@exception@    errread, errwrite)@@@
@@@STEP_LOG_LINE@exception@  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child@@@
@@@STEP_LOG_LINE@exception@    raise child_exception@@@
@@@STEP_LOG_LINE@exception@OSError: [Errno 2] No such file or directory@@@
@@@STEP_LOG_END@exception@@@
@@@STEP_EXCEPTION@@@
@@@STEP_CURSOR@cbuildbot [master-release]@@@
@@@STEP_CLOSED@@@
@@@SEED_STEP@Uncaught Exception@@@
@@@STEP_CURSOR@Uncaught Exception@@@
@@@STEP_STARTED@@@
@@@STEP_LOG_LINE@exception@Traceback (most recent call last):@@@
@@@STEP_LINK@logdog-->exception@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fmaster-release_release-R58-9334.B%2F24%2F%2B%2Frecipes%2Fsteps%2FUncaught_Exception%2F0%2Flogs%2Fexception%2F0@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/run.py", line 364, in _old_run@@@
@@@STEP_LOG_LINE@exception@    recipe_result = recipe_script.run(api, properties)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 81, in run@@@
@@@STEP_LOG_LINE@exception@    self.run_steps, properties, self.PROPERTIES, api=api)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 580, in invoke_with_properties@@@
@@@STEP_LOG_LINE@exception@    **additional_args)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 541, in _invoke_with_properties@@@
@@@STEP_LOG_LINE@exception@    return callable_obj(*props, **additional_args)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/recipes/cros/cbuildbot_internal.py", line 94, in RunSteps@@@
@@@STEP_LOG_LINE@exception@    api.chromite.run_cbuildbot()@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 578, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/build/scripts/slave/recipe_modules/chromite/api.py", line 296, in run_cbuildbot@@@
@@@STEP_LOG_LINE@exception@    self.run()@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 578, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/build/scripts/slave/recipe_modules/chromite/api.py", line 434, in run@@@
@@@STEP_LOG_LINE@exception@    args=cbb_args)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 578, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/build/scripts/slave/recipe_modules/chromite/api.py", line 236, in cbuildbot@@@
@@@STEP_LOG_LINE@exception@    return self.m.step(name, cmd, allow_subannotations=True, **kwargs)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 229, in __call__@@@
@@@STEP_LOG_LINE@exception@    return self.run_from_dict(schema.as_jsonish())@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 618, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 234, in run_from_dict@@@
@@@STEP_LOG_LINE@exception@    return self.step_client.run_step(dct)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 200, in run_step@@@
@@@STEP_LOG_LINE@exception@    return self._engine.run_step(StepConfig.create(**step_dict))@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/run.py", line 261, in run_step@@@
@@@STEP_LOG_LINE@exception@    step_result = open_step.run()@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 204, in run@@@
@@@STEP_LOG_LINE@exception@    env=step_env, cwd=step_config.cwd)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 323, in _run_cmd@@@
@@@STEP_LOG_LINE@exception@    **fhandles)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/third_party/subprocess42.py", line 249, in __init__@@@
@@@STEP_LOG_LINE@exception@    super(Popen, self).__init__(args, **kwargs)@@@
@@@STEP_LOG_LINE@exception@  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__@@@
@@@STEP_LOG_LINE@exception@    errread, errwrite)@@@
@@@STEP_LOG_LINE@exception@  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child@@@
@@@STEP_LOG_LINE@exception@    raise child_exception@@@
@@@STEP_LOG_LINE@exception@OSError: [Errno 2] No such file or directory@@@
@@@STEP_LOG_END@exception@@@
@@@STEP_CLOSED@@@
@@@SEED_STEP@Failure reason@@@
@@@STEP_CURSOR@Failure reason@@@
@@@STEP_STARTED@@@
@@@STEP_LOG_LINE@reason@Uncaught Exception: OSError(2, 'No such file or directory')@@@
@@@STEP_LINK@logdog-->reason@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fmaster-release_release-R58-9334.B%2F24%2F%2B%2Frecipes%2Fsteps%2FFailure_reason%2F0%2Flogs%2Freason%2F0@@@
@@@STEP_LOG_END@reason@@@
@@@STEP_CLOSED@@@

Comment 1 by dgarr...@chromium.org, Mar 28 2017 

My CL:
https://chromium-review.googlesource.com/c/456786/

(Reverted here)
https://chromium-review.googlesource.com/c/461289/


Changed us to using cbuildbot_launch instead of cbuildbot. The recipe change was SUPPOSED to check out a chromte repository at TOT, and invoke cbuildbot_launch there. Instead, it used a branched checkout, which doesn't yet have cbuildbot_launch. The important line is here (notice the path):

/b/build/slave/master-release-release-R58-9334.B/build/chromite/scripts/cbuildbot_launch --buildroot /b/cbuild/shared_internal --buildbot --branch release-R58-9334.B --buildnumber 24 --git-cache-dir /b/cros_git_cache master-release

Comment 2 by dgarr...@chromium.org, Mar 28 2017

Cc: d...@chromium.org
My change was intended to switch the intial checkout of chromite to TOT, and run scripts/cbuildbot_launch from that TOT checkout.

As can be seen here in stage 'bot_update', the builder still checked out chromite on the branch (which is what caused the later failures):

https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fmaster-release_release-R58-9334.B%2F24%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout


===Running git checkout --force origin/release-R58-9334.B (attempt #1)===
In directory: /b/build/slave/master-release-release-R58-9334.B/build/chromite
HEAD is now at 413c3c1c... Point R58 ARC++ MNC branch at mnc-dr-arc-m58
===Succeeded in 0.0 mins===



What concerns me is the next step is named 'read_chromite_config', which would probably need to be reading the branched config file:

I'm not sure what it's needed for though.

https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fmaster-release_release-R58-9334.B%2F24%2F%2B%2Frecipes%2Fsteps%2Fread_chromite_config%2F0%2Fstdout

Comment 3 by dgarr...@chromium.org, Mar 28 2017

Summary: Branch builders checking out chromite on branch breaks cbuildbot_launch. (was: Release Branch Master Builders Dying (purple) on a recipe file read)
Project Member

Comment 4 by sheriffbot@chromium.org, Apr 12 2017 

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 5 by dgarr...@chromium.org, Apr 12 2017

Status: Fixed (was: Untriaged)
The problematic change was reverted, and a fixed version has been relanded.

Comment 6 by dchan@google.com, May 30 2017

Labels: VerifyIn-60

Comment 7 by dchan@chromium.org, Aug 1 2017

Labels: VerifyIn-61

Comment 8 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)
►

Sign in to add a comment