Looking...

A more detailed error:

/usr/bin/python src/chrome/test/chromedriver/run_buildbot_steps.py --revision e0aeea286663d47c475599ba16f91635aed1baff
cleaning temp directory: /tmp
@@@BUILD_STEP archive prebuilts@@@
Traceback (most recent call last):
  File "src/chrome/test/chromedriver/run_buildbot_steps.py", line 516, in <module>
    main()
  File "src/chrome/test/chromedriver/run_buildbot_steps.py", line 480, in main
    _ArchivePrebuilts(commit_position)
  File "src/chrome/test/chromedriver/run_buildbot_steps.py", line 53, in _ArchivePrebuilts
    zip_path = util.Zip(os.path.join(chrome_paths.GetBuildDir(['chromedriver']),
  File "/b/build/slave/Linux/build/src/chrome/test/chromedriver/chrome_paths.py", line 37, in GetBuildDir
    ', '.join(required_paths))
RuntimeError: Cannot find build directory containing chromedriver
<Thread(Thread-1, started 140446109779712)> ProcessRead: proc.stdout finished.
<Thread(Thread-1, started 140446109779712)> ProcessRead: cleaning up.
<Thread(Thread-2, started daemon 140446101387008)> TimedFlush: Finished
<Thread(Thread-1, started 140446109779712)> ProcessRead: finished.
Stopping Xvfb with pid 27176 ...
Xvfb pid file removed
 killed dbus-daemon with PID 27174
 cleared DBUS_SESSION_BUS_ADDRESS environment variable
@@@STEP_CURSOR@annotated_steps@@@
step returned non-zero exit code: 1

So the required directory 'chromedriver' is gone missing.

I'm going to try clobbering the build directory once the current build finishes. I don't have any better ideas at the moment.

After clobber, it hits a new exception:

@@@STEP_CURSOR@Uncaught Exception@@@
@@@STEP_STARTED@@@
@@@STEP_LOG_LINE@exception@Traceback (most recent call last):@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/run.py", line 420, in run@@@
@@@STEP_LOG_LINE@exception@    recipe_result = recipe_script.run(api, api._engine.properties)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 59, in run@@@
@@@STEP_LOG_LINE@exception@    self.RunSteps, properties, self.PROPERTIES, api=api)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 514, in invoke_with_properties@@@
@@@STEP_LOG_LINE@exception@    **additional_args)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/loader.py", line 475, in _invoke_with_properties@@@
@@@STEP_LOG_LINE@exception@    return callable_obj(*props, **additional_args)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/recipes/chromium.chromedriver.recipe_autogen.py", line 332, in RunSteps@@@
@@@STEP_LOG_LINE@exception@    dispatch_directory[api.properties["buildername"]](api)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/recipes/chromium.chromedriver.recipe_autogen.py", line 290, in Linux_steps@@@
@@@STEP_LOG_LINE@exception@    goma_dir = api.goma.ensure_goma()@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 239, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/recipe_modules/goma/api.py", line 28, in ensure_goma@@@
@@@STEP_LOG_LINE@exception@    self.m.cipd.install_client()@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 239, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/recipe_modules/cipd/api.py", line 58, in install_client@@@
@@@STEP_LOG_LINE@exception@    step_test_data=lambda: self.test_api.example_install_client(version)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 239, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/python/api.py", line 16, in __call__@@@
@@@STEP_LOG_LINE@exception@    return self.m.step(name, cmd + list(args or []), **kwargs)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 155, in __call__@@@
@@@STEP_LOG_LINE@exception@    return self.run_from_dict(self._engine.create_step(schema))@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/recipe_api.py", line 279, in _inner@@@
@@@STEP_LOG_LINE@exception@    return func(*a, **kw)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_modules/step/api.py", line 160, in run_from_dict@@@
@@@STEP_LOG_LINE@exception@    return self._engine.run_step(dct)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/run.py", line 369, in run_step@@@
@@@STEP_LOG_LINE@exception@    open_step = self._step_runner.open_step(step)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 192, in open_step@@@
@@@STEP_LOG_LINE@exception@    self._print_step(step_stream, step_dict, step_env)@@@
@@@STEP_LOG_LINE@exception@  File "/b/build/scripts/slave/.recipe_deps/recipe_engine/recipe_engine/step_runner.py", line 281, in _print_step@@@
@@@STEP_LOG_LINE@exception@    step_stream.write_line('in dir %s:' % (step.get('cwd') or os.getcwd()))@@@
@@@STEP_LOG_LINE@exception@OSError: [Errno 2] No such file or directory@@@
@@@STEP_LOG_END@exception@@@

Oh I think I know what's going on.

We just did the GYP->GN migration for this bot (see https://bugs.chromium.org/p/chromedriver/issues/detail?id=1384), and the chromedriver binary (not directory) is now built in the out/Default directory instead of out/Release. Unfortunately chrome_paths.py wasn't updated with this new path: https://cs.chromium.org/chromium/src/chrome/test/chromedriver/chrome_paths.py?q=chrome_paths.py&sq=package:chromium&l=1

The last successful build was #20791, which also built into out/Default: https://build.chromium.org/p/chromium.chromedriver/builders/Linux/builds/20791/steps/compile/logs/stdio

I think this means that the script was picking up stale builds out of out/Release until out/Release got clobbered due to a reboot or something :(

I'll prepare a CL to fix the path.

...and on the next build it's back to the original exception. OK, clobber failed.

Could it be the build is misconfigured somehow?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/31c70814160f9d3b9a1631797de672d4bee68f70

commit 31c70814160f9d3b9a1631797de672d4bee68f70
Author: samuong <samuong@chromium.org>
Date: Thu Jul 28 20:25:52 2016

[chromedriver] Update chrome_paths to use GN build directory.

BUG= 632389 
TBR=stgao@chromium.org

Review-Url: https://codereview.chromium.org/2188323002
Cr-Commit-Position: refs/heads/master@{#408463}

[modify] https://crrev.com/31c70814160f9d3b9a1631797de672d4bee68f70/chrome/test/chromedriver/chrome_paths.py

The current build looks good so far (at least it's gotten past where it was failing before).

I'm not sure about the error from #c5. Line 290 of chromium.chromedriver.recipe_autogen.py ensures that goma is running, although I'm not really sure what ways this could fail. Is it possible that clobbering would cause goma to fail somehow?

OK, that fixed it, closing the bug. I'm not sure about #c5 failure either. If you want to experiment, let me know and I can clobber the build again.

Sergey, could you clobber the other bots too (Win7, Mac, Linux32)? I just want to make sure they're not having the same issue...

On it...

Hmm... vm867-m1 (Linux32) is running a build; last build took 3.5h - I won't be around when it completes... 

vm640-m1 (Mac 10.6) and vm890-m1 (Win) are in the same boat. I'd have to kill the running builds to clobber these. What would you prefer - kill the builds, or leave them alone (and unclobbered)?

that's fine, you can kill the current builds

vm867-m1 (Linux32) is clobbered.
vm640-m1 (Mac 10.6) wouldn't let me to connect :-(
vm890-m1 (Win7): clobbered, rebooted the VM ('cause it's windows...)

It seems Linux32 actually broke because of the clobber.

Finally, clobbered vm640-m1 (Mac) - I can log in to it now (not sure what was the previous rejection about - maybe it was rebooting?).

It's getting late, I'm postponing this till tomorrow. Let me know if you see more errors, and if they are infra-related.

linux32 broke because it was missing a sysroot, i've just gone and installed it, so hopefully it cycles green in build #20125 (or at least passes the meta build step)

So, wait, you need to install something by hand on a bot? Can we configure that into Puppet or something? Or make the build itself install it if possible?

We'd like to live in a world where a newly provisioned machine would just run your build out of the box. Is there a way to do this?

I just followed the instructions in the error message:

Missing sysroot (//build/linux/debian_wheezy_amd64-sysroot). To fix, run: build/linux/sysroot_scripts/install-sysroot.py --arch=amd64

https://build.chromium.org/p/chromium.chromedriver/builders/Linux32/builds/20125/steps/meta%20build/logs/stdio

It's passing that step now, but if this can be automated for newly provisioned machines, that would be great. I think this somehow already happens for the linux64 bot, although I'm not sure how this works.

I see. I asked around, and I the best way to fix it is to run this command (or an equivalent to ensure sysroot) as part of the build. Just add it to the recipe.

In general, you shouldn't need to do anything manually on the bot to get the build pass. All the setup must be done either in the bot's image / puppet config, or in the build itself.

I'm passing the bug to you to triage and find the right owner for this action item. Thanks!

dpranke@, you mentioned that the sysroot-installing logic might have been getting confused on the old 32-bit builders (https://bugs.chromium.org/p/chromedriver/issues/detail?id=1384#c25) but it's still failing to install on the new 64-bit builder. I installed it manually so it's not a problem right now, but do you know who owns whatever installs the sysroot on the buildbots?

Sorry, I just noticed this is still running on the old build slave. Let's see what happens when we flip it over to the new 64-bit one (issue 631826).

The bug linked in #23 suggests that the new slave is working, and the builder looks like it has a long string of successes. Can we close this bug?

Yes. Following up my comment in #22, we've flipped it over to a 64-bit host (with a 64-bit userland) and the sysroot installs automatically. There's still one other problem, but that's being tracked by  issue chromedriver:1449 .