This is happening consistently for me. If I launch a remote cbuildbot job for the chromiumos-sdk trybot builder, it fails in the UploadPrebuilts stage, with the following error (which does not look like it could be due to any of my patches):
01:03:57: DEBUG: GSContext: would've run: /b/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -a public-read -- /b/cbuild/repository/chroot/var/lib/portage/pkgs/dev-perl/Net-SMTP-SSL-1.30.0.tbz2 gs://chromeos-prebuilt/host/amd64/amd64-host/chroot-2017.04.23.105932/packages/dev-perl/Net-SMTP-SSL-1.30.0.tbz2
01:03:57: DEBUG: GSContext: would've run: /b/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -a public-read -- /tmp/cbuildbot-tmpQvzc8J/chromite.upload_prebuilts.index.JNjSQt gs://chromeos-prebuilt/host/amd64/amd64-host/chroot-2017.04.23.105932/packages/index.html
STEP_LINK: Prebuilts[amd64-host]: chroot-2017.04.23.105932; https://storage.googleapis.com/chromeos-prebuilt/host/amd64/amd64-host/chroot-2017.04.23.105932/packages/index.html
Revving git file /b/cbuild/repository/src/third_party/chromiumos-overlay/chromeos/config/make.conf.amd64-host
Updating FULL_BINHOST="gs://chromeos-prebuilt/host/amd64/amd64-host/chroot-2017.04.23.024258/packages/" to FULL_BINHOST="gs://chromeos-prebuilt/host/amd64/amd64-host/chroot-2017.04.23.105932/packages/"
01:04:11: DEBUG: Trying to push /b/cbuild/repository/src/third_party/chromiumos-overlay/chromeos/config to prebuilt_branch:refs/heads/master
01:04:13: DEBUG: ending retries with error: <class 'chromite.lib.cros_build_lib.RunCommandError'>(return code: 1; command: git rebase refs/remotes/cros/master
Cannot rebase: You have unstaged changes.
Please commit or stash them.
cmd=['git', 'rebase', u'refs/remotes/cros/master'], cwd=/b/cbuild/repository/src/third_party/chromiumos-overlay/chromeos/config)
upload_prebuilts: Unhandled exception:
Traceback (most recent call last):
File "/b/cbuild/repository/chromite/bin/upload_prebuilts", line 168, in <module>
DoMain()
File "/b/cbuild/repository/chromite/bin/upload_prebuilts", line 164, in DoMain
commandline.ScriptWrapperMain(FindTarget)
File "/b/cbuild/repository/chromite/lib/commandline.py", line 837, in ScriptWrapperMain
ret = target(argv[1:])
File "/b/cbuild/repository/chromite/scripts/upload_prebuilts.py", line 870, in main
options.sync_binhost_conf)
File "/b/cbuild/repository/chromite/scripts/upload_prebuilts.py", line 571, in SyncHostPrebuilts
RevGitFile(git_file, {key: binhost}, dryrun=self._dryrun)
File "/b/cbuild/repository/chromite/scripts/upload_prebuilts.py", line 171, in RevGitFile
git.PushWithRetry(prebuilt_branch, cwd, dryrun=dryrun, retries=retries)
File "/b/cbuild/repository/chromite/lib/git.py", line 1343, in PushWithRetry
SyncPushBranch(git_repo, remote_ref.remote, local_ref.ref)
File "/b/cbuild/repository/chromite/lib/git.py", line 1297, in SyncPushBranch
RunGit(git_repo, ['rebase', rebase_target])
File "/b/cbuild/repository/chromite/lib/git.py", line 817, in RunGit
['git'] + cmd, **kwargs)
File "/b/cbuild/repository/chromite/lib/retry_util.py", line 122, in GenericRetry
ret = functor(*args, **kwargs)
File "/b/cbuild/repository/chromite/lib/cros_build_lib.py", line 625, in RunCommand
raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: git rebase refs/remotes/cros/master
Cannot rebase: You have unstaged changes.
Please commit or stash them.
cmd=['git', 'rebase', u'refs/remotes/cros/master'], cwd=/b/cbuild/repository/src/third_party/chromiumos-overlay/chromeos/config
01:04:13: ERROR:
return code: 1; command: /b/cbuild/repository/chromite/bin/upload_prebuilts --build-path /b/cbuild/repository --board amd64-host --debug --dry-run --set-version 2017.04.23.105932 --git-sync --prepend-version chroot --upload gs://chromeos-prebuilt --binhost-conf-dir src/third_party/chromiumos-overlay/chromeos/binhost --set-version 2017.04.23.105932 --sync-host --upload-board-tarball --prepackaged-tarball /b/cbuild/repository/built-sdk.tar.xz --toolchain-upload-path '2017/04/%%(target)s-%(version)s.tar.xz' --toolchain-tarball armv7a-cros-linux-gnueabi:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/armv7a-cros-linux-gnueabi.tar.xz --toolchain-tarball armv6j-cros-linux-gnueabi:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/armv6j-cros-linux-gnueabi.tar.xz --toolchain-tarball i686-pc-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/i686-pc-linux-gnu.tar.xz --toolchain-tarball aarch64-cros-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/aarch64-cros-linux-gnu.tar.xz --toolchain-tarball arm-none-eabi:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/arm-none-eabi.tar.xz --toolchain-tarball mipsel-cros-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/mipsel-cros-linux-gnu.tar.xz --toolchain-tarball x86_64-cros-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/x86_64-cros-linux-gnu.tar.xz --key FULL_BINHOST
cmd=['/b/cbuild/repository/chromite/bin/upload_prebuilts', '--build-path', '/b/cbuild/repository', '--board', 'amd64-host', '--debug', '--dry-run', '--set-version', '2017.04.23.105932', '--git-sync', '--prepend-version', u'chroot', '--upload', 'gs://chromeos-prebuilt', '--binhost-conf-dir', 'src/third_party/chromiumos-overlay/chromeos/binhost', '--set-version', '2017.04.23.105932', '--sync-host', '--upload-board-tarball', '--prepackaged-tarball', '/b/cbuild/repository/built-sdk.tar.xz', '--toolchain-upload-path', '2017/04/%%(target)s-%(version)s.tar.xz', '--toolchain-tarball', 'armv7a-cros-linux-gnueabi:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/armv7a-cros-linux-gnueabi.tar.xz', '--toolchain-tarball', 'armv6j-cros-linux-gnueabi:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/armv6j-cros-linux-gnueabi.tar.xz', '--toolchain-tarball', 'i686-pc-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/i686-pc-linux-gnu.tar.xz', '--toolchain-tarball', 'aarch64-cros-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/aarch64-cros-linux-gnu.tar.xz', '--toolchain-tarball', 'arm-none-eabi:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/arm-none-eabi.tar.xz', '--toolchain-tarball', 'mipsel-cros-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/mipsel-cros-linux-gnu.tar.xz', '--toolchain-tarball', 'x86_64-cros-linux-gnu:/b/cbuild/repository/chroot/tmp/toolchain-pkgs/x86_64-cros-linux-gnu.tar.xz', '--key', 'FULL_BINHOST'], cwd=/b/cbuild/repository
@@@STEP_FAILURE@@@
01:04:13: ERROR: <class 'chromite.lib.failures_lib.BuildScriptFailure'>: /b/cbuild/repository/chromite/bin/upload_prebuilts failed (code=1)
Traceback (most recent call last):
File "/b/cbuild/repository/chromite/lib/failures_lib.py", line 190, in wrapped_functor
return functor(*args, **kwargs)
File "/b/cbuild/repository/chromite/cbuildbot/stages/artifact_stages.py", line 614, in PerformStage
**common_kwargs)
File "/b/cbuild/repository/chromite/cbuildbot/prebuilts.py", line 166, in UploadPrebuilts
return _UploadPrebuilts(buildroot=buildroot, **kwargs)
File "/b/cbuild/repository/chromite/cbuildbot/prebuilts.py", line 217, in _UploadPrebuilts
commands.RunBuildScript(buildroot, cmd, chromite_cmd=True)
File "/b/cbuild/repository/chromite/cbuildbot/commands.py", line 140, in RunBuildScript
raise failures_lib.BuildScriptFailure(ex, cmd[0])
File "/b/cbuild/repository/chromite/cbuildbot/commands.py", line 140, in RunBuildScript
raise failures_lib.BuildScriptFailure(ex, cmd[0])
BuildScriptFailure: /b/cbuild/repository/chromite/bin/upload_prebuilts failed (code=1)
01:04:13: INFO: Running cidb query on pid 42336, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x51ee310>
01:04:13: INFO: Running cidb query on pid 42336, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x51ee610>
01:04:13: INFO: Running cidb query on pid 42336, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x51ee950>
************************************************************
** Finished Stage UploadPrebuilts [amd64-host] - Mon, 24 Apr 2017 01:04:14 -0700 (PDT)
************************************************************
01:04:14: INFO: Running cidb query on pid 42336, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x51eeb90>
The full build log for this trybot run can be seen at: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chromiumos-sdk/builds/2192
Comment 1 by cmt...@chromium.org
, Apr 24 2017