build_packages is attempting to pull in prebuilts that don't exist |
|||||
Issue description
I ran build_packages --board cyan this morning after syncing. I'm getting these errors:
10:20:30: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/Package
s /tmp/tmpP_M2ww.tmp
10:20:31: WARNING: Copy Error: src gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/Packages dest /tmp/tmpP_M2ww.tmp: return code: 1; command: /mnt/host/source/.cache/common/gsutil_4
.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/Packages /tmp/tmpP_M2ww.tmp
CommandException: No URLs matched: gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/Packages
cmd=['/mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/Packages', '/
tmp/tmpP_M2ww.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto'} (Temp log for crbug.com/642986 )
10:20:31: ERROR: return code: 1; command: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/pa
ckages/Packages /tmp/tmpP_M2ww.tmp
CommandException: No URLs matched: gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/Packages
cmd=['/mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/Packages', '/
tmp/tmpP_M2ww.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto'}
Fetcher exited with a failure condition.
!!! Error fetching binhost package info from 'gs://chromeos-prebuilt/host/amd64/x86-generic/paladin-R59-9446.0.0-rc1/packages/'
!!! FETCHCOMMAND_GS failed
10:20:32: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/Packa
ges /tmp/tmp3RA3QR.tmp
10:20:33: WARNING: Copy Error: src gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/Packages dest /tmp/tmp3RA3QR.tmp: return code: 1; command: /mnt/host/source/.cache/common/gsutil
_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/Packages /tmp/tmp3RA3QR.tmp
CommandException: No URLs matched: gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/Packages
cmd=['/mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/Packages',
'/tmp/tmp3RA3QR.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto'} (Temp log for crbug.com/642986 )
10:20:33: ERROR: return code: 1; command: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/
packages/Packages /tmp/tmp3RA3QR.tmp
CommandException: No URLs matched: gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/Packages
cmd=['/mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/Packages',
'/tmp/tmp3RA3QR.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto'}
Fetcher exited with a failure condition.
!!! Error fetching binhost package info from 'gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc1/packages/'
!!! FETCHCOMMAND_GS failed
10:20:34: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/Package
s /tmp/tmpuoRw12.tmp
10:20:35: WARNING: Copy Error: src gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/Packages dest /tmp/tmpuoRw12.tmp: return code: 1; command: /mnt/host/source/.cache/common/gsutil_4
.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/Packages /tmp/tmpuoRw12.tmp
CommandException: No URLs matched: gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/Packages
cmd=['/mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/Packages', '/
tmp/tmpuoRw12.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto'} (Temp log for crbug.com/642986 )
10:20:35: ERROR: return code: 1; command: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/pa
ckages/Packages /tmp/tmpuoRw12.tmp
CommandException: No URLs matched: gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/Packages
cmd=['/mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/Packages', '/
tmp/tmpuoRw12.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto'}
Fetcher exited with a failure condition.
!!! Error fetching binhost package info from 'gs://chromeos-prebuilt/host/amd64/arm-generic/paladin-R59-9446.0.0-rc1/packages/'
!!! FETCHCOMMAND_GS failed
And it does look like the particular version of prebuilts that it's trying to copy don't exist:
(cr) ((d0dc881...)) chirantan@belgium ~/trunk/src/scripts $ gsutil ls gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-944*
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9440.0.0-rc1/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9440.0.0-rc1/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9440.0.0-rc2/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9440.0.0-rc2/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9440.0.0-rc3/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9440.0.0-rc3/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc1/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc1/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc2/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc2/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc3/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc3/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc4/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc4/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc5/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9441.0.0-rc5/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc1/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc1/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc2/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc2/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc3/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc3/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc4/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9442.0.0-rc4/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9443.0.0-rc1/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9443.0.0-rc1/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9443.0.0-rc2/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9443.0.0-rc2/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9444.0.0-rc1/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9444.0.0-rc1/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc2/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9446.0.0-rc2/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9447.0.0-rc1/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9447.0.0-rc1/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9447.0.0-rc2/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9447.0.0-rc2/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9447.0.0-rc3/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9447.0.0-rc3/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9448.0.0-rc1/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9448.0.0-rc1/packages/
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9448.0.0-rc2/:
gs://chromeos-prebuilt/host/amd64/amd64-generic/paladin-R59-9448.0.0-rc2/packages/
There is a R59-9446.0.0-rc2 but not a R59-9446.0.0-rc1.
Not sure where this belongs so CC'ing a few people who might know better than me.
,
Apr 10 2017
Adding dgarrett in case he knows what might be causing this issue.
,
Apr 10 2017
Based on my understanding, missing prebuilts shouldn't break your build, just slow it down. Is that not what you are seeing?
,
Apr 10 2017
Well, other than the Chrome prebuilt, that is. It's a special case.
,
Apr 10 2017
The Chrome PFQ has been failing to build images since this
build:
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4205
That's R59-9441.0.0-rc3. So, that's why the builds don't exist.
But, I can't explain why anything should be relying on those
builds...
,
Apr 10 2017
"missing" can have multiple meanings ;). if the binpkgs are stale/old/not specified, then we often call those "missing". in that case, things will be slow, but should still work. however, if the Packages file says "a binpkg exists at XXX", but it doesn't actually exist there, then i think that is a fatal error as chirantan@ is showing here. and that is a scenario that shouldn't have happened ... the Packages file should normally be updated only after we've actually uploaded the various binpkg files. if we uploaded a Packages file that points to a tbz2 that we failed to upload or otherwise went missing, that's something we need to fix.
,
Apr 10 2017
OK. My observations regarding the Chrome PFQ are a red herring. I mis-read this bug. But... The CQ has been failing since about the same time (apparently, for an unrelated reason), which is build R59-9446.0.0-rc2. So, the CQ is somehow doing the wrong thing, possibly because builders are getting killed when they're found to be pointless.
,
Apr 10 2017
hmm, wrt proactively killing CQ runs, it *shouldn't* be an issue (although shouldn't != doesn't). ideally we should be treating this as an atomic operation ... the binpkgs all get uploaded, and only once that has finished, do we upload the Packages file.
,
Apr 10 2017
nxia@ understands the root cause, and is working on a fix. Basically, R59-9446.0.0-rc1 had no CLs in it, so it succeeded, but then recorded non-existent prebuilts. This sort of thing corrects itself once there's a green run with at least one CL in it, but the CQ has been failing ever since. The immediate fix is to get a green CQ run.
,
Apr 10 2017
> The immediate fix is to get a green CQ run. ha! :)
,
Apr 10 2017
While nxia@ looks at the long term fix, I'm going to investigate whether it's feasible to find and revert the problematic binhost changes.
,
Apr 11 2017
What's the current status of this? I'm hitting the same problem when I try to do any setup_board in my chroot...
,
Apr 11 2017
We've had a green CQ run, so the immediate problem should be past. You'll need to run 'repo sync' to pick up the results from that run. We're still working on a long-term fix that will prevent a recurrence.
,
Apr 11 2017
We've seen at least one green CQ, so in theory, this is fixed. You'll have to repo sync to get that fix.
,
Apr 11 2017
yes, repo sync fixed it for me. Thanks.
,
Apr 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/8b8b69a429169a413834fa7f60e663783a1fe7da commit 8b8b69a429169a413834fa7f60e663783a1fe7da Author: Ningning Xia <nxia@chromium.org> Date: Mon Apr 17 23:23:54 2017 Do not publish uprevs when no CLs to test in the CQ When there're no picked up CLs in a CQ run, no slave builds will be scheduled, in this case, do not publish uprevs. Also, clean up the PublishUprevChangesStage unit tests. BUG= chromium:710036 TEST=unit_tests Change-Id: Id9116e3b6f500602b46fe4ce8392cccc2da20d27 Reviewed-on: https://chromium-review.googlesource.com/474564 Commit-Ready: Ningning Xia <nxia@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> Reviewed-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/8b8b69a429169a413834fa7f60e663783a1fe7da/cbuildbot/stages/completion_stages_unittest.py [modify] https://crrev.com/8b8b69a429169a413834fa7f60e663783a1fe7da/cbuildbot/stages/completion_stages.py [modify] https://crrev.com/8b8b69a429169a413834fa7f60e663783a1fe7da/cbuildbot/builders/simple_builders.py
,
Apr 24 2017
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by chirantan@chromium.org
, Apr 10 2017