Build_packages, fails for app-i18n/input-tools, bad file size for download |
||||||||
Issue description
I have a new chroot (created & downloaded early this morning); I'm trying to build lumpy completely from source, and it dies while building app-i18n/input-tools. It's actually dying in the downloading stage, complaining about mis-matched file sizes:
$ USE='chrome_internal afdo_use' ~/trunk/src/scripts/build_packages --nousepkg --withdev --withtest --withautotest --skip_toolchain_update --nowithdebug --board=lumpy --accept_licenses=@CHROMEOS --skip_chroot_upgrade app-i18n/input-tools
ChromeOS version information:
CHROME_BRANCH=56
CHROME_VERSION=
CHROMEOS_VERSION_STRING=8928.0.2016_10_24_1329
CHROME_BASE=
INFO : Selecting profile: /mnt/host/source/src/private-overlays/overlay-lumpy-private/profiles/base for /build/lumpy
INFO : Cross toolchain already up to date. Nothing to do.
INFO : Clearing shadow utils lockfiles under /build/lumpy
INFO : Checking package dependencies are correct: app-i18n/input-tools
INFO : Merging board packages now
Starting fast-emerge.
Building package app-i18n/input-tools on /build/lumpy
Calculating deps...
Deps calculated in 0m7.1s
[ebuild N ] app-i18n/input-tools-6.6.7.9-r1::chromeos to /build/lumpy/ 98268 KiB
Total: 1 package (1 new), Size of downloads: 98268 KiB
Pending 0/1, Fetching 1/1, [Time 13:31:03 | Elapsed 0m7.5s | Load 0.83 0.58 0.39]
Fetched app-i18n/input-tools-6.6.7.9-r1 in 10.22s
=== Start output for job input-tools-6.6.7.9-r1 (0m10.2s) ===
input-tools-6.6.7.9-r1: >>> Fetching (1 of 1) app-i18n/input-tools-6.6.7.9-r1::chromeos for /build/lumpy/
input-tools-6.6.7.9-r1: 13:31:04: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz /var/cache/chromeos-cache/distfiles/target/input-tools-6.6.7.9.tar.gz.tmp
input-tools-6.6.7.9-r1: !!! Fetched file: input-tools-6.6.7.9.tar.gz VERIFY FAILED!
input-tools-6.6.7.9-r1: !!! Reason: Filesize does not match recorded size
input-tools-6.6.7.9-r1: !!! Got: 100593493
input-tools-6.6.7.9-r1: !!! Expected: 100625840
input-tools-6.6.7.9-r1: Refetching... File renamed to '/var/cache/chromeos-cache/distfiles/target/input-tools-6.6.7.9.tar.gz._checksum_failure_.hPKlVl'
input-tools-6.6.7.9-r1:
input-tools-6.6.7.9-r1: !!! Couldn't download 'input-tools-6.6.7.9.tar.gz'. Aborting.
input-tools-6.6.7.9-r1: * Fetch failed for 'app-i18n/input-tools-6.6.7.9-r1'
input-tools-6.6.7.9-r1: >>> Failed to emerge app-i18n/input-tools-6.6.7.9-r1 for /build/lumpy/
=== Complete: job input-tools-6.6.7.9-r1 (0m10.2s) ===
Started app-i18n/input-tools-6.6.7.9-r1 (logged in /tmp/input-tools-6.6.7.9-r1-kEE2Tv)
=== Start output for job input-tools-6.6.7.9-r1 (0m6.6s) ===
input-tools-6.6.7.9-r1: >>> Emerging (1 of 1) app-i18n/input-tools-6.6.7.9-r1::chromeos for /build/lumpy/
input-tools-6.6.7.9-r1: 13:31:15: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz /var/cache/chromeos-cache/distfiles/target/input-tools-6.6.7.9.tar.gz.tmp
input-tools-6.6.7.9-r1: !!! Fetched file: input-tools-6.6.7.9.tar.gz VERIFY FAILED!
input-tools-6.6.7.9-r1: !!! Reason: Filesize does not match recorded size
input-tools-6.6.7.9-r1: !!! Got: 100593493
input-tools-6.6.7.9-r1: !!! Expected: 100625840
input-tools-6.6.7.9-r1: Refetching... File renamed to '/var/cache/chromeos-cache/distfiles/target/input-tools-6.6.7.9.tar.gz._checksum_failure_.hPKlVl'
input-tools-6.6.7.9-r1:
input-tools-6.6.7.9-r1: !!! Couldn't download 'input-tools-6.6.7.9.tar.gz'. Aborting.
input-tools-6.6.7.9-r1: * Fetch failed for 'app-i18n/input-tools-6.6.7.9-r1', Log file:
input-tools-6.6.7.9-r1: * '/build/lumpy/tmp/portage/logs/app-i18n:input-tools-6.6.7.9-r1:20161024-203114.log'
input-tools-6.6.7.9-r1: >>> Failed to emerge app-i18n/input-tools-6.6.7.9-r1 for /build/lumpy/, Log file:
input-tools-6.6.7.9-r1: >>> '/build/lumpy/tmp/portage/logs/app-i18n:input-tools-6.6.7.9-r1:20161024-203114.log'
input-tools-6.6.7.9-r1:
input-tools-6.6.7.9-r1: * Messages for package app-i18n/input-tools-6.6.7.9-r1 merged to /build/lumpy/:
input-tools-6.6.7.9-r1:
input-tools-6.6.7.9-r1: * Fetch failed for 'app-i18n/input-tools-6.6.7.9-r1', Log file:
input-tools-6.6.7.9-r1: * '/build/lumpy/tmp/portage/logs/app-i18n:input-tools-6.6.7.9-r1:20161024-203114.log'
=== Complete: job input-tools-6.6.7.9-r1 (0m6.6s) ===
Failed app-i18n/input-tools-6.6.7.9-r1 (in 0m6.6s), retrying later.
Pending 1/1, Building 0/0, Retrying 1, [Time 13:31:20 | Elapsed 0m24.3s | Load 0.8 0.59 0.39]
Retrying emerge of app-i18n/input-tools-6.6.7.9-r1.
Started app-i18n/input-tools-6.6.7.9-r1 (logged in /tmp/input-tools-6.6.7.9-r1-yxVYjQ)
=== Start output for job input-tools-6.6.7.9-r1 (0m6.8s) ===
input-tools-6.6.7.9-r1: >>> Emerging (1 of 1) app-i18n/input-tools-6.6.7.9-r1::chromeos for /build/lumpy/
input-tools-6.6.7.9-r1: 13:31:21: INFO: RunCommand: /mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz /var/cache/chromeos-cache/distfiles/target/input-tools-6.6.7.9.tar.gz.tmp
input-tools-6.6.7.9-r1: !!! Fetched file: input-tools-6.6.7.9.tar.gz VERIFY FAILED!
input-tools-6.6.7.9-r1: !!! Reason: Filesize does not match recorded size
input-tools-6.6.7.9-r1: !!! Got: 100593493
input-tools-6.6.7.9-r1: !!! Expected: 100625840
input-tools-6.6.7.9-r1: Refetching... File renamed to '/var/cache/chromeos-cache/distfiles/target/input-tools-6.6.7.9.tar.gz._checksum_failure_.hPKlVl'
input-tools-6.6.7.9-r1:
input-tools-6.6.7.9-r1: !!! Couldn't download 'input-tools-6.6.7.9.tar.gz'. Aborting.
input-tools-6.6.7.9-r1: * Fetch failed for 'app-i18n/input-tools-6.6.7.9-r1', Log file:
input-tools-6.6.7.9-r1: * '/build/lumpy/tmp/portage/logs/app-i18n:input-tools-6.6.7.9-r1:20161024-203120.log'
input-tools-6.6.7.9-r1: >>> Failed to emerge app-i18n/input-tools-6.6.7.9-r1 for /build/lumpy/, Log file:
input-tools-6.6.7.9-r1: >>> '/build/lumpy/tmp/portage/logs/app-i18n:input-tools-6.6.7.9-r1:20161024-203120.log'
input-tools-6.6.7.9-r1:
input-tools-6.6.7.9-r1: * Messages for package app-i18n/input-tools-6.6.7.9-r1 merged to /build/lumpy/:
input-tools-6.6.7.9-r1:
input-tools-6.6.7.9-r1: * Fetch failed for 'app-i18n/input-tools-6.6.7.9-r1', Log file:
input-tools-6.6.7.9-r1: * '/build/lumpy/tmp/portage/logs/app-i18n:input-tools-6.6.7.9-r1:20161024-203120.log'
=== Complete: job input-tools-6.6.7.9-r1 (0m6.8s) ===
Failed app-i18n/input-tools-6.6.7.9-r1 (in 0m6.8s). Your build has failed.
Pending 1/1, [Time 13:31:27 | Elapsed 0m31.1s | Load 0.89 0.61 0.4]
Packages failed:
app-i18n/input-tools-6.6.7.9-r1
ERROR : Mon Oct 24 13:31:28 PDT 2016
ERROR : PGID PPID PID ELAPSED TIME %CPU COMMAND
ERROR : 9 6 9 04:24 00:00:00 0.0 -bash
ERROR : 317 9 317 01:47 00:00:00 0.1 \_ /bin/bash /home/mobiletc-prebuild/trunk/src/scripts/build_packages --nousepkg --withdev --withtest --withautotest --skip_toolchain_update --nowithdebug --board=lumpy --accept_licenses=@CHROMEOS --skip_chroot_upgrade app-i18n/input-tools
ERROR : 317 317 1888 00:00 00:00:00 0.0 \_ /bin/bash /home/mobiletc-prebuild/trunk/src/scripts/build_packages --nousepkg --withdev --withtest --withautotest --skip_toolchain_update --nowithdebug --board=lumpy --accept_licenses=@CHROMEOS --skip_chroot_upgrade app-i18n/input-tools
ERROR : 317 1888 1889 00:00 00:00:00 0.0 \_ ps f -o pgid,ppid,pid,etime,cputime,%cpu,command
ERROR : Arguments of 317: '--nousepkg' '--withdev' '--withtest' '--withautotest' '--skip_toolchain_update' '--nowithdebug' '--board=lumpy' '--accept_licenses=@CHROMEOS' '--skip_chroot_upgrade' 'app-i18n/input-tools'
ERROR : Backtrace: (most recent call is last)
ERROR : build_packages:265:main(), called: die_err_trap
ERROR :
ERROR : Command failed:
ERROR : Command '( set -o pipefail; sudo -E "${EMERGE_CMD[@]}" "${EMERGE_FLAGS[@]}" "${PACKAGES[@]}" | tee "${tmpfile}" )' exited with nonzero code: 1
$
,
Oct 24 2016
looking at other files that are owned by that id (they're all input-tools), and the person who updated the input-tools package via gerrit, it looks like that id belongs to Lan Wei <azurewei@google.com>. azurewei: remember, once a tarball has been posted to the mirror and an ebuild uses it, you must never modify it again. i've deleted the new one and restored the old one: $ gsutil rm gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz#1476951546753000 $ gsutil mv -p gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz#1474418218984000 gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz $ gsutil ls -la gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz 100625840 2016-10-24T21:50:15Z gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz#1477345815690000 metageneration=1
,
Oct 24 2016
Mike, can we just change gentoo to use the first generation of a versioned file?
,
Oct 24 2016
tl;dr: no multiple points: - we set our mirrors to fetch via https:// which has no way (that i know of) of discovering or fetching via specific generation numbers - Gentoo has no specific knowledge of gs:// URIs ... it offloads to a chromite tool that just processes the given URI - we could change the chromite tool to enumerate all the generations and pick the oldest, but that'll slow things down - in general, this sort of automagic i dislike as it makes it harder to reason about a system -- "`gsutil cp xxx` isn't giving me the same file !?" i wish there was a way in GS to "lock" files. we could add a cronjob that'd poll all the files in there and change their owners to a service account and drop all write access for Googlers. it would make it a bit more difficult to recover if we did need to actually modify files, but maybe that's OK ?
,
Oct 24 2016
Difficult. I personally need to be able to change files sometimes, but rarely after a few hours (and not after sending to cq). Changing owners afterward sounds like a reasonable idea. Maybe this could be done by the buildbots the first time they pull the revision and compare download to the manifest?
,
Oct 25 2016
it should be possible to tease out the full list of files in SRC_URI for the sdk and target boards and then check their ACLs in GS (for the ones we're fetching from the localmirror repos), and then "take over" them. we'd have to implement a caching system (maybe using cidb?) because running `gsutil acl get` on one file is pretty slow, let alone a few hundred. prob best in something like the master CQ as part of the "finish" stage. it wouldn't be difficult work, but it'd take work :).
,
Oct 25 2016
Don't know how much work, but this failure case wastes probably 1-2h engineer hours per month? Could be more. If it takes more than an engineer week to implement the benefits/cost may not be clear.
,
Oct 25 2016
that sounds like a reasonable estimate. we turned on versioning in the mirror buckets specifically to be able to recover in scenarios like this, and it seemed to make recovery trivial, so maybe it's good enough for now.
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Aug 3 2017
Closing. Please reopen it if its not fixed. Thanks! |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by vapier@chromium.org
, Oct 24 2016$ gsutil ls -al gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz 100625840 2016-09-21T00:36:58Z gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz#1474418218984000 metageneration=1 100593493 2016-10-20T08:19:06Z gs://chromeos-localmirror-private/distfiles/input-tools-6.6.7.9.tar.gz#1476951546753000 metageneration=1 looks like someone re-uploaded it on Oct 20th. `gsutil acl get` on it shows: { "entity": "user-00b4903a97ea4152d8121e4e304d3426e65d95582d5d88874dc8e8d74c7ccb04", "entityId": "00b4903a97ea4152d8121e4e304d3426e65d95582d5d88874dc8e8d74c7ccb04", "role": "OWNER" }, no idea how to look up that id though. let's cc the peeps who normally work on input-tools in case they know.