BuildPackages failed at downloading chromeos-chrome packages from GS |
|||||||
Issue descriptionBuildPackage failed at downloading files from GS. It currently retries once and fails the command if both tries fail. Increasing the retry times and add backoff time might help avoid this kind of GS flake. https://luci-milo.appspot.com/buildbot/chromeos/auron-paladin/9723 chromeos-chrome-58.0.3028.0_rc-r1: Downloading ...-base/chromeos-chrome-58.0.3028.0_rc-r1.tbz2.tmp: 143.88 MiB/143.88 MiB chromeos-chrome-58.0.3028.0_rc-r1: ('The read operation timed out',) chromeos-chrome-58.0.3028.0_rc-r1: CommandException: Some components of /build/auron/packages/chromeos-base/chromeos-chrome-58.0.3028.0_rc-r1.tbz2.tmp were not downloaded successfully. Please retry this download. chromeos-chrome-58.0.3028.0_rc-r1: chromeos-chrome-58.0.3028.0_rc-r1: cmd=['/mnt/host/source/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'cp', '-v', '--', 'gs://chromeos-prebuilt/board/cyan/chrome-R58-9331.0.0-rc1/packages/chromeos-base/chromeos-chrome-58.0.3028.0_rc-r1.tbz2', '/build/auron/packages/chromeos-base/chromeos-chrome-58.0.3028.0_rc-r1.tbz2.tmp'], extra env={'BOTO_CONFIG': '/mnt/host/source/src/private-overlays/chromeos-overlay/googlestorage_account.boto'} chromeos-chrome-58.0.3028.0_rc-r1: >>> Failed to emerge chromeos-base/chromeos-chrome-58.0.3028.0_rc-r1 for /build/auron/
,
Apr 27 2017
,
Apr 27 2017
Something that might confuse people: gsutil downloads large files in multiple parts. I think this is why the download seems to "complete" before the failure happens:
chromeos-chrome-58.0.3028.0_rc-r1: Downloading ...-base/chromeos-chrome-58.0.3028.0_rc-r1.tbz2.tmp: 143.88 MiB/143.88 MiB
chromeos-chrome-58.0.3028.0_rc-r1: ('The read operation timed out',)
The .tbz2 file in storage is really 431.65 MiB, but it's fetching 143.88 MiB. 431.65 / 3 = 143.88.
So one of the 3 parts finishes downloading, and my guess is that a read operation times out when attempting to download the *next* part.
,
Apr 27 2017
Issue 676434 has been merged into this issue.
,
Apr 27 2017
,
Apr 27 2017
From semenzato via issue 683423 : Looks like a Google Storage glitch. The error message says "Please retry this download" and it would seem that we retry once. But from the log it would seem that we don't stop at the second failure, instead we try to source the file, which doesn't exist. Why do we do that? Also, there are too few time stamps, and also, the two download progress lists are identical, so one suspects that we didn't really retry, just printed the log twice. This sort of things make the root errors hard to pinpoint, and there is a mismatch: the producer of these messages (someone in the infra team) is not the first-line consumer (the sheriff).
,
Aug 18 2017
another failure example https://luci-milo.appspot.com/buildbot/chromeos/monroe-paladin/14226
,
Aug 18 2017
,
Mar 30 2018
,
Mar 30 2018
,
May 17 2018
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by xixuan@chromium.org
, Apr 27 2017