Canary timeout in paygen stage while trying to copy a gsutil file
Reported by
bfreed@chromium.org,
Mar 2 2016
|
||||
Issue descriptionToday's samus canary failed with the log at https://uberchromegw.corp.google.com/i/chromeos/builders/samus%20canary/builds/2977/steps/Paygen/logs/stdio It looks like it is repeatedly trying to copy a small json file over a period of 2 hours: : 13:19:49: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-releases/canary-channel/samus/7993.0.0/ChromeOS-recovery-R51-7993.0.0-samus.instructions.json : 15:19:35: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.16.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-releases/canary-channel/samus/7993.0.0/ChromeOS-recovery-R51-7993.0.0-samus.instructions.json
,
Mar 2 2016
Builders were renamed yesterday. The builder still exists here: https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release/builds/2977/steps/Paygen/logs/stdio
,
Mar 2 2016
The file it was trying to cat (gs://chromeos-releases/canary-channel/samus/7993.0.0/ChromeOS-recovery-R51-7993.0.0-samus.instructions.json) wasn't uploaded until 3:22pm PST, so it's not particularly surprising it timed out on a job starting at 10AM PST
,
Mar 2 2016
$ gsutil stat gs://chromeos-releases/canary-channel/samus/7993.0.0/ChromeOS-recovery-R51-7993.0.0-samus.instructions.json gs://chromeos-releases/canary-channel/samus/7993.0.0/ChromeOS-recovery-R51-7993.0.0-samus.instructions.json: Creation time: Tue, 01 Mar 2016 23:22:05 GMT Content-Length: 1012 Content-Type: application/octet-stream Hash (crc32c): 5niEhA== Hash (md5): mLpyTmIcWEzOa5EPn6cegA== ETag: CIDio8TQoMsCEAI= Generation: 1456874524832000 Metageneration: 2
,
Mar 2 2016
> The file it was trying to cat (gs://chromeos-releases/canary-channel/samus/7993.0.0/ChromeOS-recovery-R51-7993.0.0-samus.instructions.json) wasn't uploaded until 3:22pm PST, so it's not particularly surprising it timed out on a job starting at 10AM PST That is a bug, isn't it? It should not be trying to cat a file that is not yet uploaded. I know little about paygen. Is it started in parallel with a buildbot that generates this file? If so, then the problem is that the buildbot did not generate the expected file in time. If we WontFix this issue, we accept it happening again.
,
Apr 27 2016
|
||||
►
Sign in to add a comment |
||||
Comment 1 by pgervais@chromium.org
, Mar 2 2016Labels: -Infra Infra-Troopers