empty omaha_status.json from gs |
|||||||
Issue descriptionThis step: https://uberchromegw.corp.google.com/i/chromeos/builders/parrot-release/builds/3326/steps/Paygen/logs/stdio failed with this error: 04:37:59: INFO: Starting: Build definition (board='parrot', version='9102.0.0', channel='canary-channel') 04:37:59: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil ls 'gs://chromeos-releases/canary-channel/parrot/9102.0.0/chromeos_*_parrot_*_*_*.bin' 04:37:59: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil ls 'gs://chromeos-releases/canary-channel/parrot/9102.0.0/ChromeOS-test-*-9102.0.0-parrot.tar.xz' 04:38:03: INFO: Images found: 04:38:03: INFO: 1: chromeos_9102.0.0_parrot_recovery_canary-channel_mp-v4.bin 04:38:03: INFO: 2: ChromeOS-test-R57-9102.0.0-parrot.tar.xz 04:38:03: INFO: No active FSI builds found 04:38:03: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil cat gs://chromeos-build-release-console/omaha_status.json 04:38:04: ERROR: Failed to parse JSON downloaded from gs://chromeos-build-release-console/omaha_status.json. Here's what we got: '' Error: No JSON object could be decoded It could be a GS failure (if so, please file bug against GS), or maybe omaha_status.json was really empty at that time.
,
Dec 19 2016
,
Dec 19 2016
,
Mar 14 2017
Seems to be back with a vengeance today, caused ~20 out of 27 canary failures on the last run. https://luci-milo.appspot.com/buildbot/chromeos/asuka-release/936
,
Mar 15 2017
gsutil ls -L gs://chromeos-build-release-console/omaha_status.json
gs://chromeos-build-release-console/omaha_status.json:
Creation time: Wed, 15 Mar 2017 02:43:48 GMT (7:43 PST)
Update time: Wed, 15 Mar 2017 02:43:48 GMT
Storage class: STANDARD
Content-Length: 471311
Content-Type: text/json; charset=UTF-8
Hash (crc32c): wZT+RA==
Hash (md5): pVEzYwnJeBA7Cz9oGEQXlQ==
ETag: CLDt58++19ICEAE=
Generation: 1489545828366000
Metageneration: 1
ACL: ACCESS DENIED
Note: You need OWNER permission on the object to read its ACL
TOTAL: 1 objects, 471311 bytes (460.26 KiB)
I'm not familiar how this file is generated and uploaded to gs. Looks like last time that this file was generated was just now. I wonder whether it's really there at that moment?
,
Mar 15 2017
This file is upload by Golden Eye. They've investigated in the past and shown that they are uploading valid files on a regular schedule. We should follow up with the Google Storage team and see if they thing the issue is on their side. That might have already been filed. PS: if you drop the -L, you have permissions to read the file. It's currently populated the way I would expect.
,
Mar 15 2017
,
Mar 15 2017
Also, I thought we'd landed a workaround that would retry fetching the file if the json failed to part. Do the logs show that here?
,
Mar 15 2017
Yeah, the logs show the retries. The retry interval is short though ~1s. I had thought about increasing the time in between retries, but I wasn't sure what exactly it should be. We've seen "outages" of up to 30 mins. Also, I noticed that this is a totally different file from usual! This is "omaha_status.json" whereas I've seen "boards.json" fail in the past.
,
Mar 15 2017
Has anyone contacted the GS team? At this point, we should be able to pick a file (including the generation number), show GE logs for it's upload, and our logs downloading it. An important question that may/may not be in the GE logs is the generation number. It should match between upload and the following download.
,
Mar 15 2017
b/34812286 was the attempt that leecy@ made that I know of. They said that they saw no downloads of less than 1000 bytes during that outage time.
,
Jul 18 2017
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by semenzato@chromium.org
, Dec 19 2016