canaries fail gsutil uploads with AccessDeniedException 403 |
|||
Issue descriptionMultiple reks and ultima release builds have failed with these errors in several stages: 05:14:00: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/ultima-release/R57-9077.0.0/stateful.tgz gs://chromeos-image-archive/ultima-release/R57-9077.0.0/stateful.tgz 05:14:16: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/ultima-release/R57-9077.0.0/stateful.tgz gs://chromeos-moblab-lenovo/ultima-release/R57-9077.0.0/stateful.tgz 05:14:18: WARNING: GS_ERROR: AccessDeniedException: 403 Forbidden looks like it's been going on through the week-end. For instance: https://uberchromegw.corp.google.com/i/chromeos/builders/reks-release/builds/658 Wasn't there a similar problem recently? If someone would link that bug, that would be great. I couldn't find it. It would be nice to know if it was the same problem.
,
Dec 13 2016
Where did you find those buckets in cloud console? I can't seem to figure out the right project.
,
Dec 13 2016
The buckets come from the stdio logs.
,
Dec 15 2016
Most recent builds are passing on ultima-release and reks-release, so the issue is probably fixed?
,
Dec 15 2016
It's fixed until it comes back! The majority of infra failures are things that we've seen before, and it's common to say "let's close this and reopen if it happens again". But that strategy no longer works: there are too many of these, and they keep coming back. We need to find the root cause and make sure that either it gets fixed, or its frequency goes down by two orders of magnitude. Probably not as urgent though :)
,
Dec 15 2016
#1: Aviv, my apologies. Those errors 403 are from the stdio logs, I also cannot find the buckets. Also I didn't realize that the question was for Ben. Ben, do you still remember how to get to those buckets?
,
Dec 15 2016
Isn't it an ACL issue that requires some manual steps to set up the GS bucket correctly?
,
Mar 9 2017
|
|||
►
Sign in to add a comment |
|||
Comment 1 by benchan@chromium.org
, Dec 12 2016