New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Archived
Closed: Mar 2017
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Sign in to add a comment

canaries fail gsutil uploads with AccessDeniedException 403

Project Member Reported by, Dec 12 2016 Back to list

Issue description

Multiple reks and ultima release builds have failed with these errors in several stages:

05:14:00: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/ultima-release/R57-9077.0.0/stateful.tgz gs://chromeos-image-archive/ultima-release/R57-9077.0.0/stateful.tgz
05:14:16: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/ultima-release/R57-9077.0.0/stateful.tgz gs://chromeos-moblab-lenovo/ultima-release/R57-9077.0.0/stateful.tgz
05:14:18: WARNING: GS_ERROR: AccessDeniedException: 403 Forbidden
looks like it's been going on through the week-end.  For instance:

Wasn't there a similar problem recently?  If someone would link that bug, that would be great.  I couldn't find it.  It would be nice to know if it was the same problem.

Seems like GS ACL problem.  I just checked the cloud console, access to chromeos-moblab-intel and chromeos-moblab-quanta are fine, but chromeos-moblab-lenovo isn't. 
Where did you find those buckets in cloud console? I can't seem to figure out the right project.
The buckets come from the stdio logs.
Most recent builds are passing on ultima-release and reks-release, so the issue is probably fixed?
Labels: -Pri-1 Pri-2
It's fixed until it comes back!

The majority of infra failures are things that we've seen before, and it's common to say "let's close this and reopen if it happens again".  But that strategy no longer works: there are too many of these, and they keep coming back.  We need to find the root cause and make sure that either it gets fixed, or its frequency goes down by two orders of magnitude.

Probably not as urgent though :)

#1: Aviv, my apologies.  Those errors 403 are from the stdio logs, I also cannot find the buckets.  Also I didn't realize that the question was for Ben.  Ben, do you still remember how to get to those buckets?
Isn't it an ACL issue that requires some manual steps to set up the GS bucket correctly?
Status: Archived

Sign in to add a comment