Some suspicion related to https://b.corp.google.com/issues/34847155 , but not certain.

 Issue 687935  has been merged into this issue.

Anyone know which gcloud project the /container-vm-image-staging/ bucket belongs to, or how to figure that out?

Since when BuildPackages started to upload artifacts to GCS? What account is the build run as? This is the ACL of gs://container-vm-image-staging.

$ gsutil acl get gs://container-vm-image-staging
[
  {
    "entity": "project-owners-284542672193",
    "projectTeam": {
      "projectNumber": "284542672193",
      "team": "owners"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-editors-284542672193",
    "projectTeam": {
      "projectNumber": "284542672193",
      "team": "editors"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-viewers-284542672193",
    "projectTeam": {
      "projectNumber": "284542672193",
      "team": "viewers"
    },
    "role": "READER"
  },
  {
    "entity": "project-owners-134157665460",
    "projectTeam": {
      "projectNumber": "134157665460",
      "team": "owners"
    },
    "role": "WRITER"
  },
  {
    "entity": "project-editors-134157665460",
    "projectTeam": {
      "projectNumber": "134157665460",
      "team": "editors"
    },
    "role": "WRITER"
  },
  {
    "email": "gci-release-pipeline@cloud-kernel-build.iam.gserviceaccount.com",
    "entity": "user-gci-release-pipeline@cloud-kernel-build.iam.gserviceaccount.com",
    "role": "READER"
  },
  {
    "email": "cloud-image-release@system.gserviceaccount.com",
    "entity": "user-cloud-image-release@system.gserviceaccount.com",
    "role": "READER"
  },
  {
    "email": "gci-update-test@cloud-image-test.google.com.iam.gserviceaccount.com",
    "entity": "user-gci-update-test@cloud-image-test.google.com.iam.gserviceaccount.com",
    "role": "READER"
  }
]

This bucket had been used for quite some time now. If a new account is introduced, I can add it to the ACL of the bucket.

Can you give me ownership of that bucket, or of the project that it is in? I am not able to even list acls without owner permission.

chingcodes@ recently added a new artifact which is uploaded by BuildPackages. However, that has been around for a few days at least, so not sure how it could have suddenly caused this.

dgarrett@ what service accounts do our builders authenticate to gs as?

On one of the gce bots:

chrome-bot@cros-beefy0-c2:(Linux 14.04):~$ gcloud auth list
Credentialed accounts:
 - 3su6n15k.default@developer.gserviceaccount.com (active)

But I'm not yet convinced that weird account name is the one we broadly on builders.

I see a .boto file on the baremetal bots that claims to be a credential for chromeos.bot@gmail.com

+tandrii@ today's chrome trooper and +friedman puppet guru. Have there been any changes to puppet-deployed credentials on builders in the last day?

I see that the gsutil commands on the builder are wrapped in a pointer to a particular boto file

cmd=['/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', '-m', 'cp', '-v', '--', '/b/cbuild/internal_master/buildbot_archive/lakitu-paladin/R58-9244.0.0-rc2/build-events.json', u'gs://container-vm-image-staging/lakitu-paladin/R58-9244.0.0-rc2/build-events.json'], extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'}


That boto file on the builder is indeed for chromeos.bot@gmail.com

Please don't cc troopers, mark as Infra>Troopers instead. FTR, tandrii@'s shift has ended 55 minutes ago.

that said, I don't see in recent commits to puppet anything related to gsutil:
https://chrome-internal.googlesource.com/infra/puppet/

Signers having issues too, smells related https://bugs.chromium.org/p/chromium/issues/detail?id=687862

https://pantheon.corp.google.com/iam-admin/iam/project?project=chromeos-bot

 Issue 687862  has been merged into this issue.

Now fixed, and hopefully confirmed by running a test that talks to the signers.