The most recent change is updating to v1.3 assets

https://chromium-review.googlesource.com/c/chromium/src/+/892046

Maybe the gs privilege is not properly setup.

+dgarrett who knows more about build bots.

The bots are failing with vr_assets runhook step. It gets this error:
[Err: AccessDeniedException: 403 chromeos.bot@gmail.com does not have storage.objects.list access to chrome-vr-assets.]

Talked with tiborg@, we have a CL on Jan 8 (https://chromium-review.googlesource.com/c/chromium/src/+/852353) that starts to use gs://chrome-vr-assets bucket (created in issue 793943). And new 1.3 files have the same ACL as the old ones.

Not sure why bots now gets 403 after the 1.3 CL.

Don, it looks this fails for all internal bots. The external ones such as amd64-generic-chromium-pfq etc are green. And tot informational bots are green too.

Could it be a permission issue that is only with internal bots?

If you can, IAM permissions on the project are much easier (and more reliable) than ACLs on the bucket or on files.

However, while authenticated as a builder, I can run these commands with no error:

gsutil ls gs://chrome-vr-assets/
gsutil cp gs://chrome-vr-assets/6b0e506b19b79ec1be4963a9bd92ab8b92ccca1c  /tmp

That suggests that the ACLs are correct, the fact that it reports the correct user "chromeos.bot@gmail.com" suggests that the .boto file is being used correctly.

But I'm confused as to why a betty-pre-cq builder was fetching chrome build assets. Shouldn't they have been stored in it's binary prebuilt?

re #4: The external builders don't build Chrome, only Chromium.

The internal/external build machines have exactly the same GS credentials.

The assets only get downloaded if you have checkout_src_internal set to true. I believe the external bots have that set to false. I followed the standard procedure for creating an internal bucket.

Re #5: The failed bots are not pre-cq ones. They are PFQ ones and will sync and build chrome. We failed two builds in a row with the same error. :(

Oh... PFQ, I misread that inside the URL.

The process for creating the buckets is probably dated. IAM has only recently been added and become flexible enough to really use.

But that doesn't seem to be the issue here.

All I can think is that it's related to the restricted environment used to build ebuilds, but that should have reproduced when you tested the new stuff locally.

I presume you did something similar to:
  cros build --no-binary --board lumpy chrome

Assuming local builds work normally, what's different about the builders?

There are several kinds of .boto authentication. Does your local .boto file contain a value for "gs_oauth2_refresh_token" ?

That's the authentication mechanism used on the builders, and the logic to get .boto's into the chroot, and into the ebuild framework might treat it differently than the other auth mechanism.

And yes, that's a wild guess because I have no good ideas.

The VR assets are not required for ChromeOS. Is it possible to specify in the condition that we are on ChromeOS and not download the assets (https://cs.chromium.org/chromium/src/DEPS?rcl=d816f08c33dc74e24e6b11cc5db6b3987d50df15&l=1303)?

Maybe that guy? https://cs.chromium.org/chromium/tools/depot_tools/gclient.py?rcl=3a998d1b232fc968cd0d83d23f6b8a9226fa7ff6&l=1271

And if they aren't needed, we would strongly like to avoid them. space on the rootfs is an issue.

Sounds good to me. There is no point to download for chromeos if it does not need it. I don't see "checkout_chromeos" is used ATM but looks like the right condition.

tiborg@ has a CL

https://chromium-review.googlesource.com/c/chromium/src/+/901773

that would exclude vr_assets from building for chromeos and should make PFQ happy.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ad8048fa3ee015903943fa16ddf22ffb14407790

commit ad8048fa3ee015903943fa16ddf22ffb14407790
Author: Tibor Goldschwendt <tiborg@chromium.org>
Date: Mon Feb 05 18:26:00 2018

[vr] Only fetch vr_assets if target OS is Android

Bug:  808601 
Change-Id: I57dfd2201bd4677665d5e142bbca564b42f03a96
Reviewed-on: https://chromium-review.googlesource.com/901773
Reviewed-by: Xiyuan Xia <xiyuan@chromium.org>
Commit-Queue: Tibor Goldschwendt <tiborg@chromium.org>
Cr-Commit-Position: refs/heads/master@{#534429}
[modify] https://crrev.com/ad8048fa3ee015903943fa16ddf22ffb14407790/DEPS

Can you check whether this fixes the PFQ bot?

CL in #17 should fix vr_assets error on PFQ. Passing to stevenjb@/xiaoyinh@ to check.

Merged ad8048fa3ee015903943fa16ddf22ffb14407790 to branch 3340:

$ git drover --branch 3340 --cherry-pick ad8048fa3ee015903943fa16ddf22ffb14407790

I will now trigger a new 3340 branch at http://chromepmo.appspot.com/trigger.

(YAQS instructions: https://yaqs.googleplex.com/eng/q/5516743884144640)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/46682d4355d8aff9cb9df62740c561262f154286

commit 46682d4355d8aff9cb9df62740c561262f154286
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Mon Feb 05 21:30:19 2018

[vr] Only fetch vr_assets if target OS is Android

TBR=tiborg@chromium.org

(cherry picked from commit ad8048fa3ee015903943fa16ddf22ffb14407790)

Bug:  808601 
Change-Id: I57dfd2201bd4677665d5e142bbca564b42f03a96
Reviewed-on: https://chromium-review.googlesource.com/901773
Reviewed-by: Xiyuan Xia <xiyuan@chromium.org>
Commit-Queue: Tibor Goldschwendt <tiborg@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#534429}
Reviewed-on: https://chromium-review.googlesource.com/902782
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Cr-Commit-Position: refs/branch-heads/3340@{#4}
Cr-Branched-From: cf4b77e4ccc0da994c6322f71863b0d17bb5c99f-refs/heads/master@{#534315}
[modify] https://crrev.com/46682d4355d8aff9cb9df62740c561262f154286/DEPS

It does not look like merging the DEPS change to the branch worked (not entirely unexpected), so I think it is best to wait for tonight's build.

 Issue 808923  has been merged into this issue.

Right, for a DEPS change we will have to wait for tonight's build (9P) 

Beckap delete girl fogram colons

Pada 6 Feb 2018 07:53, "josa… via monorail" <
monorail+v2.3502508457@chromium.org> menulis:

This appears to have broken the 65 release branch.

For the DEPS change I think it needs to be modified in the branch's DEPS file on master, like https://chrome-internal-review.googlesource.com/#/c/chrome/tools/buildspec/+/563041/

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/buildspec/+/1d48e46f38ce9aaac06f8f0ee193f775d5a84c0a

commit 1d48e46f38ce9aaac06f8f0ee193f775d5a84c0a
Author: Bernie Thompson <bhthompson@google.com>
Date: Tue Feb 06 02:49:01 2018

Working now.