ANGLE CQ Mac Testers are failing with gclient runhooks "Xcode build version 8E2002 failed to install" |
|||||||||||
Issue descriptionThis seemed to start last night. All builds are currently failing and the ANGLE CQ is broken, so marking a high priority and attn: troopers. Please deescalate when addressed. Example builds: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_angle_rel_ng/742 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_angle_compile_dbg_ng/698 Error text: ________ running '/b/s/w/ir/cache/vpython/09242d/bin/python src/build/mac_toolchain.py' in '/b/s/w/ir/cache/builder' Detected old hermetic installation at /b/s/w/ir/cache/builder/src/build/mac_files. Deleting. [I 2018-05-01 04:47:44] About to install Xcode 8e2002 in /b/s/w/ir/cache/builder/src/build/mac_files/Xcode.app for mac Error: resolving package: failed to resolve package version (line 1): access denied (not authenticated or not enough permissions). Error: resolving package: failed to resolve package version (line 1): access denied (not authenticated or not enough permissions). [E 2018-05-01 04:47:45] original error: exit status 1 [E 2018-05-01 04:47:45] [E 2018-05-01 04:47:45] goroutine 1: [E 2018-05-01 04:47:45] #0 mac_toolchain/install.go:40 - main.installPackages() [E 2018-05-01 04:47:45] reason: failed to install CIPD packages: infra_internal/ios/xcode/mac 8e2002 [E 2018-05-01 04:47:45] [E 2018-05-01 04:47:45] [E 2018-05-01 04:47:45] #1 mac_toolchain/install.go:152 - main.installXcode() [E 2018-05-01 04:47:45] #2 mac_toolchain/main.go:115 - main.(*installRun).Run() [E 2018-05-01 04:47:45] #3 github.com/maruel/subcommands/subcommands.go:360 - subcommands.Run() [E 2018-05-01 04:47:45] #4 mac_toolchain/main.go:239 - main.main() [E 2018-05-01 04:47:45] #5 runtime/proc.go:185 - runtime.main() [E 2018-05-01 04:47:45] #6 runtime/asm_amd64.s:2337 - runtime.goexit() Xcode build version 8E2002 failed to install: Command '['/b/s/w/ir/kitchen-workdir/mac_toolchain', 'install', '-kind', 'mac', '-xcode-version', '8e2002', '-output-dir', '/b/s/w/ir/cache/builder/src/build/mac_files/Xcode.app']' returned non-zero exit status 1 Access to Xcode CIPD package requires authentication. ----------------------------------------------------------------- You appear to be a Googler. I'm sorry for the hassle, but you may need to do a one-time manual authentication. Please run: cipd auth-login and follow the instructions. NOTE: Use your google.com credentials, not chromium.org. ----------------------------------------------------------------- Error: Command '/b/s/w/ir/cache/vpython/09242d/bin/python src/build/mac_toolchain.py' returned non-zero exit status 1 in /b/s/w/ir/cache/builder
,
May 1 2018
Also failing on the mac_optional_gpu_tests_rel so it's blocking the Chromium CQ for GPU tests as well: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_optional_gpu_tests_rel/2604
,
May 1 2018
This looks very similar to https://bugs.chromium.org/p/chromium/issues/detail?id=731546&desc=2#c20 So someone with the correct rights should run: cipd acl-edit infra_internal/ios/xcode -reader user:chromium-try-gpu-builder@chops-service-accounts.iam.gserviceaccount.com Not that the GPU optional and ANGLE trybots are running on a different service account. Tentatively assigning to maruel@ since he is in the same timezone and might have the rights.
,
May 1 2018
,
May 1 2018
I think you meant Sergey.
,
May 1 2018
Is anyone around who can fix the ACLs for this service account per #3? These trybots are still broken: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_optional_gpu_tests_rel https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_angle_rel_ng https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_angle_compile_dbg_ng Is there any way to send tryjobs for future changes in this area? I'm not sure whether it was https://chromium.googlesource.com/chromium/src.git/+/82c8626c643d401e58c7b8772c1ab0e4df6c5d87 which broke them, but if it was then running mac_optional_gpu_tests_rel on changes to the Mac configurations would catch this.
,
May 1 2018
,
May 1 2018
$ cipd acl-edit infra_internal/ios/xcode -reader user:chromium-try-gpu-builder@chops-service-accounts.iam.gserviceaccount.com ACL changes applied. $ cipd acl-edit infra_internal/ios/xcode -reader user:angle-try-builder@chops-service-accounts.iam.gserviceaccount.com ACL changes applied.
,
May 1 2018
> Is there any way to send tryjobs for future changes in this area? Not easily. One can do manually for each builder <xyz>: $ led get-builder <xyz> | <edit CIPD package requirements> | led launch
,
May 1 2018
this build appears to be running fine https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_optional_gpu_tests_rel/2674 so, re-assigning to Sergey.
,
May 1 2018
,
May 1 2018
,
May 1 2018
Thanks Andrii for fixing this. Let's credit you with the fix. Filed follow-on Issue 838606 about the gap in testing which caused this to happen.
,
May 1 2018
Just noticed the bug now - thanks, tandrii@ for the fix! This was a one-time major migration of the entire Mac SDK pipeline, I don't expect any other such major changes in the near future. Hopefully we've found most of the affected builders. And sorry for the breakages. With the current setup, it's practically infeasible to find all the places the script is used in advance, so this was, unfortunately, somewhat expected. Going forward, newly LUCI-converted builders will fail without these ACLs early on (while still experimental).
,
May 1 2018
Thanks Andrii for fixing!
,
May 2 2018
Issue 838550 has been merged into this issue. |
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by mar...@chromium.org
, May 1 2018