"Goma Release 2018-05-08" on goma-users might be what triggered the problem?

dba: Should mac slaves have the AppKit PyObjC module installed?

Things shouldn't be reliant on the system python anymore and should be using vpython is my understanding.

http://crbug.com/836696 is related.

dba: Can you help figure out what's going on here? Are we using system python when we should be using vpython? Not sure what vpython is.

It appears that way to me, but that's outside of my area of expertise, I'd recommend possibly roping in troopers to see if they can help.

Roping in troopers as per suggestion in comment #5. Troopers: is it possible that this config isn't using vpython when it should be? Any idea of how to fix that? See earlier comments in this issue.

Taking as a trooper, will look a bit more soon. 

Quick observation: LUCI builders run on swarming, and `python` should be transparently replaced by vpython. See http://go/vpython-and-you for more info.

I'm not yet sure how goma_ctl.py specifies its dependencies. vpython requires an explicit spec, either directly inside a file, or a sidecar file goma_ctl.py.vpython, or a .vpython file in one of the parent folder. Finding which one is used (if any) may help to debug this.

A bit of an oddity: in https://chromium-swarm.appspot.com/task?id=3d591dcc1fd69a10 (the swarming task for https://ci.chromium.org/p/angle/builders/luci.angle.ci/mac-dbg/6), the requested named cache is "cache/goma", while the "goma*" steps refer to "cache/goma_client". Not a show stopper, it just means the installations won't persist and will be erased and reinstalled for every build.

After some digging, I still can't figure out where goma_ctl.py is coming from, nor I can find its sources... If anyone on this bug knows, please chime in.

It's here: https://chromium.googlesource.com/infra/goma/client/+/master/client/goma_ctl.py

My guess it's pulled here: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/goma/api.py?type=cs&q=ensure_goma+file:recipe&sq=package:chromium&l=149 (not sure though)

As dba@ has pointed out, this is a regression due to goma client roll.
https://bugs.chromium.org/p/chromium/issues/detail?id=836696

The rule below was not enough.
wheel: <
    name: "infra/python/wheels/pyobjc/${vpython_platform}"
    version: "version:4.1"
    match_tag: < platform: "macosx_10_10_intel" >
  >

I will rollback the client roll and go with plan B (goma's paths are given from C++ code not from python module. Specifying a proper vpython rule is extremely difficult.)

yyanagisawa@: assigning the bug to you, since you're effectively fixing it. Feel free to reassign if appropriate.

BTW, also feel free to give us feedback on vpython and what exactly you find difficult about it, so we can improve it. Perhaps, as a bug against Infra>SDK - I hope that's the right component for vpython.

Thanks!

Sergey or Nico, any idea how we can fix the issue you pointed out in comment #8:

the requested named cache is "cache/goma", while the "goma*" steps refer to "cache/goma_client"

I'd wait for an input from yyanagisawa@ on that. In the logs I see the correct cache is set in a shell var:

'GOMA_CACHE_DIR': '/b/s/w/ir/cache/goma/mac_dbg',

https://logs.chromium.org/v/?s=angle%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8947084707774657760%2F%2B%2Fsteps%2Fpreprocess_for_goma%2F0%2Fsteps%2Fgoma_jsonstatus%2F0%2Fstdout

The cache itself is defined here: https://chrome-internal.googlesource.com/infradata/config/+/master/configs/cr-buildbucket/swarming_task_template.json#19

Maybe it makes sense to add goma_client cache as well?

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/9568037db4461d85e1875ecbc766dc6a4395cbab

commit 9568037db4461d85e1875ecbc766dc6a4395cbab
Author: Jamie Madill <jmadill@chromium.org>
Date: Wed May 09 18:39:05 2018

Reland "Update cq.cfg with new builders."

This is a reland of c009255c79d2e6594a76cb88aeaa157ce99d4ea5

Re-landing after  crbug.com/840825  was updated.

Original change's description:
> Update cq.cfg with new builders.
>
> These builders are replacing the old standalone builders and the old
> compile-only builders. They should be much faster. They also can be
> extended in the future when we support running tests from ANGLE
> standalone.
>
> Bug:  chromium:833999 
> Change-Id: Ice44c0fb8cb32d8be573f81d5df858509b00a107
> Reviewed-on: https://chromium-review.googlesource.com/1049959
> Reviewed-by: Jamie Madill <jmadill@chromium.org>

Bug:  chromium:833999 
Bug:  chromium:840825 
Change-Id: Ie08ebbef6b5802f0bb57053e082fac2e0f9aae34
Reviewed-on: https://chromium-review.googlesource.com/1052747
Reviewed-by: Jamie Madill <jmadill@chromium.org>

[modify] https://crrev.com/9568037db4461d85e1875ecbc766dc6a4395cbab/infra/config/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6028742d6670a8507aa72f54e11048e6ae9d387e

commit 6028742d6670a8507aa72f54e11048e6ae9d387e
Author: angle-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <angle-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed May 09 22:56:21 2018

Roll src/third_party/angle/ ce07f967c..a932b6b51 (2 commits)

https://chromium.googlesource.com/angle/angle.git/+log/ce07f967c21e..a932b6b51505

$ git log ce07f967c..a932b6b51 --date=short --no-merges --format='%ad %ae %s'
2018-05-01 lucferron Vulkan: Fix in DynamicBuffer, allocating too many buffers for no reason
2018-05-08 jmadill Reland "Update cq.cfg with new builders."

Created with:
  roll-dep src/third_party/angle
BUG= chromium:833999 , chromium:840825 


The AutoRoll server is located here: https://angle-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
TBR=fjhenigman@chromium.org

Change-Id: I1f2c94a095c31181848a018250723929b189a0e4
Reviewed-on: https://chromium-review.googlesource.com/1053167
Commit-Queue: angle-chromium-autoroll <angle-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: angle-chromium-autoroll <angle-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#557361}
[modify] https://crrev.com/6028742d6670a8507aa72f54e11048e6ae9d387e/DEPS

Re: #12
Since crbug.com was frozen yesterday, I could not update the issue.
I have reverted the rollout of version 155, and the issue should be fixed now.

Re: #13
I think "cache/goma/*" and "cache/goma_client" are caching different things.
"cache/goma_client" is used for sharing goma client binary itself.  Since goma client binary would be update only bump up of goma client, it should be reasonable to keep it among builds instead of re-installing every time.
"cache/goma/<build>" is a cache used by goma client.  goma client caches Transport Layer Security Certificate Revocation List (TLS CRL), compiler info, and include processed results.  cache/goma/<build> should be created per LUCI build to share such information.

Re: #14
I hope to listen to advice from LUCI folks but cache/goma_client should also be shared among swarming builds to avoid downloading goma client every build.

yyanagisawa, that's a good idea. Please add an entry to https://chrome-internal.googlesource.com/infradata/config/+/master/configs/cr-buildbucket/swarming_task_template.json#19 with the path you want to be cached. Send the CL to tandrii or nodir.

Marking this as fixed since the exception is resolved. Thanks yyanagisawa! Please mark any follow up issues as blocked on this one.