sheyang, sergiyb: I've cc-ed you as your trooper on call today. Can you triage?

The following ios-device-gn builds on build541-m4 are green now. Not sure what changed. Updating priority to P-2 and keep watching.

I believe this is a legitimate issue.

ios/try recipe implicitly gets ensure_goma through chromium.run_mb:

    if use_goma:
      goma_dir = self.c.compile_py.goma_dir
      if not goma_dir:
        # This method defaults to use_goma=True, which doesn't necessarily
        # match build-side configuration. However, MB is configured
        # src-side, and so it might be actually using goma.
        self.ensure_goma()
        goma_dir = self.c.compile_py.goma_dir

In fact, I've confirmed MB gets the proper path to CIPD goma.

Then the path gets overwritten later in ios.build :

    use_goma = (self.compiler == 'ninja' and
                ('use_goma=1' in gyp_defines or 'use_goma=true' in gn_args))
    if use_goma:
      if 'without patch' not in suffix:
        # TODO( crbug.com/603641 ):
        # Configs aren't deapplied, so we only want to apply these
        # configs once. Really, we should refactor this so that we're
        # not applying configs at all in build(), but rather do it
        # in an earlier step.
        self.m.chromium.apply_config('ninja')
        self.m.chromium.apply_config('default_compiler')
        self.m.chromium.apply_config('goma')

I defer to Sana for adjusting ios/try and ios recipe module. We're somewhat lucky here that /b/build/goma happens to still exist on most bots. But the issue is real.

Another example:
https://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device-gn/builds/32671

Just happened on build541-m4 . 

Raising to P1 as this affects the CQ.

+smut

I've been working on this as part of  bug 626005  , and had been puzzled by the first part of what phajdan wrote in comment #5, but I didn't connect it to the second part of the comment (which makes sense).

I'll take this and get it fixed.

Issue 625286 has been merged into this issue.

Fix posted to https://codereview.chromium.org/2133483002 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac

commit d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac
Author: dpranke <dpranke@chromium.org>
Date: Thu Jul 07 23:15:37 2016

Fix configuration of goma in iOS recipes.

The way we were configuring the chromium module so that goma
would work in compiles didn't work correctly after we switched
the location of goma on the machine; the 'goma' config no longer
gives the right directory, and the `ensure_goma` step wasn't
being called reliably and consistently.

This CL reworks things so that we call ensure_goma consistently
when needed during the read_build_config() step of the ios
recipes, rather than during the build() step.

This CL also fixes an issue where if the GYP_DEFINES were specified
as a list of strings in the build config rather than a dict, things
wouldn't work right. We need this fix so that we can use goma
even with GYP, in a way that is consistent between GYP and GN.

R=smut@chromium.org
BUG= 626033 

Review-Url: https://codereview.chromium.org/2133483002

[modify] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipe_modules/ios/api.py
[modify] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipes/ios/try.expected/gn.json
[add] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipes/ios/try.expected/gyp_goma.json
[modify] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipes/ios/try.py
[modify] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipes/ios/unified_builder_tester.expected/goma.json
[modify] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipes/webrtc/ios.expected/basic.json
[modify] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipes/webrtc/ios.expected/no_tests.json
[modify] https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac/scripts/slave/recipes/webrtc/ios.expected/trybot.json

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/8dcc78f44c9e9d0e1857e927f62acf8a24df6cfd

commit 8dcc78f44c9e9d0e1857e927f62acf8a24df6cfd
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Jul 07 23:18:04 2016

I believe this is now fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/f1d779004fc7aefc230f4027aaf4655fe896982c

commit f1d779004fc7aefc230f4027aaf4655fe896982c
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jul 08 02:06:32 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/d4098d2dd3fb9ab846e3f31702afcb0843a4f4ac Fix configuration of goma in iOS recipes. (dpranke@chromium.org)
  https://crrev.com/01aec8c9c900d1a5ab3df82142ffc74db16c76cf Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
depot_tools:
  https://crrev.com/bf47014c1a87c15629c9cd6fd9b94b669d498a00 Added '.m' pattern to git cl format files to be covered. (zengster@google.com)

R=dpranke@chromium.org,smut@chromium.org,zengster@google.com,recipe-roller@chromium.org,martiniss@chromium.org,phajdan.jr@chromium.org
BUG= 626033 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2135473002

[modify] https://crrev.com/f1d779004fc7aefc230f4027aaf4655fe896982c/infra/config/recipes.cfg

EMEA Troopers only provide support for P0 issues. Please just use Infra-Troopers label and set priority accordingly: http://g.co/bugatrooper.