chromium.fyi and chromium.webrtc.fyi are different waterfalls. I'll have a look shortly (in meeting).

Can you find any similar error in build.chromium.org/p/chromium.linux/waterfall or build.chromium.org/p/chromium.android/waterfall ?

I'm compared the last green and first red build and came up with: https://chromium.googlesource.com/chromium/src/+log/11da5fe6e7d548ea5c471f48a3caf6083ee19afe..f105471106f4c821364165bdd991ce13196f5496
that indicates https://chromium.googlesource.com/chromium/src/+/f5a497cd5ce83519a9c71bd1ef877f85ec022d38 is the culprit. I'll ping the author and ask if there's a workaround (since I can confirm this is only hitting our bots).

I have a theory of why this is failing only on our bots: as part of the work I did on bug 657315 our builders that builds the test for these bots only compiles the content_browsertests + content_browsertests_apk targets. 

I believe this means that there's a missing dependency introduced by https://chromium.googlesource.com/chromium/src/+/f5a497cd5ce83519a9c71bd1ef877f85ec022d38 that is not detected by the regular Android bots in https://build.chromium.org/p/chromium.linux/waterfall or https://build.chromium.org/p/chromium.android/waterfall since they build a much larger set of targets (and archive huge amounts of redundant data to Google storage because of that).

I'll land a CL adding 'service_manager' to our build targets to see if that works around the problem for now (based on https://cs.chromium.org/chromium/src/services/service_manager/BUILD.gn).

I was derailed by meetings and the presubmit check but now I have a CL in the CQ for adding service_manager_unittests to the targets being built: https://codereview.chromium.org/2655653002/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c6f2c39f80a9238f90e3d425b844538e301302dc

commit c6f2c39f80a9238f90e3d425b844538e301302dc
Author: kjellander <kjellander@chromium.org>
Date: Tue Jan 24 15:39:34 2017

Add compiling of service_manager_unittests as temporary workaround.

This will hopefully resolve the error WebRTC Android bots are seeing
when trying to run content_browsertests.

BUG= 684406 
TBR=aleloi@chromium.org
NOTRY=True

Review-Url: https://codereview.chromium.org/2655653002
Cr-Commit-Position: refs/heads/master@{#445733}

[modify] https://crrev.com/c6f2c39f80a9238f90e3d425b844538e301302dc/testing/buildbot/chromium.webrtc.fyi.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3d4141d40471d274b0b66d2e938821137bcc0f9c

commit 3d4141d40471d274b0b66d2e938821137bcc0f9c
Author: kjellander <kjellander@chromium.org>
Date: Tue Jan 24 16:20:41 2017

Revert of Add compiling of service_manager_unittests as temporary workaround. (patchset #1 id:1 of https://codereview.chromium.org/2655653002/ )

Reason for revert:
Turns out another revert greened up this build. This is no longer needed.

Original issue's description:
> Add compiling of service_manager_unittests as temporary workaround.
>
> This will hopefully resolve the error WebRTC Android bots are seeing
> when trying to run content_browsertests.
>
> BUG= 684406 
> TBR=aleloi@chromium.org
> NOTRY=True
>
> Review-Url: https://codereview.chromium.org/2655653002
> Cr-Commit-Position: refs/heads/master@{#445733}
> Committed: https://chromium.googlesource.com/chromium/src/+/c6f2c39f80a9238f90e3d425b844538e301302dc

TBR=aleloi@chromium.org,rockot@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 684406 

Review-Url: https://codereview.chromium.org/2650233002
Cr-Commit-Position: refs/heads/master@{#445740}

[modify] https://crrev.com/3d4141d40471d274b0b66d2e938821137bcc0f9c/testing/buildbot/chromium.webrtc.fyi.json

Another revert greened up this build. 

FYI I've filed issue 684594 to try to get at the root of the issue.

Now something similar has started again. The same list of bots are failing (https://build.chromium.org/p/chromium.webrtc.fyi/console, https://build.chromium.org/p/chromium.webrtc.fyi/builders/Android%20Tests%20(dbg)%20(K%20Nexus5) ).

The missing file is different: 

  File "/b/c/b/Android_Tests__dbg___K_Nexus5_/src/third_party/catapult/devil/devil/android/device_utils.py", line 1357, in _PushFilesImpl
    for h, _ in host_device_tuples)
  File "/b/c/b/Android_Tests__dbg___K_Nexus5_/src/third_party/catapult/devil/devil/android/device_utils.py", line 1357, in <genexpr>
    for h, _ in host_device_tuples)
  File "/b/c/b/Android_Tests__dbg___K_Nexus5_/src/third_party/catapult/devil/devil/utils/host_utils.py", line 10, in GetRecursiveDiskUsage
    running_size = os.path.getsize(path)
  File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '/b/c/b/Android_Tests__dbg___K_Nexus5_/src/out/Debug/gen/services/shape_detection/manifest.json'

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ccb90539d5af550d7e753984fc79efd62f6495f9

commit ccb90539d5af550d7e753984fc79efd62f6495f9
Author: rockot <rockot@chromium.org>
Date: Wed Jan 25 17:44:45 2017

Remove all service manifests from data_deps

Manifest outputs are never needed at runtime, so these entries
in data_deps are inaccurate. They also break stuff now on some
(non-tree-closing) builders.

This removes all references to service manifest rules in
data_deps.

BUG= 684406 
TBR=ben@chromium.org

Review-Url: https://codereview.chromium.org/2652273002
Cr-Commit-Position: refs/heads/master@{#446052}

[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/components/filesystem/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/components/leveldb/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/components/nacl/broker/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/browser/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/catalog_viewer/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/example/views_examples/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/example/window_type_launcher/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/session/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/simple_wm/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/task_viewer/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/mash/webtest/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/media/mojo/services/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/service_manager/README.md
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/service_manager/background/tests/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/service_manager/runner/host/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/service_manager/tests/connect/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/service_manager/tests/lifecycle/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/service_manager/tests/service_manager/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/shape_detection/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/ui/demo/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/ui/gpu/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/ui/test_wm/BUILD.gn
[modify] https://crrev.com/ccb90539d5af550d7e753984fc79efd62f6495f9/services/video_capture/BUILD.gn