Well, this is there at least: 
https://webrtc.googlesource.com/src/+/refs/branch-heads/68

but the above thing looks for refs/remotes/branch-heads/68, and that one 404's.

I find it weird that the builder fails 75% of the time but succeeds every now and then.

If I look in my WebRTC repo, I do have a 68 branch in refs/remotes:
$ cat .git/refs/remotes/branch-heads/68
8d11d5ab93e14b7c6f4329b93ce14a02d452f2a8

If I run
$ git log 547e3169d9e059117de6a22f34c6e959707598e7..refs/remotes/branch-heads/

I get the commit I created on the branch:
Author: Patrik Höglund <phoglund@webrtc.org>
Date:   Thu May 24 09:33:44 2018 +0200

    Create WebRTC 68 branch for Chrome 68
    
    Bug: None
    Change-Id: Id325210536e9759fa94d89d992dbb08134b76d60
    Reviewed-on: https://webrtc-review.googlesource.com/78702
    Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org>
    Cr-Commit-Position: refs/branch-heads/68@{#1}
    Cr-Branched-From: 547e3169d9e059117de6a22f34c6e959707598e7-refs/heads/master@{#23360}

Which is exactly what I would expect. I have no idea what the problem here is. Is the bot not syncing --with_branch_heads or something?

According to the "bot_update" step the WebRTC repository is being cloned in its entirety (tags and branch-heads included). I'm not sure why it then seems to try to clone the repository from the git cache again in the "gclient_sync" step.

+mmoss in case it's got anything to do with either the DEPS file (the WebRTC repository points to a branch name, not a specific commit) or with infra in general.

I really have no idea what's wrong with that. I'm able to gclient sync M68 locally, and it gets the webrtc branch as expected, and as you noted, it appears that the bot_update step is able to sync and checkout that branch just fine too.

OK, I was finally able to reproduce the issue locally by following the exact steps the bot does. Namely, if I

1. sync and fetch m68 and deps
2. cd src/ && git clean -dffx
3. run gclient sync again

third_party/webrtc isn't checked out correctly. The trick here is that git clean -dffx needs to be run (I guess rm -fr third_party/webrtc directly would also work).

And yes, as far as I can see if I change src's DEPS and use a SHA1 hash instead of a branch name for webrtc everything works fine.

Ok, did we do something wrong when creating the branch or is the DEPS file incorrect?

Ultimately this looks like a gclient bug to me (even though I don't know why the bot's running git clean to begin with), but it'd help if you guys could use a hash instead of a branch name in the M68 DEPS file in the meantime (I can see that it's already done in M67 and M69).

OK, I think there are a couple problems here, and a couple ways to fix it.

The first issue is that after 'git clean' deletes that dir, the subsequent 'gclient sync' is run without "--with_branch_heads --with_tags", so when it reinitializes the repository, it's no longer configured to fetch branches and tags. Adding "--with_branch_heads --with_tags" to that sync step is probably the simplest fix. I was able to reproduce the error with the steps in #5, and then able to fix it by modifying the 'gclient sync' command like that.

The second issue is that, when given a branch name to fetch, gclient is trying to fetch 'refs/remotes/branch-heads/68', which is the local mapping for the branch name, but the server doesn't know that name, it only knows the canonical name 'refs/branch-heads/68' (as specified by the DEPS file). If gclient ran fetch with the proper branch name (i.e. 'git -c core.deltaBaseCacheLimit=2g fetch origin refs/branch-heads/68') it should work. That might be a more involved fix, since there's clearly logic somewhere in gclient to mess with the branch name, and that's probably needed for something.

Note, there's actually a third issue, which is that the release DEPS should point to a commit hash, like normal, and not to a branch ref. This was an error introduced by the m68 milestone process, and has since been fixed, but it's probably why this hasn't been a problem before now.

I can send a CL to change the gclient sync invocation so we can work around the other problems.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0b46e32f0301fb1a7bc98d0152f8c727432f7a45

commit 0b46e32f0301fb1a7bc98d0152f8c727432f7a45
Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Date: Wed May 30 17:32:46 2018

publish_tarball: Call gclient sync with --with_{branch_heads,tags}

It is not strictly necessary, but it helps get around some other failures
caused by the M68 branch having third_party/webrtc pointing at a branch name
instead of a commit hash.

While that is caused by other issues, calling gclient sync with
--with_branch_heads ensures we can at least work around any remaining bugs.

Bug:  847213 
Change-Id: I1d6bec9fd409c6ba1aa34ec8fccf5b0d5f0caff5
Reviewed-on: https://chromium-review.googlesource.com/1078823
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>

[modify] https://crrev.com/0b46e32f0301fb1a7bc98d0152f8c727432f7a45/recipes/recipes/publish_tarball.py
[modify] https://crrev.com/0b46e32f0301fb1a7bc98d0152f8c727432f7a45/recipes/recipes/publish_tarball.expected/basic.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/453e0ba33527ead966025609a8c40c6dbd5365d3

commit 453e0ba33527ead966025609a8c40c6dbd5365d3
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Wed May 30 22:15:15 2018

Increase timeout for publish_tarball

The publish_tarball bot is beginning to reach its 3hr limit:
https://ci.chromium.org/buildbot/chromium.infra.cron/publish_tarball/11115
https://ci.chromium.org/buildbot/chromium.infra.cron/publish_tarball/11109

This CL increases the timeout to 6hr.  The "Build From Tarball" bot is also
approaching its 3hr limit since one run was 2h42m:
https://ci.chromium.org/buildbot/chromium.infra.cron/Build%20From%20Tarball/253

This CL also increases the timout for that bot, since it's likely to start
seeing timeouts too as Chrome's build time increases.

BUG= 847213 
R=dpranke

Change-Id: Ic52138b19d5f64f3efc6862cdd42102cc255d75a
Reviewed-on: https://chromium-review.googlesource.com/1079701
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>

[modify] https://crrev.com/453e0ba33527ead966025609a8c40c6dbd5365d3/masters/master.chromium.infra.cron/master.cfg

The webrtc issue appears to be fixed.  Thank you raphael.kubo.da.costa@ for the fix!

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/2868b4f4ce92189c423a7304aefafc52652e9301

commit 2868b4f4ce92189c423a7304aefafc52652e9301
Author: Tom Anderson <thomasanderson@google.com>
Date: Wed May 30 22:29:26 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ddeaa86815de5c180a603c2a509d69d8cf260f34

commit ddeaa86815de5c180a603c2a509d69d8cf260f34
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Fri Jun 01 03:11:43 2018

publish_tarball: rebuild lite, nacl, and testdata tarballs if necessary

publsh_tarball used to only check if the full tarball was published, and if it
was, it would assume the lite, nacl, and testdata ones were published too.  This
assumption was recently violated since the nacl tarballs for specific versions
weren't getting published (since the bot was reaching its timeout and the nacl
tarballs get pushlished last).  The new behavior in this CL is to check each
tarball individually.

BUG= 847213 
R=mmoss

Change-Id: I476805da5a44c216e33d2cc953c1e189776645f3
Reviewed-on: https://chromium-review.googlesource.com/1081106
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Michael Moss <mmoss@chromium.org>

[modify] https://crrev.com/ddeaa86815de5c180a603c2a509d69d8cf260f34/recipes/recipe_modules/omahaproxy/api.py
[modify] https://crrev.com/ddeaa86815de5c180a603c2a509d69d8cf260f34/recipes/recipes/publish_tarball.expected/trigger.json
[modify] https://crrev.com/ddeaa86815de5c180a603c2a509d69d8cf260f34/recipes/README.recipes.md
[modify] https://crrev.com/ddeaa86815de5c180a603c2a509d69d8cf260f34/recipes/recipes/publish_tarball.py
[modify] https://crrev.com/ddeaa86815de5c180a603c2a509d69d8cf260f34/recipes/recipes/publish_tarball.expected/basic.json

[bulk-edit: disregard if N/A] Can the owner please set milestone to this bug if applicable?