Looks to be at least two issues:
 - 30+ minute swarming task overhead, for which I'll file a separate bug.
 - some very flaky tests, for which we should use this bug.

I'm dropping the suite to experimental.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7bfc1624155dbb535fe2ff49ddb483f48968b6e8

commit 7bfc1624155dbb535fe2ff49ddb483f48968b6e8
Author: John Budorick <jbudorick@chromium.org>
Date: Fri Jul 27 14:13:21 2018

Drop the layout tests to experimental on Mac10.13 Tests.

TBR=dpranke@chromium.org

Bug:  868180 
Change-Id: Ic9cb48f82954f102f60dd3e1b9c29bcca209f3bb
Reviewed-on: https://chromium-review.googlesource.com/1152566
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578626}
[modify] https://crrev.com/7bfc1624155dbb535fe2ff49ddb483f48968b6e8/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/7bfc1624155dbb535fe2ff49ddb483f48968b6e8/testing/buildbot/test_suite_exceptions.pyl

mac_chromium_rel_ng is actually running the layout tests on 10.12. The 30+ min delay is a known (open) bug, bug 851355. We believe that shifting the tests to 10.13 (which we should do anyway) should make that part go away, at least we haven't seen the delays on 10.13.

I was working on shifting the traffic in https://crrev.com/c/1111030, but got distracted for some reason that I've forgotten at the moment, and then I went on vacation. I will pick this CL back up today and see if we can make some progress.

Do we have a specific reason to believe that switching to 10.13 will fix the 30+ minute overhead?

Mostly that we haven't seen the delays on 10.13 bots. We're not sure if it'd due to 10.13 vs. 10.12, or VMs vs. minis, or what, and I don't know that we have a big enough data set to be sure we don't see it.

That's 10.13 bots on chromium.webkit, I assume?

Mac10.13 Tests (dbg) on chromium.mac, Chromium Mac 10.13 on chromium.fyi, WebKit Mac 10.13 (retina) on chromium.webkit, I think. 

into the sheriff queue now that the suite has been dropped to experimental. we need to identify why these started failing & find an owner.

I can take investigate the mac failure if this bug is not urgent. I am still a bit swarmed with perf benchmarking/Telemetry stuffs

I think P1 is correct; it's not blocking the CQ any more, but it's a pretty significant loss of coverage on the CQ.

 Issue 869051  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d88b90701038dc35f256565078358d5bc19862c

commit 1d88b90701038dc35f256565078358d5bc19862c
Author: John Budorick <jbudorick@chromium.org>
Date: Tue Jul 31 02:25:33 2018

Revert "Drop the layout tests to experimental on Mac10.13 Tests."

This reverts commit 7bfc1624155dbb535fe2ff49ddb483f48968b6e8.

Reason for revert: tests appear to have mostly recovered; unclear why.

Original change's description:
> Drop the layout tests to experimental on Mac10.13 Tests.
> 
> TBR=dpranke@chromium.org
> 
> Bug:  868180 
> Change-Id: Ic9cb48f82954f102f60dd3e1b9c29bcca209f3bb
> Reviewed-on: https://chromium-review.googlesource.com/1152566
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: John Budorick <jbudorick@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#578626}

TBR=dpranke@chromium.org,jbudorick@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  868180 
Change-Id: Ibb217f63bb5b834565fafac4cb7ad15543ba70bb
Reviewed-on: https://chromium-review.googlesource.com/1155614
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Peter Boström <pbos@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Peter Boström <pbos@chromium.org>
Cr-Commit-Position: refs/heads/master@{#579286}
[modify] https://crrev.com/1d88b90701038dc35f256565078358d5bc19862c/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/1d88b90701038dc35f256565078358d5bc19862c/testing/buildbot/test_suite_exceptions.pyl

Tentatively closing this as fixed, the bots seem to have recovered.

I don't this this bug can be closed until we take the suite of out experimental mode

#16: that's what #14 did.