Well I just fixed one error tracked in  issue 738441 . Which was failing more often.

The bots have now found a different error, which hasn't reproduced locally. I've filed  issue 743028  to track that. I'll also land a patch to remove the NOTREACHED which is throwing the error and slowing the CQ.

If that doesn't calm them down then I'll disable the tests.

I appreciate the complexity and efforts of fixing all the bugs. I am still wondering whether we can make all of these tests just FYI tests that don't block the CQ until they have proven to not fail in 2/3 of all cases.

We have a FYI bot which we use to see which new tests can be covered. However the FYI bot is always behind.

It's a tough call, as these tests have caught actual failures of the mus+ash project at check-in time. Conversely we have had them disabled before only for significant bugs to land which would have been caught.

However if this doesn't quickly catch up I will take them offline again.

Disabling the NOTREACHED submitted: https://chromium-review.googlesource.com/c/571274/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b18491dc7383250a16e2256ba451fa70930cb501

commit b18491dc7383250a16e2256ba451fa70930cb501
Author: Jonathan <jonross@chromium.org>
Date: Fri Jul 14 16:29:19 2017

Disable Compositor:DidFailToInitializeLayerTreeFrameSink NOTREACHED

One third of CQ runs of mash_browser_tests are failing when hitting the
NOTREACHED in Compositor:DidFailToInitializeLayerTreeFrameSink so I'm disabling
it while the cause is researched.

TBR=sadrul@chromium.org

TEST: mash_browser_tests
Bug:  742966 ,  743028 
Change-Id: I9363ebfbdeac594d26696b5426004cbafcec9c9b
Reviewed-on: https://chromium-review.googlesource.com/571274
Reviewed-by: Jonathan Ross <jonross@chromium.org>
Commit-Queue: Jonathan Ross <jonross@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486771}
[modify] https://crrev.com/b18491dc7383250a16e2256ba451fa70930cb501/ui/compositor/compositor.cc

On a related note, I've landed https://chromium-review.googlesource.com/c/571195/
which takes linux_chromium_chromeos_ozone_rel_ng out of the CQ. However, it also flips linux_chromium_chromeos_rel_ng from the old X11 config to Ozone, so the net change is that the Cros/X11 config is no longer in the CQ.

The CrOS/Ozone config is a core config for us, if it's unreliable fixing it needs to be a high priority.

 Issue 742989  has been merged into this issue.

Things still look pretty broken, so I'm going to remove the test suite from both cros bots (Linux ChromiumOS Tests (1) and Linux ChromiumOS Ozone Tests (1)), so that it'll fix both tryserver configs.

I had landed the change in #5 in hopes that the root cause of  issue 743028  could be addressed quickly.

However there is now a divergence between the compositors standard configuration, and how it is initialized in the mash world. There has not been work to maintain the mash path.

Due to this the compositor as a whole cannot be used in this suite, as various issues will show up.

I'm disabling the test in: https://chromium-review.googlesource.com/c/571503/

However I sent this in before the change in #6 I'll draw up another patch to disable it in the new config as well

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4fb42e5f2308168254a047df07173d8e78aa242

commit c4fb42e5f2308168254a047df07173d8e78aa242
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Jul 14 18:02:40 2017

Temporarily remove mash_browser_tests from CrOS bots.

It looks like they have several issues. This removes them from
both the "Linux ChromiumOS Ozone Tests (1)" builder and the
"Linux ChromiumOS Tests (1)" builder, so it should stop breaking
the CQ builds.

TBR=jonross@chromium.org, kbr@chromium.org, qyearsley@chromium.org
NOTRY=true
BUG= 742966 

Change-Id: Ie96885593e40b1ff62e76dc1b09aa01d15e20529
Reviewed-on: https://chromium-review.googlesource.com/572079
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486805}
[modify] https://crrev.com/c4fb42e5f2308168254a047df07173d8e78aa242/testing/buildbot/chromium.chromiumos.json

With #11 this test suite is off of the waterfall.

I'll update the master  issue 678687  which tracks the state of the test suite.

Sorry for trashing the bot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1482be49c36f85a5de865899a5de759489112c7c

commit 1482be49c36f85a5de865899a5de759489112c7c
Author: Jonathan Ross <jonross@chromium.org>
Date: Fri Jul 14 18:08:49 2017

Revert "Disable Compositor:DidFailToInitializeLayerTreeFrameSink NOTREACHED"

This reverts commit b18491dc7383250a16e2256ba451fa70930cb501.

Reason for revert:  Viz currently does not support the mash configuration. Due to this the compositor cannot be used in mash_browser_tests. The tests have been disabled on the waterfall. See  crbug.com/742966 

Original change's description:
> Disable Compositor:DidFailToInitializeLayerTreeFrameSink NOTREACHED
> 
> One third of CQ runs of mash_browser_tests are failing when hitting the
> NOTREACHED in Compositor:DidFailToInitializeLayerTreeFrameSink so I'm disabling
> it while the cause is researched.
> 
> TBR=sadrul@chromium.org
> 
> TEST: mash_browser_tests
> Bug:  742966 ,  743028 
> Change-Id: I9363ebfbdeac594d26696b5426004cbafcec9c9b
> Reviewed-on: https://chromium-review.googlesource.com/571274
> Reviewed-by: Jonathan Ross <jonross@chromium.org>
> Commit-Queue: Jonathan Ross <jonross@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#486771}

TBR=sadrul@chromium.org,jonross@chromium.org

Change-Id: Ic7eb38c23b6330dc27d5fddb0e546b83ec08b6d9
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  742966 ,  743028 
Reviewed-on: https://chromium-review.googlesource.com/572003
Reviewed-by: Jonathan Ross <jonross@chromium.org>
Commit-Queue: Jonathan Ross <jonross@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486808}
[modify] https://crrev.com/1482be49c36f85a5de865899a5de759489112c7c/ui/compositor/compositor.cc