This is fallout from the fixes for bug 669297, where I took the CrOS CQ builders out of the CQ "temporarily". Then I forgot to re-add them before closing that bug.

I'll take ownership of this bug and use it to track re-adding the builders:

- chromeos_x86-generic_chromium_compile_only_ng
- linux_chromium_chromeos_compile_dbg_ng
- linux_chromium_clobber_rel_ng

Interestingly, this is only the second or third report I've gotten of an issue since they were removed on 11/29, which suggests that they are providing little value.

Do they provide little value if they prevent tree-closing changes from slipping through CQ? 

That sounds like a rate of one report every two weeks, with a large chunk of that time having low throughput due to the holidays. Avoiding an additional tree closure every two weeks seems high value to me! Maybe I'm not calibrated yet though?

I will note that I only reported this at kbr@'s suggestion, otherwise I likely would have shrugged and moved along. So much (most?) of the time it probably goes unreported. But it sounds like it's being taken care of so it I guess we're all basically in agreement.

> Do they provide little value if they prevent tree-closing changes from 
> slipping through CQ? 

That is the right question :). 

I don't actually know what the right answer is yet, i.e., I'm not calibrated yet, either ...

In my opinion, preventing tree closing compile breakage is a top priority. Even if there have been only a few such examples recently, preventing them by adding back the CQ bots should be done as long as there is capacity.

@kbr - duly noted :).

 Issue 723760  has been merged into this issue.

The one builder that we really really want is:

linux_chromium_chromeos_dbg_ng

We'll take linux_chromium_chromeos_compile_dbg_ng, but sometimes tests do fail on debug builds only.

I would say that once every week or two the "Linux ChromiumOS Builder (dbg)" builder on the main Chromium waterfall breaks because we do not have any trybot coverage.

Unfortunately that builder appears to take an average of 2 hours which surprises me. The compile time alone is ~ 1 hour 15 minutes. Can we not use goma for these?

We don't run debug tests in the CQ; they are typically too slow.

We should be running linux_chromium_chromeos_rel_ng , which should have DCHECKS enabled, which should catch most of the failures.

In addition, we should be running chromeos_x86-generic_chromium_compile_only_ng and linux_chromium_chromeos_compile_dbg_ng but those are compile-only, obviously.

The other CrOS configs currently in the CQ are:
- chromeos_amd64-generic_chromium_compile_only_ng
- chromeos_daisy_chromium_compile_only_ng
- linux_chromium_chromeos_ozone_rel_ng

I think the last one is probably testing an uninteresting config at this point, but it'd be great if you could confirm that. 

We've also had a request to add the linux_chromium_ozone_compile_only_ng (and or linux_chromium_ozone_rel_ng), which isn't a CrOS config, but does overlap with it a fair amount, and so we probably need to de-dup at least one of these ...

At our mustash meeting this week we decided linux_chromium_chromeos_ozone_rel_ng and linux_chromium_chromeos_rel_ng can be collapsed (we're ozone-everywhere as of last week). Someone is going to audit the test suites and make sure we end up with the full list running on the final bot -- rjkroege, do we have a bug for that?

RE: Not running Debug tests in the CQ - That makes sense and sounds reasonable as long as we are at least building the targets, which presumably we should be doing in linux_chromium_chromeos_compile_dbg_ng.

James covered the ozone situation.

The amd64 and daisy builders are Simple Chrome builders which build Chrome for those HW targets and are important to keep in the CQ; we are planning to add a VMTest stage to those to reduce breakage of the Chrome OS PFQ.

RE: Comment 11,  http://crbug.com/671355  is the bug that you're thinking of.

I think we can close this now. The Linux Ozone situation is covered in  bug 694033 .

We still don't have debug CrOS coverage, but it's not clear that we need it given the above and the frequency of breakages.