We have gotten away with this because the gap between unit/browser test coverage in the chromium waterfall and autotest coverage in the PFQ and pfq-informational builders is small, but I 100% agree that we should have a linux-chromeos builder running unit/browser tests on official builds in the chromeos.chrome waterfall:
https://uberchromegw.corp.google.com/i/chromeos.chrome

Note: we do build simple chrome with --internal in chrome-tot-chromeos-amd64-generic in that waterfall, but it does not currently run VM tests:  issue 724327 .

Do we have regular browser tests (using the field trial test config) running for ChromeOS? They should be running on regular non-official builds.

@sky - are you talking about running binaries on Chromebooks (or CrOS VMs), or on regular desktop builds?

bpastene has been working on getting tests running on both on upstream bots in bug 732531 (and some related issues). I think this is the big ticket item and would probably block any additional work we'd do for official configs in this bug.

I'm not sure if it makes sense to add bots to official.desktop, given that the chrome builders don't actually do the official builds of chrome, the chromeos builders do.

We should at least have one (either VM or desktop ChromeOS build). IMO the VM approach seems ideal given there are a fair number of conditionals in the code as to whether running on actual hardware or not.

1) We do build Simple Chrome in chrome-tot-chromeos-amd64-generic and there are plans to run tests there eventually ( issue 724327 ).

2) We *do not run unit or browser tests on a device* anywhere. Possibly we should but that is a much much larger discussion.

My understanding is that we build and run unit and browser tests for official chrome builds on official.desktop/waterfall. We should have a linux-chromeos representative there (or somewhere, maybe an FYI waterfall). Currently we do run unit or browser tests anywhere for chrome (i.e. not chromium) with chromeos=1 on linux.

> 2) We *do not run unit or browser tests on a device* anywhere. Possibly we
> should but that is a much much larger discussion.

Doing so is very much part of what I want us to be doing upstream on the chromium waterfalls.

> 2) We *do not run unit or browser tests on a device* anywhere. Possibly we should but that is a much much larger discussion.

We do run select unittests on devices in the chrome os waterfall by wrapping an autotest around them, for instance security_SandboxLinuxUnittests runs sandbox_linux_unittests, We also run some video encode/decode unittests in a similar manner. It's not difficult to add more unittests, and the coverage would include both chrome and chromium. However, these don't help chromium devs much since the failures would be in the chromeos PFQ, long after the changes have landed in chromium.

But Dirk/Ben are planning on running unittests in the cros VM in the chromium waterfall as a part of the simple chrome VM effort.

We don't run any browser tests on devices, and probably shouldn't - browser tests have a weird environment quite incompatible with the cros system environment, they stub out cros services, and the maintenance effort to keep these tests green in the past hasn't been worth it. Autotests (and in the future Tast) are our preferred vehicle for integration tests on chromeos.

> My understanding is that we build and run unit and browser tests for official chrome builds on official.desktop/waterfall. We should have a linux-chromeos representative there (or somewhere, maybe an FYI waterfall). Currently we do run unit or browser tests anywhere for chrome (i.e. not chromium) with chromeos=1 on linux.

I thought linux-chromeos-rel does this, but I'm not sure?
https://uberchromegw.corp.google.com/i/chromium.chromiumos/builders/linux-chromeos-rel/

> I thought linux-chromeos-rel does this, but I'm not sure?
> https://uberchromegw.corp.google.com/i/chromium.chromiumos/builders/linux-chromeos-rel/

That's chromium, not chrome.

I assume the original request was to have a bot that builds official chrome for chromeos and runs tests on linux desktop, not on VM?
I thought we already have it (as we have or had a bot that runs tests on branch), but looks like we dont :( I agree that this is P1. 

Re. running all (or most of) tests on device: While this would be great if we can, this can be very tricky and should probably be tracked in a separate bug.

1) There are difference in code path (this is intentional) between device and non device, which the test relies on (explicitly or implicitly).
  Updating tests to make sure that tests passes on both configurations may require quite a lot of work.
2) It should be a part of CQ, which means that we need to implement swaming on devices.
3) When this fails only on the device, it could be difficult for non cros engineer to troubleshoot, even if the fix could be very simple.

I'm trying to add a finch trial. https://g3doc.corp.google.com/analysis/uma/g3doc/finch/guide.md?cl=head has a reference to the buildbots that run the test config. They are all desktop official builders. AFAICT I can't find an official ChromeOS builder that runs the tests, hence this bug.

This appears to be related to internal waterfall, which mmoss@ knows more about.

I'm not very (or at all) familiar with the cros builders, but is this basically asking for the builder that was removed in https://chrome-internal-review.googlesource.com/c/chrome/tools/build/+/471693 ( Issue 764571 )?

I don't think that builder was doing quite what this issue is requesting (either that or it was poorly named).

This issue is requesting a builder that:
* Builds official Chrome on Linux with chromeos=1
* Runs all unit and browser tests

i.e. the builder would be the same as https://ci.chromium.org/buildbot/chromium.chromiumos/linux-chromeos-rel/ except with an internal checkout and with extra GN args: is_official_build = true (and probably is_chrome_branded = true and proprietary_codecs = true).

So maybe something like:
https://luci-milo.appspot.com/buildbot/official.desktop.continuous/linux64%20trunk/8320

but with "chromeos=1"? Or do you specifically not want it to run all those other tests too? Should it be a continuous official builder (like the one I linked) or only for release build?

A continous builder would be best.

I'm not sure what "all those other tests" refers to, but it should ideally run all browser/unit tests (including some that are chromeos specific, e.g. chromeos_unittests and ash_unittests). i.e. the tests run by linux-chromeos-rel in comment #14.

OK, I wasn't sure if the references to browser/unit tests were referring to the literal "browser_tests" and "unit_tests" steps (those generic names are not at all confusing).

:) Yes, sorry, I wasn't very clear. We would like to run the "complete set" of tests that we run in linux-chromeos-rel (including e.g. "interactive ui tests").
Thanks!

Ben, are you the right person to take care of this?