Is this something that a trybot can handle? Is Android often affected by similar regressions? If now, how is that prevented on other platforms? This is something we have allocated resources to look into and fix in Q1.

On Android, we prevent benchmark failures with trybots added to chromium CQ. We have a smoke test in telemetry_perf_unittests that runs an abbreviated version of each benchmark on the device.

Thanks for the info Annie. If we want to replicate the Android setup for Chromebooks (ie set up one x86 and one ARM device), is there someone in your group or the Chrome speed infra team that we can consult with for topics like how to provision the devices and update frequency, etc?

I think you want to talk to Chrome infra team about that. Adding dpranke who can help start the conversation. Dirk: right now we are running these tests on chromium CQ for every CL. I'm not sure what the state of CrOS trybots is, and how feasible it is to set one up to run telemetry_perf_unittests?

Can this be run on the "desktop ChromeOS" builds, or does this need to run on real boards? Currently we have no real boards in the CQ, and adding them would be expensive.

The desktop ChromeOS build only emulates the x86 environment, so it has a testing hole on ARM platforms. Also we have had some regressions in the ChromeOS OOBE login phase and it seems that they can only be caught by using real devices in the CQ. 

I'm aware of the differences in environments :).

You mentioned that adding real boards for ChromeOS will be expensive, and I assume you mean lab space and maintenance. Probably there are more issues I don't see so please let us know what else needs to be covered. We'd like to share the loads to make it happen as we have spent more time in triaging/bisecting builds to find the offending change while the actual fix is often trivial.

can we use machines from the ChromeOS labs for this? 
Chrome infra bots cannot access the CHromeOS lab?

I'm happy to discuss what might be required to get CrOS builders into the CQ. Please feel free to set up a meeting or something for this.

I didn't, however, see an answer to my question: if we can catch regressions in catapult by running the tests on https://build.chromium.org/p/chromium.chromiumos/waterfall?show=Linux%20ChromiumOS%20Tests%20(1) and the matching CQ builders, we can more-or-less do that immediately. Is that useful?

Thanks Dirk! I will visit MTV from 2/7 for a week and I'll schedule a meeting to learn more about the builder setup.

I think running Catapult tests using the Linux ChromeOS builder won't exercise the ChromeOS path as there is code which checks the host version and it will think it's running on Linux. Achuith@, is my understanding correct?

Dirk and I met yesterday to discuss this in more details. Here is a rough plan:

- Have two pools of up to 30 Chromebooks in each pool for x86 and ARM respectively for continuous builds.
- Use the simple chrome build flow to deploy Chrome to Chromebooks.

Dirk will raise the issue in upcoming chrome-infra meeting and we will find out what the best way is to provision Chromebooks.

When you say "for continuous builds", you mean for a waterfall? Or for the commit queue? Ideally we can catch problems before landing.

We discussed the possibilities for either.