You're just wanting to try this out locally, correct?

+Ned

I think we want to run these on bots.

We actually want to run this on the bots.

We will want to add some additional things we want to test on the bots in subsequent steps; for example, the amount of overdraw. These are a little bit difficult (as in, unreliable/flaky) to measure using unit-tests. So our hope is that we can run these on the [perf?] bots, so that we can track possible regressions over time. Does that make sense? Are telemetry tests the right solution for this?

We don't have Chrome OS working in our standard telemetry infrastructure. There's a separate Chrome OS specific setup somewhere though?

I think bccheng@ has context here?

We shouldn't need to change the testing infrastructure to run tests on an ozone-x11 ChromeOS Chrome. This would be a huge step forward in testing Chrome for ChromeOS because we'd actually measure the impact of all ChromeOS UI code on smoothness, latency etc.

Some day though, we should have ChromeOS devices running the tests just like we have Android devices running the tests. 

mfomitchev@: I think you mean the former in this bug per our discussion yesterday? Can you clarify?

comment #6: Yup.

This sounds like it would need a custom benchmark harness, in the new benchmarking model.

Ned, where do you think this would fit?

1) I think we should first figure out which CHrome perf benchmarks best represent ChromeOs's performance. The list of benchmarks ChromeOS team was using few years ago is too out-dated. 

One that is addressed, this bug seems like mostly about setting ChromeOS infrastructure to run the benchmarks from (1) an ozone-x11 ChromeOS Chrome (is this a new ChromeOS device?) and not about new benchmarking harness.

+dpranke

We don't have plans to add bots for ozone-x11 ChromeOS to the perf waterfall currently. We're already a bit underwater as far as supporting too many configurations, so adding a new one isn't likely in the near term.

Maybe bccheng can comment on whether his team has infrastructure that could meet this need?

One thing I wanted to note: if there are resource constraints (in terms of too many configs to support), can we consider replacing the linux-x11 config, with chromeos-ozone-x11 config instead? With chromeos-ozone, we get test coverage of all of chromeos, including the chrome browser, and the system ui (ash). And, with the ozone-x11 backend, we also get coverage of the x11-specific parts of the code, most of which are shared between ozone-x11 and linux-x11 configs. So the results from the chromeos-ozone-x11 config should give a pretty close approximation to linux-x11. In addition to that, this also allows us to turn a run-time flag on to test mus+chrome (note that this does not need a separate build config; the default chromeos config supports the runtime flag for mus)

Switching the configs seems like a reasonable suggestion.

Looking at this again: if I'm following the suggestion properly, the idea in #c12 (and my reply in #c13) say that we would *stop* testing the configuration of Linux that we're actually shipping, and *start* testing the chromeos-ozone-x11 config instead.

Right?

I still think this is probably okay, but I'd want buy-in from the Linux desktop folks (rpop, jschuh, thomasanderson) as well as the perf folks. 

Also, eventually the linux-x11 code is going away and we'll only be using the ozone-x11 code on desktop as well, right? 

Assuming we do decide to make the change, I think you just need to adjust the GN flags used by the linux builder on the chromium.perf waterfall in mb_config.pyl:

https://cs.chromium.org/chromium/src/tools/mb/mb_config.pyl?l=322

> the idea in #c12 (and my reply in #c13) say that we would *stop*
> testing the configuration of Linux that we're actually shipping,
> and *start* testing the chromeos-ozone-x11 config instead.
> 
> Right?

That is what I am proposing, correct. The reason for that is chromeos-ozone-x11 should provide a close approximation for both chrome-linux-x11 (the existing telemetry perf tests have little x11 dependency, so the exact way we communicate with x11 should not have a significant impact on the results), and chromeos-ozone-drm, i.e. *two* configurations that we actually ship :-)

At the same time, chromeos-ozone-x11 would allow writing a new set of perf tests that allows perf-testing the window-manager of chromeos (ash), which is not possible with the current linux-x11 config.

> I still think this is probably okay, but I'd want buy-in from
> the Linux desktop folks (rpop, jschuh, thomasanderson) as well
> as the perf folks. 
> 
> Also, eventually the linux-x11 code is going away and we'll only
> be using the ozone-x11 code on desktop as well, right?

That I believe is the desire/plan, yes.

"a new set of perf tests that allows perf-testing the window-manager of chromeos (ash), which is not possible with the current linux-x11 config."

Make sure to talk to Ned about this at some point, as the benchmarking team is being fairly careful currently about adding new types of benchmarks.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/77c67de3a506c39bf84fec3f92585c507f16f29f

commit 77c67de3a506c39bf84fec3f92585c507f16f29f
Author: sadrul <sadrul@chromium.org>
Date: Tue Jun 06 17:29:06 2017

mash: Add catalog/manifest etc. for telemetry tests binary for mash.

For telemetry tests, the browser needs to get access to some test api
from the mus window server, which is only exposed to clients marked as
'test' in their manifest. Instead of making default chrome-browser such a
test client, create a separate telemetry-browser catalog, which is used
when run with --use-test-config command-line flag. Instead of adding
dependency on this test catalog from the main chrome executable, create a
separate test_chrome executable that can be used for running the
telemetry tests.

BUG= 704681 ,  721640 

Review-Url: https://codereview.chromium.org/2910383002
Cr-Commit-Position: refs/heads/master@{#477326}

[modify] https://crrev.com/77c67de3a506c39bf84fec3f92585c507f16f29f/chrome/BUILD.gn
[modify] https://crrev.com/77c67de3a506c39bf84fec3f92585c507f16f29f/chrome/app/BUILD.gn
[modify] https://crrev.com/77c67de3a506c39bf84fec3f92585c507f16f29f/chrome/app/chrome_main_delegate.cc
[modify] https://crrev.com/77c67de3a506c39bf84fec3f92585c507f16f29f/chrome/app/chrome_main_delegate.h
[add] https://crrev.com/77c67de3a506c39bf84fec3f92585c507f16f29f/chrome/app/chrome_test_exe_main_aura.cc
[modify] https://crrev.com/77c67de3a506c39bf84fec3f92585c507f16f29f/chrome/app/mash/BUILD.gn
[add] https://crrev.com/77c67de3a506c39bf84fec3f92585c507f16f29f/chrome/browser/chrome_test_browser_overlay.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/019bf69cc182113a1512ed0b57adca8aa59bf59c

commit 019bf69cc182113a1512ed0b57adca8aa59bf59c
Author: sadrul <sadrul@chromium.org>
Date: Fri Jun 09 16:41:18 2017

cros: Run telemetry tests on mojo-cros fyi bot.

Run telemetry_perf_unittests and telemetry_unittests on mojo-cros fyi
bot.

BUG= 704681 

Review-Url: https://codereview.chromium.org/2922333002
Cr-Commit-Position: refs/heads/master@{#478305}

[modify] https://crrev.com/019bf69cc182113a1512ed0b57adca8aa59bf59c/chrome/BUILD.gn
[modify] https://crrev.com/019bf69cc182113a1512ed0b57adca8aa59bf59c/chrome/test/BUILD.gn
[modify] https://crrev.com/019bf69cc182113a1512ed0b57adca8aa59bf59c/testing/buildbot/chromium.fyi.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0d63c437ac1df3511c4cf2a4a33a056dfb208360

commit 0d63c437ac1df3511c4cf2a4a33a056dfb208360
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Fri Jun 09 23:35:08 2017

Roll src/third_party/catapult/ 1ffc3ed05..4ef4b9509 (1 commit)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/1ffc3ed05267..4ef4b9509fe1

$ git log 1ffc3ed05..4ef4b9509 --date=short --no-merges --format='%ad %ae %s'
2017-06-09 sadrul telemetry: Add --browser-executable for 'exact'.

Created with:
  roll-dep src/third_party/catapult
BUG= 704681 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: Ic0fced7654e5605a247639995269c730227bd902
Reviewed-on: https://chromium-review.googlesource.com/529967
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#478456}
[modify] https://crrev.com/0d63c437ac1df3511c4cf2a4a33a056dfb208360/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/a876d977914a8dea80e3d77d2b0f317c9fe3035e

commit a876d977914a8dea80e3d77d2b0f317c9fe3035e
Author: Sadrul Habib Chowdhury <sadrul@chromium.org>
Date: Tue Nov 07 02:25:30 2017

Add a chromeos FYI perf bot for mus/viz services.

BUG=773774,  704681 

Change-Id: I8f9da0d3df13c4483ea2045bad4eeb9f9a89d182
Reviewed-on: https://chromium-review.googlesource.com/738357
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Sadrul Chowdhury <sadrul@chromium.org>

[modify] https://crrev.com/a876d977914a8dea80e3d77d2b0f317c9fe3035e/scripts/slave/recipe_modules/chromium_tests/chromium_perf_fyi.py
[modify] https://crrev.com/a876d977914a8dea80e3d77d2b0f317c9fe3035e/scripts/slave/recipe_modules/chromium_tests/tests/api/archive_build.expected/linux_builder.json
[modify] https://crrev.com/a876d977914a8dea80e3d77d2b0f317c9fe3035e/masters/master.chromium.perf.fyi/master.cfg
[modify] https://crrev.com/a876d977914a8dea80e3d77d2b0f317c9fe3035e/masters/master.chromium.perf.fyi/slaves.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7856ba9a1b02cadc8b8defd360b1fb5cdb2fbca0

commit 7856ba9a1b02cadc8b8defd360b1fb5cdb2fbca0
Author: Sadrul Habib Chowdhury <sadrul@chromium.org>
Date: Wed Nov 08 17:12:49 2017

mus/viz: Run smoothness.top_25_smooth tests on the fyi perf bot.

BUG= 704681 

Change-Id: I6b4483509d6b76370b6b4f8037840ba733d2da5f
Reviewed-on: https://chromium-review.googlesource.com/758217
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Sadrul Chowdhury <sadrul@chromium.org>
Cr-Commit-Position: refs/heads/master@{#514865}
[modify] https://crrev.com/7856ba9a1b02cadc8b8defd360b1fb5cdb2fbca0/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/7856ba9a1b02cadc8b8defd360b1fb5cdb2fbca0/tools/perf/chromium.perf.fyi.extras.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/dc25c1451a538dabfdaaeb8c71bdae941f8544bd

commit dc25c1451a538dabfdaaeb8c71bdae941f8544bd
Author: Sadrul Habib Chowdhury <sadrul@chromium.org>
Date: Fri Nov 10 20:55:26 2017

Rename the bot from mus/viz to mus-viz.

BUG= 704681 

Change-Id: I336a96ec99d1b663f67c9d2f62068a5658d70c96
Reviewed-on: https://chromium-review.googlesource.com/763729
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Sadrul Chowdhury <sadrul@chromium.org>

[modify] https://crrev.com/dc25c1451a538dabfdaaeb8c71bdae941f8544bd/scripts/slave/recipe_modules/chromium_tests/chromium_perf_fyi.py
[modify] https://crrev.com/dc25c1451a538dabfdaaeb8c71bdae941f8544bd/scripts/slave/recipe_modules/chromium_tests/tests/api/archive_build.expected/linux_builder.json
[modify] https://crrev.com/dc25c1451a538dabfdaaeb8c71bdae941f8544bd/masters/master.chromium.perf.fyi/master.cfg
[modify] https://crrev.com/dc25c1451a538dabfdaaeb8c71bdae941f8544bd/masters/master.chromium.perf.fyi/slaves.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4513e374e560ab6c00e2a62065e9f785353a3b45

commit 4513e374e560ab6c00e2a62065e9f785353a3b45
Author: Sadrul Habib Chowdhury <sadrul@chromium.org>
Date: Fri Nov 10 22:24:56 2017

[perf.fyi]: Rename the mus/viz bot to mus-viz.

BUG= 704681 
TBR=sky@chromium.org for trivial rename in tools/perf

Change-Id: Ie39d776f574b52b9c0e8454c8b09aec4ea44b0bf
Reviewed-on: https://chromium-review.googlesource.com/764451
Commit-Queue: Sadrul Chowdhury <sadrul@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#515715}
[modify] https://crrev.com/4513e374e560ab6c00e2a62065e9f785353a3b45/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/4513e374e560ab6c00e2a62065e9f785353a3b45/tools/perf/chromium.perf.fyi.extras.json

Removing as a blocker for 731255 and instead we'll track the time it takes for events to be processed (798775).

I believe all the blockers have been removed such that it is now possible to run telemetry with single-process-mash (use --extra-browser-args=--enable-features=SingleProcessMash). My understanding is Wu-Cheng and Chung-Yih have bots running telemetry on ChromeOS device. Could you guys add this config to your bots?

From the email thread it looked like vovoy was going to do this.

Vovoy, any update on this?

Added tests:
telemetry_Benchmarks.mash.cros_ui_smoothness
telemetry_Benchmarks.mash.memory.desktop
telemetry_Benchmarks.mash.rendering.desktop

The test results are available on:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/testDetails?testName=telemetry_Benchmarks.mash.cros_ui_smoothness
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/testDetails?testName=telemetry_Benchmarks.mash.memory.desktop
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/testDetails?testName=telemetry_Benchmarks.mash.rendering.desktop

Is there any other way for us to visualize this data and compare it against builds without mash enabled?

vovoy did the configuration changes, moving to fixed.