I landed the first CL for exo performance test [1]. It uses AshTestBase, but the AshTestBase uses InProcessContextFactory for cash and the AshTestContextFactory for mus and mash. Both InProcessContextFactory and AshTestContextFactory are quite different from the real environment (no separate GPU process or viz process, mojo IPC is not used for mus and ash, etc). So to make perf test useful, I think we need modify AshTestBase or implement a new TestBase to setup an environment more like the real environment.

[1] https://chromium.googlesource.com/chromium/src/+/059a6f74910c4fe167d1559b49db3158ef58cd5d

As you have found, AshTestHelper effectively stubs out the mojo calls to the ui service. AppLaunchTest.TestQuickLaunch does not. Unfortunately TestQuickLaunch doesn't actually run in ash. That may be ok for performance though. Do you really need the test to run in ash?

The performance test runs wayland server(exo) and wayland client in different threads in one process, and the exo is using ash, so the performance test needs ash.

I will check the AppLaunchTest.TestQuickLaunch. Maybe we can do similar things in AshTestBase.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/712a2452d02a4548f20aa4f7e45092af88ccc1fd

commit 712a2452d02a4548f20aa4f7e45092af88ccc1fd
Author: Peng Huang <penghuang@chromium.org>
Date: Wed Sep 27 16:24:25 2017

exo: run wayland_client_perftests on Mojo ChromiumOS

Bug:  765393 
Change-Id: I0b47f6862e9190e9ee01248fefa0c33a3ccf6923
Reviewed-on: https://chromium-review.googlesource.com/685501
Reviewed-by: Scott Violet <sky@chromium.org>
Reviewed-by: David Reveman <reveman@chromium.org>
Commit-Queue: Peng Huang <penghuang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#504699}
[modify] https://crrev.com/712a2452d02a4548f20aa4f7e45092af88ccc1fd/components/exo/wayland/clients/test/wayland_client_test.cc
[modify] https://crrev.com/712a2452d02a4548f20aa4f7e45092af88ccc1fd/components/exo/wayland/clients/test/wayland_client_test.h
[modify] https://crrev.com/712a2452d02a4548f20aa4f7e45092af88ccc1fd/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/712a2452d02a4548f20aa4f7e45092af88ccc1fd/testing/buildbot/gn_isolate_map.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bda41dd334dbe0f8c0657948a7f0febd79ed9d90

commit bda41dd334dbe0f8c0657948a7f0febd79ed9d90
Author: Peng Huang <penghuang@chromium.org>
Date: Fri Oct 13 22:27:36 2017

exo: Add a way to run the perftest with external wayland server.

Bug:  765393 
Change-Id: Ib91ab443f27a8797a2b0786571711a28140c3ccd
Reviewed-on: https://chromium-review.googlesource.com/713958
Commit-Queue: Peng Huang <penghuang@chromium.org>
Reviewed-by: David Reveman <reveman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#508848}
[modify] https://crrev.com/bda41dd334dbe0f8c0657948a7f0febd79ed9d90/components/exo/wayland/clients/test/run_all_client_perftests.cc
[modify] https://crrev.com/bda41dd334dbe0f8c0657948a7f0febd79ed9d90/components/exo/wayland/clients/test/wayland_client_test.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/db1e00479a9432587179fe438a7b0e733ac611f7

commit db1e00479a9432587179fe438a7b0e733ac611f7
Author: Peng Huang <penghuang@chromium.org>
Date: Mon Oct 16 21:10:56 2017

exo: ClientBase only create message loop if necessary

When clients are running as perftests, the message loop
will created by the test suite. In that case, we don't
need create it in the ClientBase.

Bug:  765393 
Change-Id: I7e19bc177c157262806f88783c1756c7643ec158
Reviewed-on: https://chromium-review.googlesource.com/721604
Reviewed-by: David Reveman <reveman@chromium.org>
Commit-Queue: Peng Huang <penghuang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#509170}
[modify] https://crrev.com/db1e00479a9432587179fe438a7b0e733ac611f7/components/exo/wayland/clients/client_base.cc
[modify] https://crrev.com/db1e00479a9432587179fe438a7b0e733ac611f7/components/exo/wayland/clients/client_base.h
[modify] https://crrev.com/db1e00479a9432587179fe438a7b0e733ac611f7/components/exo/wayland/clients/rects.cc
[modify] https://crrev.com/db1e00479a9432587179fe438a7b0e733ac611f7/components/exo/wayland/clients/simple_main.cc
[modify] https://crrev.com/db1e00479a9432587179fe438a7b0e733ac611f7/components/exo/wayland/clients/subsurface.cc
[modify] https://crrev.com/db1e00479a9432587179fe438a7b0e733ac611f7/components/exo/wayland/clients/yuv.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a6cbbda22241958caae4f266818e2d54ba0d51eb

commit a6cbbda22241958caae4f266818e2d54ba0d51eb
Author: Peng Huang <penghuang@chromium.org>
Date: Thu Oct 26 18:35:02 2017

exo run wayland_client_perftests on Linux ChromiumOS Tests (1)

Bug:  765393 
Change-Id: I0ea47f82222ea45cd0542b40dd138dcb66ad74ff
Reviewed-on: https://chromium-review.googlesource.com/726479
Reviewed-by: Scott Violet <sky@chromium.org>
Commit-Queue: Peng Huang <penghuang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#511894}
[modify] https://crrev.com/a6cbbda22241958caae4f266818e2d54ba0d51eb/testing/buildbot/chromium.chromiumos.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc

commit ebe366eb7748e43ff05a56c9fa30e90b34eaabbc
Author: David Reveman <reveman@chromium.org>
Date: Mon Oct 30 16:49:46 2017

exo: Improve simple client test by adding double-buffering

This test now measures performance when using double-buffering. Any
problems related to the release of buffers can cause this test to drop
frames.

Bug:  765393 
Change-Id: I3c0bbd678e48b0601e3b420ac0ff4c3f4a115211
Reviewed-on: https://chromium-review.googlesource.com/743685
Commit-Queue: David Reveman <reveman@chromium.org>
Reviewed-by: Peng Huang <penghuang@chromium.org>
Reviewed-by: Daniele Castagna <dcastagna@chromium.org>
Cr-Commit-Position: refs/heads/master@{#512518}
[modify] https://crrev.com/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc/components/exo/wayland/clients/client_base.cc
[modify] https://crrev.com/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc/components/exo/wayland/clients/client_base.h
[modify] https://crrev.com/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc/components/exo/wayland/clients/perftests.cc
[modify] https://crrev.com/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc/components/exo/wayland/clients/rects.cc
[modify] https://crrev.com/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc/components/exo/wayland/clients/simple.cc
[modify] https://crrev.com/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc/components/exo/wayland/clients/vulkan.cc
[modify] https://crrev.com/ebe366eb7748e43ff05a56c9fa30e90b34eaabbc/components/exo/wayland/clients/yuv.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0982a27e80b5712fc5dff6feb03858185656831b

commit 0982a27e80b5712fc5dff6feb03858185656831b
Author: Peng Huang <penghuang@chromium.org>
Date: Mon Nov 27 22:57:40 2017

exo: measure latency in wayland client simple test.

Bug:  765393 
Change-Id: I5c83d87c259b1f4884e91f9cc0f4bc3a1baf4d36
Reviewed-on: https://chromium-review.googlesource.com/772398
Commit-Queue: Peng Huang <penghuang@chromium.org>
Reviewed-by: David Reveman <reveman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#519437}
[modify] https://crrev.com/0982a27e80b5712fc5dff6feb03858185656831b/components/exo/wayland/clients/perftests.cc
[modify] https://crrev.com/0982a27e80b5712fc5dff6feb03858185656831b/components/exo/wayland/clients/simple.cc
[modify] https://crrev.com/0982a27e80b5712fc5dff6feb03858185656831b/components/exo/wayland/clients/simple.h

Migrating Proj-Mustash-Mus to components Internals>Services>WindowService and Internals>Services>Ash

Peng, I know you haven't worked on this in a while, but is there anything left to do? Where we ended up is ash is exactly the same as it was in classic mode. So, I suspect performance with mash and without is exactly the same. It would be nice to have some exo tests, but at this point I don't think it impacts mash. I'm going to remove single-process-mash. Peng, feel free to close this out.

I implemented several exo performance tests, but we are not running and recording result for them, since reveman told me those perftests are not very useful.

And I know ARC++ team has their own perf tests from CTS. And those tests are running in bots. So I think it is not necessary to have our own perf tests anymore.