xidachen@, junov@:

Based on discussion with danakj@, weiliangc@, the problem with OffscreenCanvas.commit() layout tests are not the async nature of commit() as what we thought before. 

In fact, the biggest problem is that the compositor used in layout tests mode is a different one than that in real world. To be more specific, in real world, RenderThreadImpl returns an OutputSurface here (https://cs.chromium.org/chromium/src/content/renderer/render_thread_impl.cc?rcl=0&l=1918), whilst in layout test mode, RenderThreadImpl returns a different OutputSurface created from LayoutTestDependencies (https://cs.chromium.org/chromium/src/content/renderer/render_thread_impl.cc?rcl=0&l=1903). The major difference is that the OutputSurface in layout test mode lives on renderer/main. I think they make this to ensure all compositing tests are non-flaky.

As a result, in our current layout tests, although the compositor frame was successfully submitted to the browser thread (Note that browser thread still exists in layout test mode) but was never picked up and processed.

danakj@ proposed 3 possible solutions:

1) Don't use layout tests. Use unit tests. This is not quite similar to the unit tests that we wrote for canvas classes before; we need the helper function cc::PixelTest::RunPixelTest() (https://cs.chromium.org/chromium/src/cc/test/pixel_test.h?q=cc/test/pixel&sq=package:chromium&l=33) to help us verify the correct color inside a CompositorFrame. 

2) Added extra IPC calls to make compositor frame travel a little bit more round trips between browser and renderer. The basic idea is to let the compositor that lives in renderer in layout tests mode to send a request to browser, asking for processing of that compositor frame that is already sent from OffscreenCanvas. I discussed with weiliangc@ on white board and we feel that the whole logic flow looks even more flaky than the the one we thought right now. Also, it causes a lot of overhead to have extra IPC calls in layout tests.

3) Change Mojo channel. Instead of connecting the receiving end of mojo message to browser, connect it to renderer/main. Well, then the whole IPC things becomes a cross-thread communication only. At this moment, I'm not sure how it can be implemented; would need to ask those who are familiar with both Mojo and Blink to see if it's possible. I don't think it's a trivial implementation change.

From danakj@'s opinion, option (1) looks like the most viable solution. WDYT?

I agree. Option 1) sounds reasonable to me.

junov@: in addition to using unit test, should we create a manual/ dir under fast/canvas/, so that at least we would run those tests occasionally.

actually cannot be under fast/canvas, has to be under WebKit/ManualTests

Using automation would be a much better idea.  You can achieve end-to-end testing by implementing a browser_test.  This allows you to do something very similar to a layout test with pixels results, except that it run in a regular chrome build.

In this case, we should probably even consider running this on the gpu bots, so you'd want to put this in gpu_browser_tests.

I think a reasonable group of gpu_browser_test for this would be tab_capture_end2end_tests.

Already decided to use gpu pixel test for end-to-end testing