I've put together a WIP CL @ https://crrev.com/c/1179072

Because the CL makes Layout Tests run by default in site-per-process mode, the test expectations from FlagExpectations/site-per-process have to be in theory inlines into top-level TestExpectations.  The CL does this for:

    http/tests/navigation/cross-origin-fragment-navigation-is-async.html [ Failure ]

At the same time, we want to preserve test coverage for this test in not-site-per-process mode.  The CL tries to do this, by having the following expectation in FlagExpectations/not-site-per-process:

    http/tests/navigation/cross-origin-fragment-navigation-is-async.html [ Pass ]

This doesn't really work, because even after adding gibberish to http/tests/navigation/cross-origin-fragment-navigation-is-async-expected.txt, the tests still pass:

    $ DISPLAY=:20 third_party/WebKit/Tools/Scripts/run-webkit-tests -t rel \
        --no-retry http/tests/navigation/cross-origin-fragment-navigation-is-async.html \
        --additional-driver-flag=--disable-site-isolation-trials \
        --additional-expectations=third_party/WebKit/LayoutTests/FlagExpectations/not-site-per-process
    ...
    The test ran as expected (0 passed, 1 didn't).

This is because the layout tests harness combines expectations from all the expectation files, so we end up with:

    http/tests/navigation/cross-origin-fragment-navigation-is-async.html [ Pass Failure ]

Maybe we should introduce a [ Reset ] expectation (and have it remove all expectations except maybe [ Slow ])?

Otherwise, I think switching Layout Tests to default to site-per-process is blocked on making FlagExpectations/site-per-process empty and this is not quite possible - for example see:

    # The tests below set document.domain.  Such tests are skipped, because they
    # cannot work when same-site WPT origins are isolated from each other via
    # --isolate-origins cmdline flag (which is how bots run the
    # |site_per_process_webkit_layout_tests| step).  Note that
    #  https://crbug.com/783416  tracks adding a WPT origin that is cross-site, but
    # even after this bug is fixed we may decide to keep the current
    # --isolate-origins coverage and therefore the tests might end up being skipped
    # for the foreseeable future.
    Bug(none) external/wpt/html/browsers/origin/relaxing-the-same-origin-restriction [ Skip ]
    Bug(none) external/wpt/FileAPI/url/multi-global-origin-serialization.sub.html [ Skip ]
    Bug(none) virtual/mojo-blob-urls/external/wpt/FileAPI/url/multi-global-origin-serialization.sub.html [ Skip ]
    ... 10 more tests ...

I would really prefer to not add a `[ Reset ]` expectation. The syntax/semantics for these files is already complicated and in our experience in the past having entries in one file override entries in another was worse (harder to maintain) overall.

I didn't quite follow you in #c2; are you saying there are tests that will always fail in site-per-process-mode, or tests that will always fail in not-site-per-process-mode, or both, or something else?

RE: #c3

There are tests that will always fail in site-per-process mode.

I assume that we'd want to preserve coverage offered by such tests.  I hoped that the coverage could be preserved in a not_site_per_process_webkit_layout_tests step, but test expectations behavior makes it hard as described in #c2.

IIUC, in order to better test Site Isolation, we forcibly isolate all WPT domains from each other, i.e. pretend various subdomains are different sites, even though they are the same site.

This makes the origin handling different from specs and breaks some tests that are testing exactly these behaviours.

What about creating a virtual suite that forces isolation in WPT?

If there is a small number of tests that will intentionally produce different behavior in each mode, then I would create a virtual test suite for them as robertma@ suggests; that's what virtual test suites are for.

RE: Virtual suite suggestion

This may be a direction to explore further.  If we wanted to go down the virtual test suite route, then I think we'd consider something like this:

- default to site-per-process behavior everywhere (just like //content), except
- also-isolate-wpt-origins for external/wpt via virtual suite
- not-site-per-process for http/tests via virtual suite

I think this should wait until FlagExpectations/site-per-process gets cleaned up in https://crrev.com/c/1178465.

I think this is not blocked on issue 874695 (layout tests seem more flaky with site-per-process), because https://crbug.com/874695#c6 shows that site-per-process is okay wrt flakiness.

FWIW, FlagExpectations/site-per-process covers only 3 tests outside of http/tests and external/wpt - see below.  Ideally we would get rid of these before proceeding (because otherwise we would at least temporarily loose coverage provided by these tests):
     crbug.com/769508  fast/css/acid2.html [ Failure ]
     crbug.com/711468  virtual/gpu/fast/canvas/canvas-composite-image.html [ Pass Timeout ]
     crbug.com/711468  virtual/gpu/fast/canvas/canvas-scale-strokePath-shadow.html [ Pass Timeout ]

Please correct me if I am missing something, but AFAIK virtual test suites apply to all bots/platforms.  This means that the virtual test suites route will incur higher capacity cost than the current approach of only running with/without site-per-process on the linux CQ and waterfall (via explicit site_per_process_webkit_layout_tests step defined via testing/buildbot/test_suites.pyl).  This seems to make the virtual test suites route undesirable.

From https://chromium.googlesource.com/chromium/src/+/lkcr/docs/testing/layout_tests.md:

"""Virtual test suites incur a performance penalty for the commit queue and the continuous build infrastructure. This is exacerbated by the need to restart content_shell whenever flags change, which limits parallelism. Therefore, you should avoid adding large numbers of virtual test suites."""

I guess all of http/tests + external/wpt would add up to a "large number"...

Re #9:

0. Ideally we should minimise the number of tests in the virtual suite. Is there certain parts of WPT that are more likely to behave differently when subdomains are forced to be isolated?
1. Yes currently virtual test suites run on all platforms. If for technical/capacity reasons, we can only run site isolation layout tests on Linux, we may consider adding the feature to specify platforms in virtual suites, or we can skip the whole suite on other platforms (IIRC [ Skip ] expectations can apply to a directory). Either way, this is an implementation detail we can work around.
2. The performance penalty comes from a large number of suites, not a suite with a large number of tests, because content_shell needs to be restarted between suites to apply new command-line arguments. There shouldn't be too much overhead here.

What robertma@ wrote in #c10 is mostly correct, I just have a couple of things to add.

First, while virtual test suites do impose a performance penalty for the commit queue, that's when compared to regular tests or not running the tests at all.

They are, however, usually (always?) less of a performance penalty than running an entirely separate test step, which is one of the reasons why we have so many more virtual test suites than separate steps.

Second, there is a performance penalty in simply running more tests N different ways, in addition to the restarts. So, don't do it if you don't have to.

We skip VirtualTestSuites on particular platforms fairly often (as robertma@ suggests), that's standard practice.

But, VirtualTestSuites can be awkward if you have to have to run a *lot* of tests. external/wpt would be too many, but as I've said elsewhere, external/wpt contains a lot of stuff that I would expect is not specific to site-per-process at all.