Re: https://codereview.chromium.org/2308283002/#msg16 we could implement this as a blacklist of directories instead of individual tests.

I'm not sure that the functionality described in the initial report is what we want. It says that if if the flag is not passed, we skip the order-dependent tests.

That means that if people run run-webkit-tests as normal, suddenly we'll start skipping probably a large number of tests, and that may be confusing.

(It's also not what --smoke does; we run all of the tests listed in SmokeTests even when the flag isn't passed).

Maybe a better thing to do is to add a generic --ignore-test-list flag that mirrors what --test-list does except that the listed tests are ignored (skipped)?

That way the existing behavior is unchanged and the command line is explicit when we want to do something different.

Thoughts?

dpranke, what do you think about ojan's suggestion of keeping 1 invocation of run-webkit-tests and determining order during sharding? https://bugs.chromium.org/p/chromium/issues/detail?id=601332#c10

We could add a new option to --order, e.g. --order=mostly-random where it would run non-order-dependent tests in random order and run OrderDependentTests in the same order.

You can certainly do that, and there's certainly good reasons to do so. It is slightly more work to implement handling a single invocation in webkitpy, it's almost certainly quite a bit more work to implement running run-webkit-tests twice, either locally, or on the builders, in order to make sure you don't lose coverage.

And, there's a lot to be said for making sure devs don't have to invoke things twice to get full coverage.

Really, the question is, how many order-dependent tests do we have and realistically how long do we think it'll take to make them order-independent? That question will drive what we do here (for all we know, making them be order-independent is not even very practical).

qyearsley's CL https://crrev.com/2308283002 to allow passing a seed to run-webkit-tests will make it easier to find order-dependent tests. I'll try to see if I can use that to get a ballpark number.

Note, that was originally Tim's CL and Tim already has done some work to get a preliminary list of tests that are flaky when run in random order. This doesn't yet tell us why the tests are order-dependent and how easy or feasible it is to make them order-independent.

I guess that if there are a lot of order-dependent tests and it appears infeasible to fix that, then we'll want to find some stable orders to run them in, in multiple shards. If, however, there aren't very many, or it appears like it will be feasible to make them order-independent within a quarter, then running them in one shard will probably be simpler.

Update: In some discussion earlier this week, we were talking about slightly modifying this plan:

 1. Instead of --run-order-dependent-tests and a OrderDependentTests file, add a RandomOrderExpectations file which contains extra flaky expectations that apply when running in random order.
 2. Monitor a FYI bot that runs tests in random order, and fill in RandomOrderExpectations until that's green (bug 652357).
   - We should file bugs for tests that are flaky when the tests are run in random order, and try to find owners for those issues.
 3. Run in random order by default.
 4. Fix and remove tests until RandomOrderExpectations is empty, then remove that file.

I'm going to close this bug since we're no longer adding this flag (please re-open if I'm mistaken).