The 35 seconds is currently broken down to:
Generating MANIFEST.json took 10.1520960331
Collecing tests took 11.4412479401
Parsing expectations took 13.1838738918

 Issue 907879  has been merged into this issue.

 Issue 907879  has a good breakdown of the large difference in time taken between using run_web_tests.py and just invoking content_shell directly.

nednguyen@, what is in the "Collecing tests" step, what does that do?

Note also that there are two distinct cases here to make faster. This bug was filed, I think, about making startup faster when running all tests.  Issue 907879 , on the other hand, is about startup time when running a single test. Presumably there are important differences, like "collecting tests" has to be a no-op when they're listed on the command line.

In general, wondering if we could introduce a flag that would re-use state (manifest, test list, etc) from a previous run? This would help cases where the same set of tests are run repeatedly (eg debugging). Seems possible to make it smart enough to be able to regenerate files when needed (eg: if they're missing, or if the cmd-line params change).

When we run all tests, they take so long that startup cost doesn't matter.

When trying to figure out if my change unbreaks the test, I want to run a small number of tests every time I've compiled again - startup costs matter.

Other things to think about: Whether we can have the server running without restarting it every time, or whether we can detect that tests don't need the server to be running at all.

To know with certainty that the tests in wpt don't need the server unfortunately isn't possible to determine statically, but keeping the server running, or perhaps making it start in <100ms, is a great idea.

Even if there is something in starting the server that would take a little bit of time, seems like it could at least start listening to the sockets immediately and buffer data until the rest of the setup is done, effectively parallelizing server start and browser start.

Another flag that I have been using is --watch.  This flag keeps the WPTServer alive between runs of WPT. When the test run has completed, the framework asks if you want to "Interactive watch mode: (q)uit (r)etry".  It seems you can edit your tests while the framework waits for you.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6a79321c87f697c14646dbe75c7408c15c52b2f7

commit 6a79321c87f697c14646dbe75c7408c15c52b2f7
Author: Philip Jägenstedt <foolip@chromium.org>
Date: Thu Dec 06 10:11:08 2018

Add a `run_web_tests.py --no-manifest-update` option

This is like the `./wpt run --no-manifest-update` in upstream wpt:
https://github.com/web-platform-tests/wpt/blob/81607985fc9485cbf9f3d0a8ae74359f885eb36b/tools/wptrunner/wptrunner/wptcommandline.py#L52-L53

Because updating the manifest is pretty slow, this reduces the time to
run a few tests over and over in development when the manifest is not
changing.

On a Linux workstation, the time to run a single test
(dom/historical.html) went from ~14s to ~7s.

Bug: 905406
Change-Id: I50e4b8bdd9d3bedbf1e31bc34a2338317a30aaf0
Reviewed-on: https://chromium-review.googlesource.com/c/1361874
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Reviewed-by: Doug Turner <dougt@chromium.org>
Reviewed-by: Luke Z <lpz@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#614301}
[modify] https://crrev.com/6a79321c87f697c14646dbe75c7408c15c52b2f7/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py
[modify] https://crrev.com/6a79321c87f697c14646dbe75c7408c15c52b2f7/third_party/blink/tools/blinkpy/web_tests/port/base.py
[modify] https://crrev.com/6a79321c87f697c14646dbe75c7408c15c52b2f7/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py