FYI this was a problem when landing https://chromium-review.googlesource.com/c/chromium/src/+/910122 . FFC font collections are slow to load, so content_shell was timing out when loading fonts. The issue was reported as a lot of timing out layout tests.

sergeyu@, with "Problem is that if that timeout expires it proceeds with the test as if the content_shell was started successfully", do you mean that the 'content_shell took too long to startup.' that is logged is just a log, and that instead it should fail in some other way?

Or is the problem that some errors inside start() are suppressed? Where?

The problem is that Driver._start() may fail, but it doesn't return the error to the caller and the caller does try to handle the error. The output that we get on the bots make it looks like bunch of tests are failing while these tests were not even executed, e.g. see https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/646102 .
Driver._start() calls _log.error(), but I don't see the corresponding message anywhere in the bot output, I'm not sure why.

I'm not quite sure where those error logs would/should end up. Not for the individual tests I presume, because it's not part of the stderr when running the tests, but of what wraps the tests.

Anyway, let's say we propagate the error further where there's now an 'content_shell took too long to startup.' error. I guess then in run_test we should treat that as a separate kind of failure? Or a timeout with a different message?

Anything new would have to be added here:
https://chromium.googlesource.com/chromium/src/+/28390a4663e6bcd261753891d369d42effaaf6d5/third_party/WebKit/LayoutTests/fast/harness/results.html#710

Marking this as available, but because it's a P3 (assuming that's right) would not expect it to be done any time soon.