The above approach should work pretty well for gtest and webkit layout tests. For other test harness, we could improve later.

I have thought about this before.

We can find and provide more helpful test failure logs in gtests' results. 

But we don't have stacktrace kind of logs in json_test_result_format (including webkit_layout_tests and telemetry based tests). Though we can still provide some useful information: like if the test failed/timed_out, and time of each test run(IIRC).

We have a specification for how we should be storing artifacts like stdout and stderr: bit.ly/chromium-test-artifacts. I think only the perf tests currently implement it, though.

For gtests, the stdout/stderr of individual tests are in the output.json
For Webkit layout tests, they are in separate files from the output.json IIUC. +robertma could confirm on this.

IIRC, for webkit_layout_tests, there might be stderr file for crashed tests. But not for other types of failed tests. robertma@ could you confirm?

stdout is either empty or contains the values that are checked against the -expected.txt for correctness.

stderr may be empty (ideally) or it may contain error messages or a stack trace (or both).

Yeah they are separate files in the isolated output ([testname]-{stderr,actual}.txt).

Crash log usually goes to stderr. Console messages are a bit tricky: non-WPT layout tests dump console messages to stdout, while WPT dump console to stderr.

Since developers want to see the error log, we could handle both cases for Webkit layout tests:
* If there is either stdout/stderr or both, merge them into one log.
* If none is available, then just mention that no failure log available.

wdyt?

You can't merge them correctly (in the chronological sense, i.e., as you'd see if you were actually running content_shell directly in a terminal), and so merging them might just be confusing. I would expose both logs separately, if they exist (and I think that model generalizes to arbitrary lists of artifacts).

Sorry for the confusion. By merging, I actually meant concatenate them together with a separator "stdout:" and "stderr:".

Right; I'm saying that because you can't actually interleave them, I don't think concatenation would be that great a UI or an improvement over two separate links ...

This depends on how developers like. We got quite a few feedback that we should dump more info directly into the bugs or CLs as text in a comment: don't want to click through too much.

Maybe creis@ could share some thoughts here? Are links or comments better from a developer's perspective?

Ah, I see. If you wanted to inline the results into a bug report, then, sure, I could see inlining each in that case. 

But if you were going to provide links, I think you should either provide one link per artifact or provide a link to a single page that then had special UI to display the results for all of the artifacts for a single test (e.g., sort of like the way we try to do that in the webkit layout test results).