Quickly taking a look at one example log (https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.10/builds/27658/steps/webkit_tests/logs/stdio):

Total line count: 112143, total size: 11M.

It seems like there are two categories of things that can be reduced: stderr messages and regular logging during normal operation. Most regular lines start with "worker/n ..."; for each test there's a "started" message and end message (passed, failed...).

Lines starting with worker/n: 98499, total size: 8.6M.

There are a lot of error messages that could be cut down, but the easiest single thing that could be cut may be the " started" messages (which take 47725 lines, 4.2M). How useful are those lines?


The archive_webkit_test_results log is about 21768 lines, 2.3M, most of it consisting of listing individual files that are being uploaded. There is a "-q" (quiet) flag for gsutil, so adding that would be the quickest easiest change. Although, of course, we have the separate problem that archiving is slow because we archive too much;  bug 310382 .

@qyearsley - can you land changes ASAP to add the `-q` to gsutil and comment out the "started" message ?

Sounds, good, will do now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/38f69a78a8cfdd45460ebbaa3c97f4f6d3c13296

commit 38f69a78a8cfdd45460ebbaa3c97f4f6d3c13296
Author: Quinten Yearsley <qyearsley@google.com>
Date: Mon Dec 12 18:54:55 2016

When archiving layout test results, invoke gsutil with quiet flag.

BUG= 673207 

Change-Id: I7c12aff507c8df0cca2d098b51af1375df2b2c0f
Reviewed-on: https://chromium-review.googlesource.com/419058
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/38f69a78a8cfdd45460ebbaa3c97f4f6d3c13296/scripts/slave/chromium/archive_layout_test_results.py
[modify] https://crrev.com/38f69a78a8cfdd45460ebbaa3c97f4f6d3c13296/scripts/slave/slave_utils.py
[modify] https://crrev.com/38f69a78a8cfdd45460ebbaa3c97f4f6d3c13296/scripts/slave/unittests/slave_utils_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/67aaccae4c80bb66c0a623a6e3e56b8151d6e3eb

commit 67aaccae4c80bb66c0a623a6e3e56b8151d6e3eb
Author: qyearsley <qyearsley@chromium.org>
Date: Mon Dec 12 21:51:11 2016

Stop logging the start of a run for individual layout tests.

NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true
BUG= 673207 

Review-Url: https://codereview.chromium.org/2564343003
Cr-Commit-Position: refs/heads/master@{#437924}

[modify] https://crrev.com/67aaccae4c80bb66c0a623a6e3e56b8151d6e3eb/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py

Is log size still an issue? (Is there a target log size?) If so, one possible next thing to try would be to truncate or silence most error messages when running layout tests by default.

I think we can at least downgrade this, but there's still probably room for improvement. In particular, I'd like to know how there are 14k lines that aren't starting with "worker/N" ... something seems badly wrong there.

The volume of stderr messages has probably been slowly increasing over time; see  bug 375415 .

[Unassigning for now since I don't have any immediate plans to resolve this.]

Closing this; the main issues that I cared about were addressed.