I agree; it might be useful here if you mentioned which pieces of information get trimmed out that you would like to keep...

Hi Dan, we are particularly interested in the files ending with .DEBUG. I posted https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/717724 but it seems that I misunderstood the meaning of MAX_RESULT_SIZE_KB; I though it was the limit of single files.

The largest file in the test is perf.data. However, because we upload that to somewhere in gs explicitly[1], we don't need it to be collected along with logs. IIUC, throttling happens between devserver and gs, rather than between devserver and dut, right? If so, it can be skipped completely.

By the way, it is pretty inconvenient that a log is compressed and needs to be downloaded and decompressed before we can read it.

[1] http://cs/chromeos_public/src/third_party/autotest/files/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py?l=121&rcl=400fd0336f20774a7ce68cbc0bc0aff4104f8870

Your CL should reduce the result size:
https://chromium-review.googlesource.com/#/c/chromiumos/third_party/autotest/+/717724

You can also increase MAX_RESULT_SIZE_KB to a large number, say 100MB, considering the telemetry test's nature.

The throttling happens in two places:
between lab server and duts
before lab server try to upload results to GS in gs_offloader.

If the large perf.data file is created on dut and the DEBUG log you are interested is also generated in the dut, result throttling will still trim the DEBUG log even if you delete perf.data at the end of the test.
In that case, the only way to avoid that is to increase MAX_RESULT_SIZE_KB

Compression on the log happens as the last step to reduce the size of text file. If increasing MAX_RESULT_SIZE_KB and delete perf.data are enough, compression shouldn't happen.

Looks like that perf.data is already whitelisted?

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/bin/result_tools/dedupe_file_throttler.py?q=%22.*perf.data%22&dr=C&l=27

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/bin/result_tools/delete_file_throttler.py?q=%22.*perf.data%22&dr=C

The white list is for files not to be throttled, but they are still counted in the final results. That's why autotest tries to trim all the logs and compress them but leave perf.data not touched.

With your CL removing the file after it's uploaded to GS, I think the throttling shouldn't affect the debug logs.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/693ca223cc18033b56f706ac3ba6568518a1f001

commit 693ca223cc18033b56f706ac3ba6568518a1f001
Author: Ting-Yuan Huang <laszio@chromium.org>
Date: Wed Oct 18 13:15:06 2017

telemetry_AFDOGenerate: don't upload perf.data along with logs

BUG= chromium:774328 
TEST=tryjob; Found no perf.data in the logs.

Change-Id: If15894e2d2e4cb423448fea01cdf0aca67c617c5
Reviewed-on: https://chromium-review.googlesource.com/717724
Commit-Ready: Ting-Yuan Huang <laszio@chromium.org>
Tested-by: Ting-Yuan Huang <laszio@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/693ca223cc18033b56f706ac3ba6568518a1f001/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py

Autotest logs became fully available again with the above and Dan's fix:
https://crrev.com/cdf50c281416123b59f9ff6d1c76961117177c3c