Cap the size of the client test result transfer from DUT to server/shard
Reported by
jrbarnette@chromium.org,
Apr 27 2017
|
||||||||||||||
Issue descriptionA recent lab outage happened when a large number of Chrome crashes from testing in the lab overwhelmed the results gathering/offloading system. Bug 715228 addresses the particular problem (Chrome core files) that caused the particular outage. However, there's no overall cap on the size of test results. Thus, it's not impossible that some other source of results volume could cause a similar outage, if it isn't in files named "*.core". We should consider putting a cap on the total size of results directories that we're willing to offload from a DUT. For results that exceed the cap, we should gather rump logs that include at least the following: * status and status.log * Debug logs under debug, that is "*.DEBUG", "*.INFO", etc., but _not_ anything else. * A list of all the files that were the results, with their sizes, and a note explaining why not everything is present.
,
Apr 28 2017
The reason this is important is that the hardware lab is very network sensitive. In particular rsyncing too much data from DUTs has caused 3 major lab outages in the last year and a half: a) Issue 524814#c122: Provisioning failures and stage_artifacts timeouts: - Telemetry failure created too many screenshot.png causing. b) Issue 637935 : Investigate how to better fix chrome crash overload problem - Too many ARC++ crashes. Also b/29401582 c) Issue 713004: Tests passed but got aborted by AutotestAbort - Extreme lab slowdown due to otherwise benign Chrome shutdown crash due to evdev memory corruption issue 713968. - Network was overloaded from about April 11-ish to April 22nd. But there are still signs that we experience higher network IO (see attachments). So I don't consider this fully fixed (branches).
,
Apr 28 2017
,
Apr 28 2017
There are obviously many ways to enforce this limit. I think there needs to be both a part on the client to efficiently clean up the result directory, but also another part on the server to enforce a hard limit across all (old) images and branches if the client fails to do the job. One thought that crossed my mind to further reduce network usage is to perform tarballing on the client (not server) for all results. This currently happens in gs_offloader.py on the server, but only for directories that have more than 500 files. Doing this at the start of the result pipeline on the client, and consistently for all results (not just those with 500 files) might be a network win. https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/site_utils/gs_offloader.py?q=gs_offloader_limit_file_count+package:%5Echromeos_public$&l=110 Notice moblab folks also have to deal with thin partner connections and just enabled compression.
,
Apr 28 2017
Notes: - Creating the tarball on the client has the same effect (compression ratio,CPU on DUT) as using rsync -z as we do right now. There are two benefits to tarballing on client: 1) We know the compressed size before starting the transmission (usually ratio is 1:10 but that obviously varies). 2) We save the decompression and recompression CPU load on the server. Obviously the benefit would be larger for client (logs + results) than for server tests (logs only). - I checked the servers and there are way less core files in the result dirs then last week. But there are still some mostly from Android, also a few from Chrome OS system processes, very rare Chrome. But as the behavior is bursty I just can't tell if that contributes significantly to the network load, and if the mix changes over time.
,
Apr 28 2017
There could be couple issues with tarballing on client: 1. Enough free disk space on the dut. 2. The same on the network bandwidth could be offset by the CPU load on drone. Considering we are running more cheets test, drone could be quite loaded. Untar the files on drone, as required for crash dump symbolification, will add more cpu load on drone. We can consider to tarball crash dumps only, and devserver code needs to be updated, so it can take a tarball of a crash dump.
,
Apr 28 2017
My current thinking is this:
* Run a 'du' command on the results directory on the DUT.
* If the output from 'du' indicates the results directory
is over some (selectable) limit, transfer only the
data detailed in the initial bug description.
An option to allow gathering data during critical failures
would be to use the stochastic transfer approach from
crashcollect.py:
https://cs.corp.google.com/chromeos_internal/src/third_party/autotest/files/server/crashcollect.py?l=74
That is, cap the data with a probability equal to min(LIMIT / SIZE, 1.0)
,
Apr 28 2017
@6: 1. You can use unix pipes and not create a temporary local file, so that's a non-issue. 2. Decompression is much lighter CPU-wise than compression, so that's not really an issue
,
Apr 28 2017
#6 1. We want to assemble a result directory that is < 10..100MB after compression, so at most 100MB..1GB before compression/free disk needed. 2. We probably will not try to symbolize crash dumps in the lab anymore. It never really worked. I am thinking minidumps only. :-( I think the absolute minimum transfer needs to contain a) debug directory containing the *.DEBUG file b) keyval and status files c) a text file containing all file names (with sizes) that were not transferred But before that there should be multiple sanitation phases to reduce the size of the results directory. Like: 1) delete all core files, check if results dir is small enough 2) delete all screenshot.png, check if results dir is small enough (for the sake of argument) 3) delete all binaries, check if results dir is small enough 4) delete all huge text files, check if results dir is small enough 5) If still not small enough only grab the minimum amount of files as defined in a-c. While I think a test that went through 1) may still keep its original status, but somewhere around phase 3) we need to tell the test owner that things are going really wrong and fail the test.
,
May 5 2017
design doc from dshi WIP
,
May 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b2751fcd14518af3e6c5aba50c345abeded576b9 commit b2751fcd14518af3e6c5aba50c345abeded576b9 Author: Dan Shi <dshi@google.com> Date: Fri May 19 20:57:51 2017 [autotest] add more metrics details in gs_offloader This change is to add more details of metrics collected in gs_offloader. BUG= chromium:716218 TEST=unittest, run gs_offloader locally with test GS bucket. Change-Id: I09be661f3c8df8d196a3305e529d05e41ceabd78 Reviewed-on: https://chromium-review.googlesource.com/506781 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/b2751fcd14518af3e6c5aba50c345abeded576b9/site_utils/gs_offloader.py [modify] https://crrev.com/b2751fcd14518af3e6c5aba50c345abeded576b9/utils/labellib.py [modify] https://crrev.com/b2751fcd14518af3e6c5aba50c345abeded576b9/utils/labellib_unittest.py
,
Jun 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/9f879fb899dde997dbec1da17bd8bb48a4f8b1fb commit 9f879fb899dde997dbec1da17bd8bb48a4f8b1fb Author: Dan Shi <dshi@google.com> Date: Sun Jun 04 00:21:12 2017 [autotest] Record directory information when collecting client side logs. This change added a command line tool to run on the client side to build up the directory information (path and size). The command is executed before each get_file call in server/autotest. The directory summary will be used to throttle the result size. BUG= chromium:716218 TEST=local run test, verify the summary file is collected. Change-Id: I99c8b9a30c1ea5cf9be907d1b8a7dc1730c5b437 Reviewed-on: https://chromium-review.googlesource.com/517421 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [add] https://crrev.com/9f879fb899dde997dbec1da17bd8bb48a4f8b1fb/client/bin/result_utils_unittest.py [modify] https://crrev.com/9f879fb899dde997dbec1da17bd8bb48a4f8b1fb/server/autotest.py [add] https://crrev.com/9f879fb899dde997dbec1da17bd8bb48a4f8b1fb/client/bin/result_utils.py
,
Jun 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/96c3bdc2e5de17857fabf4c40a0f371b0d657258 commit 96c3bdc2e5de17857fabf4c40a0f371b0d657258 Author: Dan Shi <dshi@google.com> Date: Sat Jun 10 04:44:01 2017 [autotest] Log test result size tko_job_keyvals Get the result size information from directory summary and save the data to tko_job_keyvals table. BUG= chromium:716218 TEST=local run test, make sure results are recorded in tko table mysql> select * from tko_job_keyvals as t where job_id=714 AND t.key like "%result%"; +------+--------+----------------------------+--------+ | id | job_id | key | value | +------+--------+----------------------------+--------+ | 8772 | 714 | result_uploaded_KB | 557731 | | 8773 | 714 | result_throttled | 0 | | 8781 | 714 | client_result_collected_KB | 569856 | | 8783 | 714 | original_result_total_KB | 557731 | +------+--------+----------------------------+--------+ Change-Id: Ice56082086d3b48b885e5a0d64966ac9598b91e7 Reviewed-on: https://chromium-review.googlesource.com/513520 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/96c3bdc2e5de17857fabf4c40a0f371b0d657258/client/common_lib/file_utils.py [modify] https://crrev.com/96c3bdc2e5de17857fabf4c40a0f371b0d657258/site_utils/gs_offloader.py [modify] https://crrev.com/96c3bdc2e5de17857fabf4c40a0f371b0d657258/tko/parse.py
,
Jun 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/05d0b898d2bcbdbf88bf1ffb45ec1d9b34fd7479 commit 05d0b898d2bcbdbf88bf1ffb45ec1d9b34fd7479 Author: Dan Shi <dshi@google.com> Date: Sat Jun 10 04:44:00 2017 [autotest] Add support to merge directory summaries from multiple result collect This change adds the main logic to merge the directory summaries from the multiple result collections during the test. During the merge, it also calculate the total size of result files transfered from the test device. At the end, the summaries are merged with all the files in the test result folder on the drone, which will be used to compile the final view of test results. BUG= chromium:716218 TEST=unittest Change-Id: I3ff326a046497f752125ed679e7c4f72bdddc5fa Reviewed-on: https://chromium-review.googlesource.com/526274 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/05d0b898d2bcbdbf88bf1ffb45ec1d9b34fd7479/client/bin/result_utils_unittest.py [modify] https://crrev.com/05d0b898d2bcbdbf88bf1ffb45ec1d9b34fd7479/client/bin/result_utils.py
,
Jun 12 2017
,
Jun 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/136240121e239efec5005e1bdc632dde3cf2fd1e commit 136240121e239efec5005e1bdc632dde3cf2fd1e Author: Dan Shi <dshi@google.com> Date: Wed Jun 21 00:24:20 2017 [autotest] Compile result summaries into a single html page The html page allows one to browse the test results and check sizes. BUG= chromium:716218 TEST=unittest, local build Rebuild html page: /usr/local/autotest/tko/parse -l 2 -r -o \ /usr/local/autotest/results/1355-dshi/100.107.151.233 local view: http://dshi1.mtv/results/1355-dshi/100.107.151.233/result_summary.html GS view: https://storage.cloud.google.com/dshi-test/1355-dshi/100.107.151.233/result_summary.html Change-Id: I39ac268e3b5da3f1c15cad314a8bf370eff00dfd Reviewed-on: https://chromium-review.googlesource.com/531605 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [add] https://crrev.com/136240121e239efec5005e1bdc632dde3cf2fd1e/client/bin/result_view.py [modify] https://crrev.com/136240121e239efec5005e1bdc632dde3cf2fd1e/client/bin/result_utils_unittest.py [modify] https://crrev.com/136240121e239efec5005e1bdc632dde3cf2fd1e/tko/parse.py [modify] https://crrev.com/136240121e239efec5005e1bdc632dde3cf2fd1e/global_config.ini
,
Jun 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f53d1269ad7321116eadcb0167d81dbea9bf4bef commit f53d1269ad7321116eadcb0167d81dbea9bf4bef Author: Dan Shi <dshi@google.com> Date: Thu Jun 22 06:38:11 2017 [autotest] Allow control file to specify maximum test results to collect Control file can set an attribute MAX_RESULT_SIZE_KB to specify the maximum test result size to collect. By default, it will be 20MB if `enable_result_throttling` is set to True in global config. BUG= chromium:716218 TEST=unittest, local run dummy tests Change-Id: Ia6b9a04d92ec91179ceb99466b5e9d6a5d1b3ef3 Reviewed-on: https://chromium-review.googlesource.com/540196 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/server/server_job.py [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/client/site_tests/dummy_Pass/control [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/client/bin/result_utils.py [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/global_config.ini [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/server/server_job_unittest.py [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/server/autotest.py [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/client/common_lib/base_job_unittest.py [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/client/common_lib/control_data_unittest.py [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/client/common_lib/base_job.py [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/server/site_tests/dummy_PassServer/control [modify] https://crrev.com/f53d1269ad7321116eadcb0167d81dbea9bf4bef/client/common_lib/control_data.py
,
Jun 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/010c0bcc9c7d91c451534d95b12305649627c180 commit 010c0bcc9c7d91c451534d95b12305649627c180 Author: Dan Shi <dshi@google.com> Date: Fri Jun 23 07:51:28 2017 [autotest] Move result_utils to its own package This change moves result_utils.py to its own package. More code will be added to enable result throttling. So it's more manageable to put all related files to a single folder. This change also push the result tools modules to client before trying to build result summary. This removes the client side dependency. So all result throttling logic will be kept on the server side. BUG= chromium:716218 TEST=unittest, local run dummy tests Change-Id: Icc3542af99439fd39b1dba36ac45aab6effd1080 Reviewed-on: https://chromium-review.googlesource.com/544049 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [rename] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/client/bin/result_tools/utils.py [modify] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/server/server_job.py [modify] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/tko/parse.py [rename] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/client/bin/result_tools/view.py [modify] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/server/autotest.py [add] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/client/bin/result_tools/__init__.py [add] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/client/bin/result_tools/common.py [rename] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/client/bin/result_tools/utils_unittest.py [add] https://crrev.com/010c0bcc9c7d91c451534d95b12305649627c180/client/bin/result_tools/runner.py
,
Jun 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e2e995d750616022e2d5cfeda5d9eb2bcd78df2d commit e2e995d750616022e2d5cfeda5d9eb2bcd78df2d Author: Dan Shi <dshi@google.com> Date: Wed Jun 28 21:45:13 2017 [autotest] Move const and helper methods to shared lib module. This change has no function changes. It splits some constants and helper methods to separate modules so others can share. BUG= chromium:716218 TEST=unittest Change-Id: Ieb4a4de8cfc825d204c4c2e1247a2e9eeeb0639d Reviewed-on: https://chromium-review.googlesource.com/548970 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/e2e995d750616022e2d5cfeda5d9eb2bcd78df2d/client/bin/result_tools/utils.py [modify] https://crrev.com/e2e995d750616022e2d5cfeda5d9eb2bcd78df2d/tko/parse.py [modify] https://crrev.com/e2e995d750616022e2d5cfeda5d9eb2bcd78df2d/client/bin/result_tools/view.py [add] https://crrev.com/e2e995d750616022e2d5cfeda5d9eb2bcd78df2d/client/bin/result_tools/unittest_lib.py [modify] https://crrev.com/e2e995d750616022e2d5cfeda5d9eb2bcd78df2d/client/bin/result_tools/utils_unittest.py [add] https://crrev.com/e2e995d750616022e2d5cfeda5d9eb2bcd78df2d/client/bin/result_tools/utils_lib.py
,
Jul 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ffd5b82912ac09842a2b55dd0706343c231bcf3e commit ffd5b82912ac09842a2b55dd0706343c231bcf3e Author: Dan Shi <dshi@google.com> Date: Wed Jul 19 08:23:38 2017 [autotest] Report test result sizes to metrics There are two reporting source: 1. Test jobs report result size information in tko/parse. 2. Special tasks report result size information at the end of autoserv process. BUG= chromium:716218 TEST=unittest, local run test Change-Id: I2410f3a3a5a16c1673446633e15c038a5e4ef81f Reviewed-on: https://chromium-review.googlesource.com/571859 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/ffd5b82912ac09842a2b55dd0706343c231bcf3e/server/site_utils.py [modify] https://crrev.com/ffd5b82912ac09842a2b55dd0706343c231bcf3e/server/autoserv [modify] https://crrev.com/ffd5b82912ac09842a2b55dd0706343c231bcf3e/client/bin/result_tools/utils_lib.py [modify] https://crrev.com/ffd5b82912ac09842a2b55dd0706343c231bcf3e/tko/parse.py
,
Jul 21 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/autotest-private-utils/+/f1a5efc54ef8f56b8be8587f0f8e111e8066682a commit f1a5efc54ef8f56b8be8587f0f8e111e8066682a Author: Dan Shi <dshi@google.com> Date: Fri Jul 14 18:58:40 2017
,
Jul 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/393fc8c903d61bad72ad0ab13d56955ac2888912 commit 393fc8c903d61bad72ad0ab13d56955ac2888912 Author: Dan Shi <dshi@google.com> Date: Mon Jul 24 21:38:15 2017 [autotest] Add a model to wrap the dictionary of result information. The wrapper class ResultInfo is added to make the result information model much cleaner. All result info related properties and methods are consolidated in a single class. BUG= chromium:716218 TEST=unittest Change-Id: Ie40974e7ea9911a813aa7c7879505a2fd5487978 Reviewed-on: https://chromium-review.googlesource.com/579573 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/utils.py [add] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/utils_lib_unittest.py [add] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/result_info.py [modify] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/view.py [modify] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/utils_unittest.py [add] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/result_info_lib.py [add] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/result_info_unittest.py [modify] https://crrev.com/393fc8c903d61bad72ad0ab13d56955ac2888912/client/bin/result_tools/utils_lib.py
,
Jul 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c45ac69f11a6c334743c9fc7cbafae733e75247b commit c45ac69f11a6c334743c9fc7cbafae733e75247b Author: Dan Shi <dshi@google.com> Date: Mon Jul 24 21:38:15 2017 [autotest] Add shrink_file_throttler used to reduce test result size. shrink_file_throttler tries to remove the middle part of test result files to reduce the result size. The logic is not triggered in the utils call yet. BUG= chromium:716218 TEST=unittest Change-Id: I7506cdbfeeeaaac0b050cb3ff9acebfde18b5a54 Reviewed-on: https://chromium-review.googlesource.com/580405 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [add] https://crrev.com/c45ac69f11a6c334743c9fc7cbafae733e75247b/client/bin/result_tools/throttler_lib.py [add] https://crrev.com/c45ac69f11a6c334743c9fc7cbafae733e75247b/client/bin/result_tools/shrink_file_throttler_unittest.py [add] https://crrev.com/c45ac69f11a6c334743c9fc7cbafae733e75247b/client/bin/result_tools/shrink_file_throttler.py [add] https://crrev.com/c45ac69f11a6c334743c9fc7cbafae733e75247b/client/bin/result_tools/throttler_lib_unittest.py
,
Jul 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/1e6dcef33df25a7435c7b111e5a2bf0d86071c9d commit 1e6dcef33df25a7435c7b111e5a2bf0d86071c9d Author: Dan Shi <dshi@google.com> Date: Tue Jul 25 08:32:47 2017 [autotest] Add a delete_file_throttler to throttle result size This throttler deletes result files from largest first to reduce result size. BUG= chromium:716218 TEST=unittest Change-Id: Ib8a8e9727bed84a1a7143eacab3b70aba9ec1ff9 Reviewed-on: https://chromium-review.googlesource.com/557306 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [add] https://crrev.com/1e6dcef33df25a7435c7b111e5a2bf0d86071c9d/client/bin/result_tools/delete_file_throttler.py [add] https://crrev.com/1e6dcef33df25a7435c7b111e5a2bf0d86071c9d/client/bin/result_tools/delete_file_throttler_unittest.py
,
Jul 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4b691e55808ec03bd55cbf4000adc98d4fb24dcb commit 4b691e55808ec03bd55cbf4000adc98d4fb24dcb Author: Dan Shi <dshi@google.com> Date: Tue Jul 25 08:32:47 2017 [autotest] Add two result throttlers (dedupe_file_throttler, zip_file_throttler) dedupe_file_throttler: This throttler tries to remove the remove repeated files sharing the same prefix, for example, screenshots or dumps in the same folder. The dedupe logic does not compare the file content, instead, it sorts the files with the same prefix and remove files in the middle part. zip_file_throttler: This throttler tries to reduce result size by compress files to tgz file. BUG= chromium:716218 TEST=unittest Change-Id: I98993da2ca18823fb4689ac9bea5679646ea2035 Reviewed-on: https://chromium-review.googlesource.com/559849 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [add] https://crrev.com/4b691e55808ec03bd55cbf4000adc98d4fb24dcb/client/bin/result_tools/zip_file_throttler_unittest.py [add] https://crrev.com/4b691e55808ec03bd55cbf4000adc98d4fb24dcb/client/bin/result_tools/dedupe_file_throttler_unittest.py [modify] https://crrev.com/4b691e55808ec03bd55cbf4000adc98d4fb24dcb/client/bin/result_tools/result_info.py [add] https://crrev.com/4b691e55808ec03bd55cbf4000adc98d4fb24dcb/client/bin/result_tools/dedupe_file_throttler.py [add] https://crrev.com/4b691e55808ec03bd55cbf4000adc98d4fb24dcb/client/bin/result_tools/zip_file_throttler.py
,
Jul 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c958a3d5b7053c6e807ec6ea7967a9271d4e09da commit c958a3d5b7053c6e807ec6ea7967a9271d4e09da Author: Dan Shi <dshi@google.com> Date: Wed Jul 26 09:07:30 2017 [autotest] Add logic in result_tools/utils to throttle results if needed. The change implements a sequence of throttling strategies to try to reduce result size to be under the limit. BUG= chromium:716218 TEST=unittest, local run test Change-Id: I8a4f7f85b074a8aae1fca600edbd70ed81e6d92d Reviewed-on: https://chromium-review.googlesource.com/562740 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/c958a3d5b7053c6e807ec6ea7967a9271d4e09da/server/autotest.py [modify] https://crrev.com/c958a3d5b7053c6e807ec6ea7967a9271d4e09da/client/bin/result_tools/utils.py [modify] https://crrev.com/c958a3d5b7053c6e807ec6ea7967a9271d4e09da/client/bin/result_tools/runner.py [modify] https://crrev.com/c958a3d5b7053c6e807ec6ea7967a9271d4e09da/client/bin/result_tools/utils_unittest.py [modify] https://crrev.com/c958a3d5b7053c6e807ec6ea7967a9271d4e09da/client/bin/result_tools/utils_lib.py
,
Jul 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4f8c0244d0ce557488a7ddbb29f28a2719a2523f commit 4f8c0244d0ce557488a7ddbb29f28a2719a2523f Author: Dan Shi <dshi@google.com> Date: Wed Jul 26 09:07:30 2017 [autotest] Add logic in test run and crashcollect to throttle results. Add throttling call in tko/parse.py to throttle the results created on the server side. For special tasks, there is no parse stage, so the final result throttling is triggered in autoserv (and crashcollect where the code directly calls host.get_file). BUG= chromium:716218 TEST=unittest, local run test Change-Id: I637bbead04f82f8f529d1d0982e3567445901938 Reviewed-on: https://chromium-review.googlesource.com/564140 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/server/hosts/abstract_ssh.py [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/client/bin/result_tools/utils.py [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/server/autoserv [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/server/crashcollect.py [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/client/bin/result_tools/result_info.py [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/tko/parse.py [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/client/bin/result_tools/runner.py [modify] https://crrev.com/4f8c0244d0ce557488a7ddbb29f28a2719a2523f/client/bin/result_tools/result_info_unittest.py
,
Jul 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/92c34c9d38647903943c066d07ae3d096baf5860 commit 92c34c9d38647903943c066d07ae3d096baf5860 Author: Dan Shi <dshi@google.com> Date: Wed Jul 26 11:15:02 2017 [autotest] Support excludes argument in send_file method This argument allows caller to skip certain files when sending files to dut. One use case is for result throttling to deploy scripts on the dut. When send_file uses commands that do not support --exclude, for example, copy file locally or using scp, send_file method will fail. BUG= chromium:716218 TEST=local run test, check files on the dut. Change-Id: I9efa44a15b8a04fa6a3d10601253b3e2aceba66c Reviewed-on: https://chromium-review.googlesource.com/572143 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/92c34c9d38647903943c066d07ae3d096baf5860/client/common_lib/hosts/base_classes.py [modify] https://crrev.com/92c34c9d38647903943c066d07ae3d096baf5860/server/hosts/abstract_ssh.py [modify] https://crrev.com/92c34c9d38647903943c066d07ae3d096baf5860/client/bin/local_host.py [modify] https://crrev.com/92c34c9d38647903943c066d07ae3d096baf5860/server/hosts/adb_host.py
,
Jul 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/132c7a9b1075acf9c120a84cd24ea12ef05a9e33 commit 132c7a9b1075acf9c120a84cd24ea12ef05a9e33 Author: Dan Shi <dshi@google.com> Date: Wed Jul 26 23:46:54 2017 [autotest] Make result_tools run faster 1. Add a module flag to only deploy result_tools to dut once across the test. This avoid 0.2s for no-op rsync calls to copy result_tools. 2. Add excludes for only deploy py modules to dut. 3. Only update original size at the end of build_from_path. This prevents recursive updating parent directories' size information, and reduce the run time to 1/5, for a result with 600 files (more savings for results with large number of files.) BUG= chromium:716218 TEST=local run test, check files on the dut. Change-Id: Iac98ed01c1e372e98d353236f8ba8184a63104e7 Reviewed-on: https://chromium-review.googlesource.com/584864 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/132c7a9b1075acf9c120a84cd24ea12ef05a9e33/client/bin/result_tools/result_info.py [modify] https://crrev.com/132c7a9b1075acf9c120a84cd24ea12ef05a9e33/client/bin/result_tools/runner.py
,
Jul 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f4fed38f8c0f0fdec690b2354c1a1722c673c915 commit f4fed38f8c0f0fdec690b2354c1a1722c673c915 Author: Dan Shi <dshi@google.com> Date: Fri Jul 28 02:56:28 2017 [autotest] Fix a log formatting bug BUG= chromium:716218 TEST=None Change-Id: Ib59046f526e3199391f83665c28838b43f395564 Reviewed-on: https://chromium-review.googlesource.com/590669 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Paul Hobbs <phobbs@google.com> [modify] https://crrev.com/f4fed38f8c0f0fdec690b2354c1a1722c673c915/server/hosts/abstract_ssh.py
,
Jul 28 2017
,
Jul 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/36c4c69318083e7fe12a567b081a5d6f7bcc0404 commit 36c4c69318083e7fe12a567b081a5d6f7bcc0404 Author: Dan Shi <dshi@google.com> Date: Fri Jul 28 22:04:25 2017 [autotest] Add a success field in the metrics for result tools. This allows metrics to be filtered by only successful calls, removing the noise from offline duts. BUG= chromium:716218 TEST=local run test Change-Id: I175f4e907a9734ef3bca3ce54f451ad72a8ff036 Reviewed-on: https://chromium-review.googlesource.com/590951 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Paul Hobbs <phobbs@google.com> [modify] https://crrev.com/36c4c69318083e7fe12a567b081a5d6f7bcc0404/client/bin/result_tools/runner.py
,
Aug 16 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/4f01f0b1857cfbf948f7c0345604cc262ca7ef50 commit 4f01f0b1857cfbf948f7c0345604cc262ca7ef50 Author: Dan Shi <dshi@google.com> Date: Wed Aug 16 22:31:12 2017
,
Aug 24 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/ebef29a37964a30f707d56c4ce6dbb9441a7145a commit ebef29a37964a30f707d56c4ce6dbb9441a7145a Author: Dan Shi <dshi@google.com> Date: Thu Aug 24 18:39:51 2017
,
Aug 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/451e8454c7336e3d8d71c4b6b30ed623097a2c91 commit 451e8454c7336e3d8d71c4b6b30ed623097a2c91 Author: Dan Shi <dshi@google.com> Date: Fri Aug 25 19:09:04 2017 [autotest] Protect perf.data from being throttled. *perf.data can be compressed but not deleted. This change prevents that from happening. BUG= chromium:716218 TEST=unittest Change-Id: Ia08bdabe6b251497aa6a402323198d5b8275ab72 Reviewed-on: https://chromium-review.googlesource.com/633853 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Ting-Yuan Huang <laszio@chromium.org> [modify] https://crrev.com/451e8454c7336e3d8d71c4b6b30ed623097a2c91/client/bin/result_tools/delete_file_throttler.py [modify] https://crrev.com/451e8454c7336e3d8d71c4b6b30ed623097a2c91/client/bin/result_tools/delete_file_throttler_unittest.py [modify] https://crrev.com/451e8454c7336e3d8d71c4b6b30ed623097a2c91/client/bin/result_tools/dedupe_file_throttler.py [modify] https://crrev.com/451e8454c7336e3d8d71c4b6b30ed623097a2c91/client/bin/result_tools/shrink_file_throttler.py
,
Aug 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/83a60bbd061ab37e0e6791563644c9efb8064fb3 commit 83a60bbd061ab37e0e6791563644c9efb8064fb3 Author: Dan Shi <dshi@google.com> Date: Fri Aug 25 19:09:01 2017 [autotest] Add a conversion in result info for backwards compatibility For tests running with server-side packaging, the result tools are from older branches, which has a different format. This change checks for the format difference and convert it to the latest if needed. The conversion can be removed after R62 is in stable channel. BUG= chromium:716218 TEST=merge summary from older format to new one. Change-Id: I618c290e1a0231a9c1843be020b5c814e02f969e Reviewed-on: https://chromium-review.googlesource.com/634266 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/83a60bbd061ab37e0e6791563644c9efb8064fb3/client/bin/result_tools/result_info.py
,
Sep 1 2017
,
Jan 22 2018
,
Jan 23 2018
|
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by jrbarnette@chromium.org
, Apr 28 2017