Test discovery runs too slow with ssh to devserver |
|||
Issue descriptionWe are seeing extremely slow performance on test discovery when suite job starts. One cq run timed out due to that. https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/59756332-chromeos-test/hostless/debug 04/11 22:24:33.409 DEBUG| suite:1141| Parsing control files ... 04/11 22:24:33.414 DEBUG| base_utils:0177| Running 'ssh 172.17.40.28 'curl "http://172.17.40.28:8082/controlfiles?control_path=server/site_tests/hardware_MemoryIntegrity/control&build=x86-zgb-paladin/R52-8181.0.0-rc3"'' 04/11 22:24:34.554 DEBUG| base_utils:0177| Running 'ssh 172.17.40.28 'curl "http://172.17.40.28:8082/controlfiles?control_path=client/site_tests/enterprise_CFM_Perf/control&build=x86-zgb-paladin/R52-8181.0.0-rc3"'' .... 04/11 22:53:47.209 DEBUG| suite:0906| Discovered 54 stable tests. It took almost 20 minutes for the suite job to get all the tests and actually starts running test. If the repeated ssh/curl is the bottleneck we can't work around, we can consider add a devserver call to make test discovery a single devserver call. Assign to Xixuan for initial investigation. I can help with the devserver call if needed.
,
Apr 12 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c9661243a2c414a5fae6f188f8f86231e6d95f15 commit c9661243a2c414a5fae6f188f8f86231e6d95f15 Author: xixuan <xixuan@google.com> Date: Tue Apr 12 17:42:57 2016
,
Apr 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/dev-util/+/7efd000a8966c3296b3b57b6f069f8914d11e4e8 commit 7efd000a8966c3296b3b57b6f069f8914d11e4e8 Author: xixuan <xixuan@google.com> Date: Thu Apr 14 22:34:01 2016 DevServer: Add a new devserver call to list all control files' content. Now when a suite job is created, it calls devserver to list all control files' path, and then call devserver X times to get the content of each control file, where X equals to the length of the control file list. Since a suite job may fetch more than 1000+ control files, and all the thouands of calls are made over ssh. It's significantly slower than previous directly calling devserver by http. The new call 'list_suite_controls' goes through the lists of all control files for a suite and a build, and return the contents of these control files. BUG= chromium:602562 TEST=local run devserver curl http://127.0.0.1:8082/list_suite_controls?suite_name=suite_attr_wrapper&build=veyron_speedy-paladin/R52-8182.0.0-rc3 expected return: ["# Copyright control_file_content1", ..., "# Copyright control_file_contentX"] Change-Id: Id5e46e31212eda98d0e044be20cf657985f985bf Reviewed-on: https://chromium-review.googlesource.com/338931 Commit-Ready: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/7efd000a8966c3296b3b57b6f069f8914d11e4e8/devserver.py
,
Apr 19 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/dev-util/+/a1cd65265acb2f125aeecb7c96b4a7322fc6bd3d commit a1cd65265acb2f125aeecb7c96b4a7322fc6bd3d Author: Dan Shi <dshi@google.com> Date: Mon Apr 18 23:07:21 2016 Change list_suite_controls to return a dictionary of file_path:content File path is needed to parse the test name. BUG= chromium:602562 TEST=local run devserver curl http://127.0.0.1:8080/list_suite_controls?suite_name=suite_attr_wrapper\&build=veyron_jerry-release/R51-8099.0.0 curl http://127.0.0.1:8080/stage?archive_url=gs://chromeos-image-archive/veyron_jerry-release/R51-8099.0.0/\&artifacts=test_suites,control_files Change-Id: I13ff3811ceaff71fb054a0981e2d659d1d61c6a3 Reviewed-on: https://chromium-review.googlesource.com/339500 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/a1cd65265acb2f125aeecb7c96b4a7322fc6bd3d/devserver.py
,
Apr 19 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0f7755d1e9ceca25f95ebb01dd05f040f154cba3 commit 0f7755d1e9ceca25f95ebb01dd05f040f154cba3 Author: xixuan <xixuan@google.com> Date: Mon Apr 18 21:49:12 2016 Autotest: Make autotest call the new devserver API list_suite_controls Related to CL:338931, this CL is to change the code in autotest to call the API: list_suite_controls, to get the control contents of all control files in one call to devserver. BUG= chromium:602562 TEST=Ran dev_server_unittest, control_file_getter_unittest, suite_unittest.py, site_rpc_interface_unittest for ENABLE_CONTROLS_IN_BATCH=False/True; Ran test_suite:dummy in local autotest instance using local devserver for ENABLE_CONTROLS_IN_BATCH=False/True. Change-Id: I075fad07b79827a5d78b7de371bba845fcdf09b1 Reviewed-on: https://chromium-review.googlesource.com/339480 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Xixuan Wu <xixuan@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/control_file_getter.py [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/suite.py [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/client/common_lib/cros/dev_server_unittest.py [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/frontend/afe/site_rpc_interface.py [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/global_config.ini [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/client/common_lib/error.py [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/suite_unittest.py [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/control_file_getter_unittest.py [modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/client/common_lib/cros/dev_server.py
,
Apr 19 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/dc97ca0af73e154097378b5588da614341f4fa8c commit dc97ca0af73e154097378b5588da614341f4fa8c Author: xixuan <xixuan@google.com> Date: Tue Apr 19 22:25:03 2016
,
Apr 26 2016
Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable? If a fix is in active development, please set the status to Started. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Apr 26 2016
,
Apr 27 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by dshi@chromium.org
, Apr 12 2016Labels: -Pri-1 Pri-0