New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 602562 link

Starred by 3 users

Issue metadata

Status: Verified
Owner:
Closed: Apr 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: Bug



Sign in to add a comment

Test discovery runs too slow with ssh to devserver

Project Member Reported by dshi@chromium.org, Apr 12 2016

Issue description

We are seeing extremely slow performance on test discovery when suite job starts. One cq run timed out due to that.
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/59756332-chromeos-test/hostless/debug

04/11 22:24:33.409 DEBUG|             suite:1141| Parsing control files ...
04/11 22:24:33.414 DEBUG|        base_utils:0177| Running 'ssh 172.17.40.28 'curl "http://172.17.40.28:8082/controlfiles?control_path=server/site_tests/hardware_MemoryIntegrity/control&build=x86-zgb-paladin/R52-8181.0.0-rc3"''
04/11 22:24:34.554 DEBUG|        base_utils:0177| Running 'ssh 172.17.40.28 'curl "http://172.17.40.28:8082/controlfiles?control_path=client/site_tests/enterprise_CFM_Perf/control&build=x86-zgb-paladin/R52-8181.0.0-rc3"''
....
04/11 22:53:47.209 DEBUG|             suite:0906| Discovered 54 stable tests.

It took almost 20 minutes for the suite job to get all the tests and actually starts running test. If the repeated ssh/curl is the bottleneck we can't work around, we can consider add a devserver call to make test discovery a single devserver call.

Assign to Xixuan for initial investigation. I can help with the devserver call if needed.
 

Comment 1 by dshi@chromium.org, Apr 12 2016

Cc: jrbarnette@chromium.org
Labels: -Pri-1 Pri-0
This CQ test took 40mins to get tests:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/59805497-chromeos-test/hostless/debug

04/12 06:36:51.707 DEBUG|             suite:1141| Parsing control files ...
04/12 07:17:01.186 DEBUG|             suite:0906| Discovered 52 stable tests.

I will say disable ssh first. Find a solution and apply it again.
Project Member

Comment 2 by bugdroid1@chromium.org, Apr 12 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c9661243a2c414a5fae6f188f8f86231e6d95f15

commit c9661243a2c414a5fae6f188f8f86231e6d95f15
Author: xixuan <xixuan@google.com>
Date: Tue Apr 12 17:42:57 2016

Project Member

Comment 3 by bugdroid1@chromium.org, Apr 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/dev-util/+/7efd000a8966c3296b3b57b6f069f8914d11e4e8

commit 7efd000a8966c3296b3b57b6f069f8914d11e4e8
Author: xixuan <xixuan@google.com>
Date: Thu Apr 14 22:34:01 2016

DevServer: Add a new devserver call to list all control files' content.

Now when a suite job is created, it calls devserver to list all control
files' path, and then call devserver X times to get the content of each
control file, where X equals to the length of the control file list.
Since a suite job may fetch more than 1000+ control files, and all the
thouands of calls are made over ssh. It's significantly slower than
previous directly calling devserver by http.

The new call 'list_suite_controls' goes through the lists of all control files
for a suite and a build, and return the contents of these control files.

BUG= chromium:602562 
TEST=local run devserver
curl
http://127.0.0.1:8082/list_suite_controls?suite_name=suite_attr_wrapper&build=veyron_speedy-paladin/R52-8182.0.0-rc3
expected return:
["# Copyright control_file_content1", ..., "# Copyright
control_file_contentX"]

Change-Id: Id5e46e31212eda98d0e044be20cf657985f985bf
Reviewed-on: https://chromium-review.googlesource.com/338931
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/7efd000a8966c3296b3b57b6f069f8914d11e4e8/devserver.py

Project Member

Comment 4 by bugdroid1@chromium.org, Apr 19 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/dev-util/+/a1cd65265acb2f125aeecb7c96b4a7322fc6bd3d

commit a1cd65265acb2f125aeecb7c96b4a7322fc6bd3d
Author: Dan Shi <dshi@google.com>
Date: Mon Apr 18 23:07:21 2016

Change list_suite_controls to return a dictionary of file_path:content

File path is needed to parse the test name.

BUG= chromium:602562 
TEST=local run devserver
curl http://127.0.0.1:8080/list_suite_controls?suite_name=suite_attr_wrapper\&build=veyron_jerry-release/R51-8099.0.0
curl http://127.0.0.1:8080/stage?archive_url=gs://chromeos-image-archive/veyron_jerry-release/R51-8099.0.0/\&artifacts=test_suites,control_files

Change-Id: I13ff3811ceaff71fb054a0981e2d659d1d61c6a3
Reviewed-on: https://chromium-review.googlesource.com/339500
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/a1cd65265acb2f125aeecb7c96b4a7322fc6bd3d/devserver.py

Project Member

Comment 5 by bugdroid1@chromium.org, Apr 19 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0f7755d1e9ceca25f95ebb01dd05f040f154cba3

commit 0f7755d1e9ceca25f95ebb01dd05f040f154cba3
Author: xixuan <xixuan@google.com>
Date: Mon Apr 18 21:49:12 2016

Autotest: Make autotest call the new devserver API list_suite_controls

Related to CL:338931, this CL is to change the code in autotest to call
the API: list_suite_controls, to get the control contents of all control
files in one call to devserver.

BUG= chromium:602562 
TEST=Ran dev_server_unittest, control_file_getter_unittest,
suite_unittest.py, site_rpc_interface_unittest for
ENABLE_CONTROLS_IN_BATCH=False/True;
Ran test_suite:dummy in local autotest instance using local devserver
for ENABLE_CONTROLS_IN_BATCH=False/True.

Change-Id: I075fad07b79827a5d78b7de371bba845fcdf09b1
Reviewed-on: https://chromium-review.googlesource.com/339480
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/control_file_getter.py
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/suite.py
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/client/common_lib/cros/dev_server_unittest.py
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/frontend/afe/site_rpc_interface.py
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/global_config.ini
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/client/common_lib/error.py
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/suite_unittest.py
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/server/cros/dynamic_suite/control_file_getter_unittest.py
[modify] https://crrev.com/0f7755d1e9ceca25f95ebb01dd05f040f154cba3/client/common_lib/cros/dev_server.py

Project Member

Comment 6 by bugdroid1@chromium.org, Apr 19 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/dc97ca0af73e154097378b5588da614341f4fa8c

commit dc97ca0af73e154097378b5588da614341f4fa8c
Author: xixuan <xixuan@google.com>
Date: Tue Apr 19 22:25:03 2016

Project Member

Comment 7 by sheriffbot@chromium.org, Apr 26 2016

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 8 by xixuan@chromium.org, Apr 26 2016

Status: Verified (was: Assigned)

Comment 9 by benhenry@google.com, Apr 27 2016

Components: Infra>Client>ChromeOS
Labels: -Infra-ChromeOS

Sign in to add a comment