New issue
Advanced search Search tips

Issue 739466 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Jul 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

'from autotest_lib.server import hosts' takes 0.7 seconds

Project Member Reported by pprabhu@chromium.org, Jul 5 2017

Issue description

Forked from https://bugs.chromium.org/p/chromium/issues/detail?id=718181#c18

Importing the host module does something so supremely stupid that it takes human-recognizable long.

This hits us in many many places.
- We suspect that this added ~2 seconds to the scheduler tick.
- This is probably also slowing down the tests themselves because autoserv also creates the host object.

 
Some ad-hoc scrubbing of why import hosts is so slow via this (never to be landed CL): 
https://chromium-review.googlesource.com/559974

Lowest hanging fruits:

- server/site_utils.py
 - chromite imports take 1-2 seconds each and are not needed by most users of this module.
- server/hosts/adb_host.py (loses 0.15 seconds)
 - importing afe_utils.py (0.05 seconds)
  - creates a  global AFE object (0.025 seconds)
  - importing frontend_wrappers (0.025 seconds)
   - importing server.frontend (0.025)
    - importing tko.db (most of the time in autotest_stats. Let's just drop this)
Project Member

Comment 2 by bugdroid1@chromium.org, Jul 6 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/240b4672a10cc1621f909bd42248bc15d9ca33ab

commit 240b4672a10cc1621f909bd42248bc15d9ca33ab
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Jul 06 04:54:57 2017

[autotest] Delay hosts module import in drone_utility

The hosts module import takes ~.75 seconds. This has been bottlenecking
the scheduler tick by slowing down each drone_utility refresh call by
~1.5 seconds. Delay the import as a hack while we work on a real fix.

BUG= chromium:718181 
BUG= chromium:739466 
TEST=None

Change-Id: I3966259594bc5b2e5e1dd8592c8dcfd9cbda1c6c
Reviewed-on: https://chromium-review.googlesource.com/559938
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/240b4672a10cc1621f909bd42248bc15d9ca33ab/scheduler/drone_utility.py

Project Member

Comment 3 by bugdroid1@chromium.org, Jul 6 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/16b46f8039a9e29fa036f1347dcadcacac6a7d77

commit 16b46f8039a9e29fa036f1347dcadcacac6a7d77
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Jul 06 20:59:20 2017

[autotest] Delay chromite imports in site_utils

The chromite imports in site_utils are specific to some specialized
use-cases, but cost all users 2-3 seconds total during module import.
This was a bottleneck in the scheduler tick time.

The correct fix is to break apart server/site_utils since it does way
too much. The hack is to delay the imports.

BUG= chromium:739466 
TEST=None

Change-Id: Ia28481414b5db8b413d70839076ddcc50d747f6e
Reviewed-on: https://chromium-review.googlesource.com/559831
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/16b46f8039a9e29fa036f1347dcadcacac6a7d77/server/site_utils.py

Owner: pprabhu@chromium.org
Project Member

Comment 5 by bugdroid1@chromium.org, Jul 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c

commit ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Wed Jul 12 00:48:00 2017

[autotest] Drop remaining references to autotest_stats

We are neither looking at the graphite dashboard anymore, not
maintaining it really. OTOH, the autotest_stats module costs ~.25
seconds on module load time, hitting us in places where it matters.

So, drop all uses, also removing the false sense of security.

BUG= chromium:739466 
TEST=unittests.

Change-Id: I0e4118653d929a5d55f4ff62b96e9c66e855efd7
Reviewed-on: https://chromium-review.googlesource.com/559832
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c/frontend/afe/rpc_utils.py
[modify] https://crrev.com/ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c/scheduler/monitor_db.py
[modify] https://crrev.com/ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c/frontend/afe/model_logic.py
[modify] https://crrev.com/ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c/tko/db.py
[modify] https://crrev.com/ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c/frontend/afe/json_rpc/serviceHandler.py
[modify] https://crrev.com/ba8adaf2506d4bf2b9b09bf54f2490ced31dd16c/frontend/afe/rpc_interface.py

Status: Fixed (was: Untriaged)
I've landed my first responder fixes.
import time is reduced somewhat (.63 seconds), but still quite large.

I don't intend to do more here.

Sign in to add a comment