New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 868820 link

Starred by 1 user
Status: Assigned
Owner:
vapier@chromium.org
Cc:
█ oka@chromium.org
hidehiko@chromium.org
nya@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Chrome
Pri: 3
Type: Bug



Sign in to add a comment

Chromite utils are getting too fat

Project Member Reported by oka@chromium.org, Jul 30 
Spun off from https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1107024/34/common-mk/common_utils.py#11.

Even for small scripts which only uses RunCommand to do the equivalent of subprocess.check_output(), cros_build_lib draws in a lot of unnecessary libraries.
Importing of cros_build_lib takes 6x time than subprocess, which is critical for a small program executed many times.

 $ python
 >>> import sys
 >>> len(sys.modules)
 45
 >>> import chromite.lib.cros_build_lib
 >>> len(sys.modules)
 312
 $  time python -c 'import chromite.lib.cros_build_lib'

real    0m0.198s
user    0m0.140s
sys     0m0.056s
 $  time python -c 'import subprocess'

real    0m0.031s
user    0m0.024s
sys     0m0.004s

Comment 1 by vapier@google.com, Jul 31

Components: Infra>Client>ChromeOS>Build
Labels: OS-Chrome
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 3 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c70f2853a44a70bd223b7a881ece5aaa637184a5

commit c70f2853a44a70bd223b7a881ece5aaa637184a5
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Aug 03 08:31:19 2018

cros_collections: split out collections related logic to a new module

This code is standalone and infrequently used, so give it its own
module outside of the large cros_build_lib dumping ground.

BUG=chromium:868820
TEST=unittests pass

Change-Id: I2390346564f4e1784e5203ed72bf1f070a9d22a9
Reviewed-on: https://chromium-review.googlesource.com/1161462
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Lann Martin <lannm@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/c70f2853a44a70bd223b7a881ece5aaa637184a5/lib/commandline.py
[add] https://crrev.com/c70f2853a44a70bd223b7a881ece5aaa637184a5/lib/cros_collections.py
[modify] https://crrev.com/c70f2853a44a70bd223b7a881ece5aaa637184a5/lib/cros_build_lib.py
[add] https://crrev.com/c70f2853a44a70bd223b7a881ece5aaa637184a5/lib/cros_collections_unittest.py
[modify] https://crrev.com/c70f2853a44a70bd223b7a881ece5aaa637184a5/cros/test/usergroup_baseline.py
[modify] https://crrev.com/c70f2853a44a70bd223b7a881ece5aaa637184a5/lib/cros_build_lib_unittest.py
[add] https://crrev.com/c70f2853a44a70bd223b7a881ece5aaa637184a5/lib/cros_collections_unittest
Project Member

Comment 3 by bugdroid1@chromium.org, Aug 7 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/1388eb036e1bc0fbb9dc28c7feeb64cbda383914

commit 1388eb036e1bc0fbb9dc28c7feeb64cbda383914
Author: Mike Frysinger <vapier@chromium.org>
Date: Tue Aug 07 18:04:57 2018

cros_collections: move GroupByKey/GroupNamedtuplesByKey/InvertDictionary here

These are pretty standalone and seem to more logically fit in with the
new cros_collections module than the existing cros_build_lib dumping.

BUG=chromium:868820
TEST=unittests pass

Change-Id: I9746a4c75c327042797bf206d2e37fdc12107200
Reviewed-on: https://chromium-review.googlesource.com/1161976
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/cbuildbot/relevant_changes.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/lib/cros_collections.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/scripts/summarize_build_stats.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/cbuildbot/stages/sync_stages.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/lib/builder_status_lib.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/lib/cros_build_lib.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/cbuildbot/stages/handle_changes_stages.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/lib/cros_collections_unittest.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/lib/build_time_stats.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/lib/cros_build_lib_unittest.py
[modify] https://crrev.com/1388eb036e1bc0fbb9dc28c7feeb64cbda383914/cbuildbot/stages/report_stages.py
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 3 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/a504beb965b11ae4849d2c063cd9cc225dbce5bb

commit a504beb965b11ae4849d2c063cd9cc225dbce5bb
Author: oka@chromium.org <oka@chromium.org>
Date: Mon Sep 03 09:20:53 2018

common-mk: comment on common_utils.py and make it posix compliant

Addressed comments in
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1107024/34/common-mk/common_utils.py#11.

TEST=cros_workon_make --board=eve --test arc-adbd
BUG=chromium:868820

Change-Id: I33153d7f6c4955c4e53504461a6aacd1616b1931
Reviewed-on: https://chromium-review.googlesource.com/1154856
Commit-Ready: Keigo Oka <oka@chromium.org>
Tested-by: Keigo Oka <oka@chromium.org>
Reviewed-by: Keigo Oka <oka@chromium.org>

[modify] https://crrev.com/a504beb965b11ae4849d2c063cd9cc225dbce5bb/common-mk/common_utils.py

Comment 5 by hidehiko@chromium.org, Dec 11

Cc: hidehiko@chromium.org
►

Sign in to add a comment