prepare base image so nothing needs to be installed by lxc |
||||||||||
Issue descriptionLifted from b/32474906. I am worried we are putting heavy load on external sites. Lets make sure we cache all needed files or avoid the install. https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/82546591-chromeos-test/chromeos4-row6-rack9-host9/debug/ No distributions at all found for lockfile Storing debug log for failure in /root/.pip/pip.log) 10/25 12:51:22.055 WARNI| retry:0148| Retrying in 1.516466 seconds... 10/25 12:51:23.599 DEBUG| base_utils:0185| Running 'sudo apt-get update' 10/25 12:51:30.078 DEBUG| base_utils:0185| Running 'sudo apt-get install python-pip python-dev -y --force-yes' 10/25 12:51:30.431 DEBUG| lxc:0337| Packages are installed: ['python-pip', 'python-dev']. 10/25 12:51:30.432 DEBUG| base_utils:0185| Running 'sudo pip install lockfile' 10/25 12:52:16.046 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdError'>(Command <sudo pip install lockfile> failed, rc=1, Command returned non-zero exit status * Command: sudo pip install lockfile Exit status: 1 Duration: 45.5989170074 stdout: Downloading/unpacking lockfile Cannot fetch index base URL https://pypi.python.org/simple/ Could not find any downloads that satisfy the requirement lockfile Cleaning up... No distributions at all found for lockfile Storing debug log for failure in /root/.pip/pip.log) 10/25 12:52:16.047 WARNI| retry:0148| Retrying in 4.412007 seconds... 10/25 12:52:20.484 DEBUG| base_utils:0185| Running 'sudo apt-get update' 10/25 12:52:26.880 DEBUG| base_utils:0185| Running 'sudo apt-get install python-pip python-dev -y --force-yes' 10/25 12:52:27.282 DEBUG| lxc:0337| Packages are installed: ['python-pip', 'python-dev']. 10/25 12:52:27.282 DEBUG| base_utils:0185| Running 'sudo pip install lockfile' 10/25 12:52:34.848 DEBUG| lxc:0349| Python packages are installed: ['lockfile'].
,
Oct 28 2016
It is in a container. I am thinking maybe it is easiest to include lockfile with the autotest sources.
,
Oct 28 2016
the autotest repo is already a minefield of people committing tarballs/source trees of random upstream projects :/. would be really nice if we could stop doing that. wrt pip, what we do in chromite is commit the output of pip so it's ready to use: https://chromium.googlesource.com/chromiumos/chromite/pip_packages/+/master maybe that wouldn't be the worst ...
,
Oct 28 2016
lockfile is rather a small package to install. As long as retry can have it installed properly, I don't think it needs to be pre-installed in base container. For external packages, if it's too large, like jdk, we can upgrade base container to make the server-side packaging setup faster. label this as a fixit bug. Next time we need to update base container, we can do this.
,
Jan 19 2017
Installing python-pip and python-dev is taking 2-3 minutes now. We need to move this out of lxc and into the base image. Interestingly lockfile is not the problem but should be moved as well. 01/18 11:27:24.867 DEBUG| base_utils:0185| Running 'sudo apt-get update' 01/18 11:27:36.582 DEBUG| base_utils:0185| Running 'sudo apt-get install python-pip python-dev -y --force-yes' 01/18 11:30:33.719 DEBUG| lxc:0343| Packages are installed: ['python-pip', 'python-dev']. 01/18 11:30:33.719 DEBUG| base_utils:0185| Running 'sudo pip install lockfile' 01/18 11:30:34.414 DEBUG| lxc:0355| Python packages are installed: ['lockfile']. 01/18 11:27:27.984 DEBUG| base_utils:0185| Running 'sudo apt-get update' 01/18 11:27:38.846 DEBUG| base_utils:0185| Running 'sudo apt-get install python-pip python-dev -y --force-yes' 01/18 11:29:44.536 DEBUG| lxc:0343| Packages are installed: ['python-pip', 'python-dev']. 01/18 11:29:44.537 DEBUG| base_utils:0185| Running 'sudo pip install lockfile' 01/18 11:29:45.288 DEBUG| lxc:0355| Python packages are installed: ['lockfile'].
,
Jan 19 2017
python-pip python-dev should be in the base image already. Do you have the link to the test? maybe that particular drone/shard's base image is not updated.
,
Jan 19 2017
This for instance https://wmatrix.googleplex.com/testrun/unfiltered?test_ids=417439472 01/18 20:06:16.431 DEBUG| base_utils:0185| Running 'sudo apt-get install python-pip python-dev -y --force-yes' 01/18 20:08:41.586 DEBUG| lxc:0343| Packages are installed: ['python-pip', 'python-dev']. The drone is chromeos-server51.cbf.corp.google.com and the packages are already installed and newest. Strange. chromeos-test@chromeos-server51:~$ dpkg -l python-pip Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=======================-================-================-==================================================== ii python-pip 1.5.4-1ubuntu4 all alternative Python package installer chromeos-test@chromeos-server51:~$ chromeos-test@chromeos-server51:~$ dpkg -l python-dev Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=======================-================-================-==================================================== ii python-dev 2.7.5-5ubuntu3 amd64 header files and a static library for Python (defaul chromeos-test@chromeos-server51:~$ chromeos-test@chromeos-server51:~$ sudo apt-get install python-pip python-dev -y --force-yes Reading package lists... Done Building dependency tree Reading state information... Done python-dev is already the newest version (2.7.5-5ubuntu3). python-pip is already the newest version (1.5.4-1ubuntu4). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
,
Jan 19 2017
root@test_1:/# dpkg -l python-pip Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-====================================================================== ii python-pip 1.5.4-1ubuntu3 all alternative Python package installer root@test_1:/# root@test_1:/# root@test_1:/# root@test_1:/# dpkg -l python-dev Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-====================================================================== un python-dev <none> <none> (no description available) root@test_1:/# So the packages are already installed. However, after I run sudo apt-get update, and run "sudo apt-get install python-pip python-dev -y --force-yes" it does take some time to install some updates (python2.7-de) I will do the update in base container with java 8
,
Jan 19 2017
,
Jan 19 2017
Can you make sure lockfile is going to be on the container as well? It may already have been there but the operation takes 7-15s.
,
Jan 19 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/8acacd7fa925497373571161622c1fcf137b4a11 commit 8acacd7fa925497373571161622c1fcf137b4a11 Author: Dan Shi <dshi@google.com> Date: Thu Jan 19 22:11:48 2017
,
Jan 21 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/b8a8fc73227d7f56421d92ce4e5b444352f98523 commit b8a8fc73227d7f56421d92ce4e5b444352f98523 Author: Dan Shi <dshi@google.com> Date: Fri Jan 20 20:38:47 2017
,
Jan 24 2017
We have the new base image, but we still wait 2 minutes on apt-get. Do you want me to remove that code from tradefed? 01/23 16:40:00.398 INFO | tradefed_test:0158| Hostname: chromeos4-row6-rack9-host7 01/23 16:40:00.484 DEBUG| base_utils:0185| Running 'sudo apt-get update' 01/23 16:40:10.009 DEBUG| base_utils:0185| Running 'sudo apt-get install unzip default-jre -y --force-yes' 01/23 16:40:12.313 ERROR| metrics:0429| Caught exception while flushing: No module named pyasn1.codec.ber 01/23 16:41:42.306 DEBUG| lxc:0343| Packages are installed: ['unzip', 'default-jre']. 01/23 16:41:42.307 DEBUG| base_utils:0185| Running 'java -version' 01/23 16:41:42.310 ERROR| metrics:0429| Caught exception while flushing: No module named pyasn1.codec.ber 01/23 16:41:44.052 ERROR| base_utils:0280| [stderr] openjdk version "1.8.0_45-internal"
,
Jan 24 2017
maybe change the code to install package to: Check if package already exists, if so, don't install unless want to ensure the latest? https://cs.corp.google.com/android/external/autotest/site_utils/lxc.py?rcl=4ee2d43296ea970a30b11715ac443a97b31fe649&l=342
,
Jan 25 2017
I don't understand. We literally run 1000+ CTS jobs per board and day. This is more than a machine day wasted when waiting 2 minutes per job. CTS Jobs right now have nearly a 10 minute overhead. While some can't be avoided, we really can't afford this. I am going to remove all lxc.install_packages() from tradefed_test. The base images must provide everything. Even querying takes too long. Sorry, we need to slim this.
,
Jan 25 2017
Sounds good, the test runs in the lab only anyway. Feel free to chump the CL. Ping me if there is any issue.
,
Jan 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4686377b7ca79d591f984d53be819960ac64e484 commit 4686377b7ca79d591f984d53be819960ac64e484 Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Jan 25 08:53:44 2017 tradefed_test: remove dependency installs. The base image must come with all required packages. The overhead of installing packages thousands of times a day is too high. BUG= chromium:660392 TEST=CQ will test. Change-Id: I9a677cba2e36e57d2817d6c6de49602ef75a1010 Reviewed-on: https://chromium-review.googlesource.com/432009 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Rohit Makasana <rohitbm@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/4686377b7ca79d591f984d53be819960ac64e484/server/cros/tradefed_test.py
,
Jan 31 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/edd9352d7e93a3a6b59107fa83cfff1fcd92d34c commit edd9352d7e93a3a6b59107fa83cfff1fcd92d34c Author: Ilja H. Friedel <ihf@chromium.org> Date: Wed Jan 25 08:53:44 2017 tradefed_test: remove dependency installs. The base image must come with all required packages. The overhead of installing packages thousands of times a day is too high. BUG= chromium:660392 TEST=CQ will test. Change-Id: I9a677cba2e36e57d2817d6c6de49602ef75a1010 Reviewed-on: https://chromium-review.googlesource.com/432009 Commit-Ready: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Rohit Makasana <rohitbm@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> Reviewed-on: https://chromium-review.googlesource.com/435158 Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/edd9352d7e93a3a6b59107fa83cfff1fcd92d34c/server/cros/tradefed_test.py
,
Feb 6 2017
CL to reduce test overhead: https://chromium-review.googlesource.com/c/435463/
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/758ebd0b29876e9d2cf6722d2f362a47a51a3dcd commit 758ebd0b29876e9d2cf6722d2f362a47a51a3dcd Author: Dan Shi <dshi@google.com> Date: Mon Feb 06 20:47:04 2017 Do not force to install latest package in container. For server side test using SSP, installing packages requires to run apt-get update and other commands that take long time to finish and introduce unnecessary load on the drone. All these slow down the server side test by minutes. This change allows install_packages call to skip installing a package if it's already installed and user does not force the latest version of the package. BUG= chromium:660392 TEST=lxc_functional_test.py Change-Id: I4a8113a06fcec8062807899f7d25dfa4427746a8 Reviewed-on: https://chromium-review.googlesource.com/435463 Commit-Ready: Dan Shi <dshi@google.com> Tested-by: Dan Shi <dshi@google.com> Reviewed-by: Simran Basi <sbasi@chromium.org> [modify] https://crrev.com/758ebd0b29876e9d2cf6722d2f362a47a51a3dcd/client/bin/site_utils.py [modify] https://crrev.com/758ebd0b29876e9d2cf6722d2f362a47a51a3dcd/site_utils/lxc.py
,
Feb 6 2017
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by vapier@chromium.org
, Oct 28 2016