New issue
Advanced search Search tips

Issue 823996 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Dec 30
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

deploy_chrome: support deploying w/out needing sysroot+toolchain download

Project Member Reported by bpastene@chromium.org, Mar 21 2018

Issue description

We'd like to run some of the chromite scripts on our bots. Some of these scripts (eg: deploy_chrome.py) don't need images, toolchains, etc. But entering the chrome-sdk by default downloads these things and more. On an empty cache for board=arm-generic, this is what gets downloaded:
- gs://chromiumos-sdk/2018/03/armv7a-cros-linux-gnueabi-2018.03.12.172310.tar.xz
- gs://chromeos-image-archive/arm-generic-full/R67-10486.0.0-b2378850/sysroot_chromeos-base_chromeos-chrome.tar.xz
- gs://chromeos-image-archive/arm-generic-full/R67-10486.0.0-b2378850/environment_chromeos-base_chromeos-chrome.tar.xz
- gs://chromeos-image-archive/arm-generic-full/R67-10486.0.0-b2378850/metadata.json
- gs://chromeos-image-archive/arm-generic-full/LATEST-10486.0.0

Some of those can be pretty big and take a bit to download+extract. Could we add a --no-download option or somesuch to the sdk that we can use when we're pretty sure we don't need any of the artifacts?

 
What are we doing that requires the simple chrome environment but not the toolchain? I thought that was the primary point of simple chrome.

e.g. I just tested that deploy_chrome works fine from inside a chrome repo without entering the simple chrome environment:

./third_party/chromite/bin/deploy_chrome --board=samus --build-dir=out_samus/Release --to 100.115.91.61

The only difference is that --board has to be specified explicitly outside of simple chrome.

See https://chromium-review.googlesource.com/c/chromium/src/+/971303/2/build/chromeos/deploy_chrome_wrapper.py#46 for context, where I was trying to call it outside the sdk, but did indeed need to pass a board... so I was passing a nonsense string, which was kinda hacky.
Cc: vapier@chromium.org
It seems like the right thing there is to fix deploy_chrome, rather than adding another flag to support in chrome-sdk?

In theory, it may also make sense to not download the sysroot and environment for people who want to just use the VM, but not build chrome.

But yeah, I can fix deploy_chrome to not require the board.
Cc: -achuith@chromium.org
Owner: achuith@chromium.org
Status: Assigned (was: Untriaged)

Comment 6 by vapier@chromium.org, Mar 21 2018

deploy chrome wants to strip artifacts, and artifacts require a toolchain, and knowing which toolchain requires knowing the board

so if you didn't strip, or you specified the --strip-bin, and you specified a --build-dir, not needing a board and the sysroot/toolchain artifacts sounds fine.  and in reading the code, isn't this how it works now ?

Comment 7 by vapier@chromium.org, Mar 21 2018

Summary: deploy_chrome: support deploying w/out needing sysroot+toolchain download (was: Add option to cros chrome-sdk to skip downloading of sysroot and toolchain)
changing the summary as i'm fairly certain the concept of using `cros chrome-sdk` w/out a board/sysroot+toolchain makes no sense.  deploy_chrome is the only part that can work independently.
When I try running deploy_chrome from outside simple chrome without --nboard, this is what I get:

~/Work/chrome/src $ ./third_party/chromite/bin/deploy_chrome --build-dir=out_samus/Release --to $IP_ADDRESS
usage: deploy_chrome [-h]
...
deploy_chrome: error: --board is required when --build-dir is specified.

Comment 9 by vapier@chromium.org, Mar 21 2018

thanks, i misread that statement.  looking at board usage, it seems like board is only necessary when stripping and the strip tool is automatic.
_PrepareStagingDir -> options.build_dir -> _StripBinContext -> dostrip && not strip_bin -> cros_chrome_sdk.SDKFetcher

so we can refine that particular check to include the strip knobs
Labels: OS-Chrome
Status: Started (was: Assigned)
https://chromium-review.googlesource.com/c/chromiumos/chromite/+/977042
Project Member

Comment 11 by bugdroid1@chromium.org, Mar 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/31a3eb064ea32f48a704c1321f33458365d40d13

commit 31a3eb064ea32f48a704c1321f33458365d40d13
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Fri Mar 23 13:18:03 2018

deploy_chrome: Finesse --board requirements.

--board is only required for stripping, so it may be skipped if
--build-dir is not specified, --no-strip is specified, or --strip-bin
is specified.

BUG= chromium:823996 
TEST=manual + unittests

Change-Id: If2e46b3bdcc7c9670cf556d1e967ee362cf01b90
Reviewed-on: https://chromium-review.googlesource.com/977042
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/31a3eb064ea32f48a704c1321f33458365d40d13/scripts/deploy_chrome.py
[modify] https://crrev.com/31a3eb064ea32f48a704c1321f33458365d40d13/scripts/deploy_chrome_unittest.py

Project Member

Comment 12 by bugdroid1@chromium.org, Mar 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d3347e7ca583dd6cf9f36050757223d9cdcff1de

commit d3347e7ca583dd6cf9f36050757223d9cdcff1de
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Mar 23 15:35:32 2018

Roll src/third_party/chromite/ ce913b499..31a3eb064 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/ce913b499c4c..31a3eb064ea3

$ git log ce913b499..31a3eb064 --date=short --no-merges --format='%ad %ae %s'
2018-03-22 achuith deploy_chrome: Finesse --board requirements.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:823996 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I769462f350386abdb114b740c59231cbfaf81da1
Reviewed-on: https://chromium-review.googlesource.com/977887
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#545457}
[modify] https://crrev.com/d3347e7ca583dd6cf9f36050757223d9cdcff1de/DEPS

Project Member

Comment 13 by bugdroid1@chromium.org, Mar 24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f6bccdb18dc6e745be0d4b652f047ef1551418c3

commit f6bccdb18dc6e745be0d4b652f047ef1551418c3
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Sat Mar 24 23:20:15 2018

cros_vm: Determine SDK version outside SDK shell.

This CL allows us to launch a cros VM from outside the SDK shell.

If we're not in the SDK shell, the SDK_VERSION_ENV environment variable
will not be set. If we're somewhere in the chrome tree we can still
look at the misc cache in the cros sdk cache, which has the latest SDK
version.

BUG= chromium:823996 
TEST=manual

Change-Id: Ie34508f24d1d5146033b30638382ab9d5e464c30
Reviewed-on: https://chromium-review.googlesource.com/978779
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/f6bccdb18dc6e745be0d4b652f047ef1551418c3/scripts/cros_vm.py

Project Member

Comment 14 by bugdroid1@chromium.org, Mar 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dc4a7d91c392a1cd69f214268389429f7ee09d78

commit dc4a7d91c392a1cd69f214268389429f7ee09d78
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sun Mar 25 07:35:28 2018

Roll src/third_party/chromite/ e6139afa1..44123785c (2 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/e6139afa10f8..44123785ca9a

$ git log e6139afa1..44123785c --date=short --no-merges --format='%ad %ae %s'
2018-03-24 ihf chromeos-config: mark eve-arcnext-chrome-pfq important.
2018-03-23 achuith cros_vm: Determine SDK version outside SDK shell.

Created with:
  roll-dep src/third_party/chromite
BUG=chromium:825539, chromium:823996 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Icb8dcad2d3ee70b331e2446f9f873e2c21c385d8
Reviewed-on: https://chromium-review.googlesource.com/979598
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#545707}
[modify] https://crrev.com/dc4a7d91c392a1cd69f214268389429f7ee09d78/DEPS

Project Member

Comment 15 by bugdroid1@chromium.org, May 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a8050d3d86183435df694a61e3a5cf4062a12d5f

commit a8050d3d86183435df694a61e3a5cf4062a12d5f
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri May 11 06:18:06 2018

cros_run_vm_test: Pass board and cache dir args down to deploy_chrome.

BUG= chromium:823996 
TEST=ran cros_run_vm_test locally

Change-Id: I04f9ad017728177b13e10194089a3d75525fbd23
Reviewed-on: https://chromium-review.googlesource.com/1050551
Commit-Ready: Ben Pastene <bpastene@chromium.org>
Tested-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/a8050d3d86183435df694a61e3a5cf4062a12d5f/scripts/cros_run_vm_test.py

Project Member

Comment 16 by bugdroid1@chromium.org, May 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9f750a7479973146805e29c22b7b8a5fb834a18a

commit 9f750a7479973146805e29c22b7b8a5fb834a18a
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri May 11 10:08:37 2018

Roll src/third_party/chromite/ 394f1c63d..a8050d3d8 (1 commit)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/394f1c63d58a..a8050d3d8618

$ git log 394f1c63d..a8050d3d8 --date=short --no-merges --format='%ad %ae %s'
2018-05-08 bpastene cros_run_vm_test: Pass board and cache dir args down to deploy_chrome.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:823996 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I1fb09689cbf824cf46c5332a473261f6c6ca55b8
Reviewed-on: https://chromium-review.googlesource.com/1055193
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#557833}
[modify] https://crrev.com/9f750a7479973146805e29c22b7b8a5fb834a18a/DEPS

Status: Fixed (was: Started)

Sign in to add a comment