New issue
Advanced search Search tips

Issue 798617 link

Starred by 3 users

Issue metadata

Status: Duplicate
Merged: issue 729178
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Host packages config updated too early in SDK builder

Project Member Reported by manojgupta@chromium.org, Jan 3 2018

Issue description

Chromiumos-sdk builders uploads the host packages early so that Goma can pick new tookchain packages earlier. However, it also updates make.conf.amd64-host config file before SDKTest stage. SDKTest stage takes several hours to run.

For instance, the SDK builder commits show 5 hour difference between modification of FULL_BINHOST and SDK_LATEST_VERSION. 

commit 5eaf6899d02e14aa2a45fe13b8b936de56aadfc1
Author: chrome-bot <chrome-bot@chromium.org>
Date:   Mon Jan 1 20:38:01 2018 -0800

    make.conf.amd64-host: updating FULL_BINHOST
    
    Change-Id: I22aa0777295f404864baf3dc1889fbdcca70c0ba

commit 18edcc8fc80ac6b2c78100f9a3e2e8c989832e81
Author: chrome-bot <chrome-bot@chromium.org>
Date:   Tue Jan 2 01:27:12 2018 -0800

    sdk_version.conf: updating TC_PATH, SDK_LATEST_VERSION
    
    Change-Id: I23e46ad74e7f54f05ec8b8ff955c5dcbdb5633ab

As a consequence of updating the make.conf.amd64-host file early, other builders pick any updates to host packages before the SDK builder can finish and update the chroot version.
If the SDK builder passes later (common case) and the packages are good, there is no problem.

However, if there is something bad with an upreved host package and SDKTest stage fails, any invocation of update_chroot will pull in the new (bad) host package. Reverting the SDK/host's config files will fix the builders but any developer chroot will still contain the bad package till the bad package gets another uprev.

If a toolchain package is bad, the number of bad host packages that may get pulled in might be fairly large. And it may happen that the only way to fix the chroot is to recreate the SDK causing a big grief.
 
updating git right away was certainly not the intention.  it should still be delayed until after the SDKTest stages pass.
Cc: shinyak@chromium.org
shinyak@, could you double check if goma does not depends on FULL_BINHOST code?
Then, we should be able to delay the update till the test passes successfully.
Components: Infra>Goma
EstimatedDays: 1
Adding goma component.
EstimatedDays: ----
Project Member

Comment 5 by bugdroid1@chromium.org, Jan 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/772dcdf8c52655f5b137840ab4a92e8d84a33e37

commit 772dcdf8c52655f5b137840ab4a92e8d84a33e37
Author: Manoj Gupta <manojgupta@google.com>
Date: Sat Jan 20 20:08:55 2018

Chromiumos-sdk: Move SDK test stage before upload prebuilts.

Upload Prebuilts updates FULL_BINHOST without verifying that SDK Test
was ok. So move it after SDK Test stage.

Revert this CL after  bug 798617  is fixed.

BUG= chromium:798617 
TEST=SDK try job passed.

Change-Id: I7e3f8b6cc4e97a6c307f6efd0253bbcf8d26c0cf
Reviewed-on: https://chromium-review.googlesource.com/876655
Commit-Queue: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Trybot-Ready: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/772dcdf8c52655f5b137840ab4a92e8d84a33e37/cbuildbot/builders/sdk_builders.py

Owner: hidehiko@chromium.org
Components: Tools>ChromeOS-Toolchain
Owner: shinyak@chromium.org
Chatted offline with Shinya, and handed over to him.
Mergedinto: 729178
Status: Duplicate (was: Untriaged)
This is a partial revert of crbug.com/729178#c18, so let me merge the issue to crbug.com/729178.

Sign in to add a comment