New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 628751 link

Starred by 3 users

Issue metadata

Status: Duplicate
Merged: issue 698336
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

Allocate GCE instances for try servers

Project Member Reported by josa...@chromium.org, Jul 15 2016

Issue description

Following up on a conversation with dgarret@ regarding current limitations on try servers and possibility of allocating GCE instances for these too.

Background: 
=========== 
As part of Chrome OS releases, there are times when we need to patch a build with release blocker fixes/reverts and then re-kick new builds from a mini branch (ref: https://sites.google.com/a/google.com/chromeos/for-team-members/chronos-download/pmo/cros-releasetasks/stabilize)

For branches we can use existing branch builders but for ToT we have to use try servers.

The number of build slaves on release tryservers (https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release) is not enough to kick off the number of board/targets we need hence it takes days to build them all plus causing a lot of disruption to anyone else using try servers 

The end result is that we are no longer able to patch ToT builds for Dev releases causing not having Dev releases on regular basis. In other words, we would only push dev when no blocking issues on nightly builds. 

Request
=======
Allocate more builders to try servers by provisioning new GCE instances for these

Alternatively, we could explore re using some of the branch/canary builders but this may require more work/logic




 


 
Owner: d...@chromium.org

Comment 2 by d...@chromium.org, Jul 15 2016

You should be able to use the release waterfall for this. Just manually enter "master" for ToT into the master builder branch?
Status: Available (was: Untriaged)
Do you mean one of the release branch builders? e.g M53 here: https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R53-8530.B

Note that that will miss some targets that are not in the branch e.g. new boards

Comment 4 by d...@chromium.org, Jul 18 2016

Yes. Missing targets is obviously a problem, and the degree to which that's a problem will determine the best course of action. If it's reasonable to use release for 90% of the targets and tryserver the missing boards, doubling the tryserver capacity is probably overkill.

If possible, utilizing the release waterfall instead of upping tryserver capacity is a much better option, since most of the release builders are already board-dedicated and heavily underutilized.

Comment 5 by d...@chromium.org, Sep 27 2016

Owner: ----

Comment 6 by josa...@google.com, Sep 28 2016

Was there a change in the master logic when using release waterfall?

It used to work few weeks back but it seems like Master is not kicking off slaves when entering an stabilize branch from ToT anymore .. 

Comment 7 by autumn@chromium.org, Oct 12 2016

Cc: nxia@chromium.org
Do you have an example of this? 
+nxia as she's been working in this area too

Comment 8 by dshi@chromium.org, Oct 25 2016

Owner: nxia@chromium.org
nxia, any comment? Or Don is the better owner?
There are two questions.

1) Why aren't slaves kicking off? I think I know the answer, but Ningning is the better owner.

2) Creating a pile of new trybot slaves. If we are certain we want to do that, I'm the better owner (unless someone else wants to learn how).

Comment 10 by nxia@chromium.org, Oct 25 2016

I don't know why slaves were not kicked off. Found an example and looks like slaves were triggered

https://uberchromegw.corp.google.com/i/chromiumos.release/builders/master-release%20release-R53-8530.B/builds/90

Comment 11 by nxia@chromium.org, Oct 25 2016

Owner: josa...@chromium.org
pass to josafat@ to confirm if slaves could be triggered by the master build. It looks to me slaves were successfully triggered in the example pasted above.
Owner: nxia@chromium.org
Slaves are kicked off properly when using main release branch but those are not when using stabilize branch 

Common scenario below:

a) Canary build is chosen as Dev release candidate
b) Testing finds a Dev release blocker 
c) TPM creates an stabilize branch from ToT release candidate
d) Dev blocker fix is merged into the stabilize branch 
e) New build (for all boards) needed from stabilize branch 

f) Release branch builders are used to kick off new build specifying the stabilize branch in the Master 
e) Slaves are not triggered by Master


ps. I know that there may be some targets that are not in the release branch builders and they won't get build but majority should be covered with all other slaves 

Comment 13 by nxia@chromium.org, Nov 15 2016

dgarrett@ is working on a pre-bootstrap script. I guess that will solve this branch issue? dgarrett@, can you confirm?
It will help solve some issues around this, but I think the key issue is how builds are launched.

I think this requires switching the branch builders to buildbucket.

Comment 15 by nxia@chromium.org, Nov 15 2016

Release branches rely on git-commit-mechanism to trigger slaves. buildbucket shouldn't be required to support this?
I THINK the git-commit-mechanism will be branch specific in this case, and that's why slaves aren't triggering if the branch is forced on the master.

Do we know exactly what the manifest-versions commits look like for both the normal branch release builds, and the ones with a forced branch on the master? comparing the two would confirm or deny my understanding.

Comment 17 by nxia@chromium.org, Dec 27 2016

Labels: -Pri-1 Pri-3

Comment 18 by nxia@chromium.org, Dec 27 2016

we can allocate more gces when the nesting feature is released by the GCE team.

Comment 19 by nxia@chromium.org, Mar 11 2017

Mergedinto: 698336
Status: Duplicate (was: Available)

Sign in to add a comment