New issue
Advanced search Search tips

Issue 851548 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Feature


Participants' hotlists:
SIE-infra-request


Sign in to add a comment

triage for branch builders failures

Project Member Reported by nsanders@chromium.org, Jun 11 2018

Issue description

Since Chrome OS requires branches to be long lived and continuously working, in order to:
* support deploying security fixes in firmware quickly
* support new RO for updated AVL components.
* support RMA shim updates throughout the life of the project
* support follower devices based on the SoC family's validated factory branch. 

Branches have been breaking frequently, which is becoming a problem for follower devices and firmware builds.

We should continuously (1/week ?) build these branches, even if no changes have been submitted, to ensure they continue functioning or have a record when the relevant infra broke.


I think the easy fix here would just be to add a scheduler to kick all builders once a week or so.
 
Labels: -Type-Bug Type-Task
We've already implemented this: https://uberchromegw.corp.google.com/i/chromeos.branch/builders . You can see when the next branch build is scheduled on the right-handle column. Currently the sheriffs are ignoring branch builds but I think that it makes sense to ask them to investigate their failures. Do you want to update the sheriff instructions? https://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium-os currently doesn't mench the branch tree at all.

I don't think it makes sense for sheriffs to investigate the failures, since failures found by these runs would be by definition infra issues. Is there an equivalent infra-deputy page?
Labels: -Type-Task Type-Feature
Owner: ----
Status: Available (was: Untriaged)
Summary: page for branch builders failures (was: regular builds for branch builders)
There's no current capability to do pages: we're building that as a part of CI 2020 with a current anticipated delivery date of Q4.
No, like a web page document =) 
I don't think anyone needs to be paged for weekly background branch builder failures.

It's great that these builders are running already though.
Summary: triage for branch builders failures (was: page for branch builders failures)
Biggest issue we face with that is that we have no way of knowing which branches are no longer relevant to the business. If we had a way of recording that or finding an owner of the branch when it fails, that would make this more possible.
That's a good point. Would be worthwhile to purge extra branches at some point.

However, most firmware or factory branches that match shipped, non-EOL products are still needed. They don't have owners, or maintainers that could cover for infra issues, other than the infra team.

Of course anyone checking things into the branch in situations where it's generating releases is responsible for not breaking it, but otherwise it's expected to continue working for the life of the product so long as no changes have been checked in.

This was escalated as an SIE Infra request. However, I don't see an action item here.
Labels: -Infra-ChromeOS -Infra
Removing Infra labels, this is not an infra request. Please re-categorize as appropriate.

Sign in to add a comment