New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 862006 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 861867
Owner: ----
Closed: Jul 11
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Test lab firmware update request: Kefka, Terra, Relm

Project Member Reported by kinaba@chromium.org, Jul 10

Issue description

Filing the request per https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/firmware-updates, but see below also.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[Version Info]

Board: Kefka
Desired firmware version: Google_Kefka.7287.379.0
Image build with the firmware: R68-10718.34.0 (beta)


Board: Terra
Desired firmware version: Google_Terra.7287.154.102
Image build with the firmware: R68-10718.34.0 (beta)


Board: Relm
Desired firmware version: Google_Relm.7287.370.0
Image build with the firmware: R68-10718.34.0 (beta)


[Reason]

The devices are very flaky when run on old firmware (see b/79527663#comment2)
These 268 failures in this week are all from DUTs with old firmware (Google_Kefka.7287.337.0 / Google_Terra.7287.154.61 / Google_Relm.7287.341.0):

https://stainless.corp.google.com/search?view=list&first_date=2018-07-04&last_date=2018-07-10&test=cheets_%5BCG%5DTS&board=%5Eterra%7Ckefka%7Crelm%24&status=GOOD&status=WARN&status=FAIL&status=ERROR&reason=Autoserv&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=true

Some of the DUTs in the lab are already running on the newer firmware I'm now requesting, and those failures are never seen on them.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Actually I wonder if I should rather file a bug instead of the update request.

The first R67-beta image (10575.22.0; Apr 29) had already built with those new firmware revision.
According to https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/firmware-updates,
the system should auto-update to these beta firmwares. But apparently this is not happening.
 
Cc: akes...@chromium.org pprabhu@chromium.org
+Infra Deputies, could you handle this?

I found  Bug 861867 . Probably the longer term solution will be to fix the other bug (if that one is affecting this one). But as a short-term solution for flaky tests, forced firmware update is appreciated.
Cc: jrbarnette@chromium.org
friendly ping, can someone knowledgable please mind taking a look?
The failures are almost certainly  bug 861867 , since the requested
firmware versions will already have been installed on any managed
DUT for over a week.

To confirm the diagnosis:  What devices/pools are seeing the problem?

Below is the list of devices extracted from the stainless link I posted.
I believe they are all from pool:cts-perbuild.

relm chromeos6-row2-rack15-host12
terra chromeos4-row8-rack7-host7
terra chromeos4-row6-rack8-host9
terra chromeos4-row6-rack8-host7
terra chromeos4-row6-rack8-host21
terra chromeos4-row6-rack8-host19
terra chromeos4-row6-rack8-host15
terra chromeos4-row6-rack8-host13
terra chromeos4-row6-rack8-host11
terra chromeos4-row6-rack7-host21
terra chromeos4-row6-rack7-host19
terra chromeos4-row6-rack7-host15
terra chromeos4-row6-rack7-host13
terra chromeos4-row6-rack7-host11
kefka chromeos2-row6-rack7-host9
kefka chromeos2-row6-rack7-host8
kefka chromeos2-row6-rack7-host7
kefka chromeos2-row6-rack7-host5
kefka chromeos2-row6-rack7-host4
kefka chromeos2-row6-rack7-host3
kefka chromeos2-row6-rack7-host13
kefka chromeos2-row6-rack7-host10
kefka chromeos2-row4-rack8-host12
kefka chromeos2-row4-rack8-host10
Mergedinto: 861867
Status: Duplicate (was: Untriaged)
> I believe they are all from pool:cts-perbuild.

    $ atest host list ${HOSTS[@]} | count_labels -p
         24 cts-perbuild

Yup.

cts-perbuild is affected by  bug 861867 , so that's what's going on.  :-(

Sign in to add a comment