(ChromeOS WiFi) Blacktip device bricked after autotest scheduled tests on this device with R65-10323.89.0. |
||||
Issue description
Blacktip fw is qualified with version Google_Coral.10068.52.0
R65-10323.89.0 has fw version Google_Coral.10068.39.0 which is not qualified.
When the dut is provisioned with this release, old fws were updated into the release and it bricked the device. Upon verifying minicom data on ec_console, it reported as below:
minicom -D /dev/pts/34
Welcome to minicom 2.7.1
OPTIONS: I18n
Compiled on Aug 13 2017, 15:25:34.
Port /dev/pts/34, 15:28:29
v[130.091784 battery not found]
[130.100642 battery not found]
ers[130.591927 battery not found]
[130.600782 battery not found]
ion[131.092941 battery not found]
[131.102684 battery not found]
Chip: Nuvoton NPCX586G A.05
Board: 3
RO: coral_v1.1.7267-b7254f389
RW: coral_v1.1.7269-3fc31d6e2
Build: coral_v1.1.7267-b7254f389
2018-01-05 20:09:28 @build291-m2.golo.chromium.org
> [131.592248 battery not found]
[131.601105 battery not found]
[132.092439 battery not found]
[132.101295 battery not found]
Able to recover the dut after installing qualified fw versions which is Google_Coral.10068.57.0
Rootcause:
Auto test should not schedule tests on devices for which its version is not qualified.
This could be a problem with uni builds and could brick devices in future.
Is there a way we can stop scheduling tests and implement checks in autotest for this kind of issues?
Sample job from autotest where the device bricked after running the test with R65 release:
https://ubercautotest.corp.google.com/afe/#tab_id=view_job&object_id=206740146
,
Jun 14 2018
Infra team: how do we prevent this from happening in the future? Some of our wifi test suites are setup to run on <=tot-1 and in this case looks like there was a new M65 build on Jun 8th which had old fw hence bricking the device.
,
Jun 14 2018
Stainless link for reference, this is for 2 blacktip hosts where chromeos15-row4-rack12-host4 was the DVT version which was swapped with a PVT version in chromeos15-row1-rack2-host2 on May 10th. https://stainless.corp.google.com/search?view=matrix&row=queued_date&col=build&first_date=2018-04-18&last_date=2018-06-14&model=blacktip&hostname=chromeos15-row1-rack2-host2%7Cchromeos15-row4-rack12-host4&exclude_cts=true&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false
,
Sep 12
Is the problem that there was a build with a bad fw version? Or are you asking to be able to dynamically block build images for a DUT? The latter won't happen soon as we're moving away from Autotest scheduling. It would be implemented in the new Skylab scheduling system and in any case would need a careful design for such a new feature. If this was a perfectly good build that bricked devices, I think it would be easier to change how the test suites are set up to not run such old(?) builds.
,
Sep 14
|
||||
►
Sign in to add a comment |
||||
Comment 1 by dsunk...@chromium.org
, Jun 14 2018