chell canary builds roll back at installation
Reported by
jrbarnette@chromium.org,
Apr 2 2017
|
||||
Issue description
Chrome Version: N/A
OS: Since R59-9413.0.0
What steps will reproduce the problem?
(1) Install the build via AU and boot in the test lab (probably at your desk, too).
What is the expected result?
Build boots.
What happens instead?
The build crashes repeatedly, and rolls back to the previous build
Please use labels and text to provide additional information.
The problem is first seen on this build:
https://uberchromegw.corp.google.com/i/chromeos/builders/chell-release/builds/964
HWTest [sanity] fails with this error message:
provision: FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos4-row12-rack10-host1: RootfsUpdateError: Build chell-release/R59-9413.0.0 failed to boot on chromeos4-row12-rack10-host1; system rolled back to previous build, new report
More recent builds are reporting something like more this:
provision: FAIL: The host has wrong cros-version label., completed successfully
Digging through the logs of one such failure turned up the rollback
message under autoupdate_logs, but for some reason provisioning code
is reporting it differently.
Assigning to a randomly selected Sheriff for triage.
,
Apr 3 2017
,
Apr 4 2017
,
Apr 4 2017
Based on the stack trace in ramoops, seems like it's failing on a call to skl_get_module_params: [ 9.869422] Call Trace: [ 9.869445] [<ffffffffc04ef26a>] skl_get_module_params+0x18c2/0x1940 [snd_soc_skl] [ 9.869473] [<ffffffff83d89fdb>] dapm_seq_check_event.isra.30+0x150/0x1c1 [ 9.869497] [<ffffffff83d8ba84>] dapm_seq_run_coalesced+0x1b2/0x1e8 [ 9.869519] [<ffffffff83d8bb94>] dapm_seq_run+0xda/0x38c [ 9.869540] [<ffffffff83896591>] ? finish_wait+0x64/0x64 [ 9.869560] [<ffffffff83d8c3b2>] dapm_power_widgets+0x56c/0x7fd [ 9.869584] [<ffffffff83d8ca2c>] soc_dapm_mixer_update_power+0xc2/0xd1 [ 9.869608] [<ffffffff83d8cbfe>] snd_soc_dapm_put_volsw+0x14d/0x1be [ 9.869632] [<ffffffff83a22e28>] ? avc_has_extended_perms+0x28b/0x37f [ 9.869657] [<ffffffff83d6f0e5>] snd_ctl_elem_write+0xaf/0x119 [ 9.869681] [<ffffffff83948f2b>] ? __kmalloc_track_caller+0x78/0x135 [ 9.869705] [<ffffffff83d7022d>] snd_ctl_ioctl+0x35e/0x51e [ 9.869727] [<ffffffff839615d8>] do_vfs_ioctl+0x399/0x460 [ 9.869745] [<ffffffff839616f9>] SyS_ioctl+0x5a/0x7f [ 9.869769] [<ffffffff83e9c0dc>] system_call_fastpath+0x1c/0x21 [ 9.869786] Code: 48 89 45 b8 48 39 c3 75 0d 49 8b 45 18 48 8d 58 f8 e9 c3 00 00 00 49 8b 07 4c 8b 48 30 48 83 eb 10 48 8d 43 10 48 39 45 b8 74 6c <4c> 39 4b 08 75 17 48 8b 33 48 8b 7d c8 4c 89 ca 4c 89 4d b0 e8 [ 9.870005] RIP [<ffffffffc04edfcb>] skl_get_module_params+0x623/0x1940 [snd_soc_skl] We've narrowed the bad CL down to one of these four from R59-9413.0.0: https://chromium.googlesource.com/chromiumos/third_party/kernel/+log/c0b917cbd9afc1c8850d0d3c2f30672b5737ab96..55d103f4e8cd5aef897523a29c9060ef535ab085?n=10000 We think that we may be this one, but can't be sure: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ecbda3f1c478be14905f016bd619f1f9ebf6b6ef Will launch some tryjobs to test it out.
,
Apr 4 2017
from crosoncall: philipchen We've kicked off two trybot jobs here: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release 11:46philipchen One job is to revert the most suspicious CL https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ecbda3f1c478be14905f016bd619f1f9ebf6b6ef 11:47philipchen The other job is to revert all of the 4 CLs here: https://chromium.googlesource.com/chromiumos/third_party/kernel/+log/c0b917cbd9afc1c8850d0d3c2f30672b5737ab96..55d103f4e8cd5aef897523a29c9060ef535ab085 11:48philipchen The job queue is pretty long now, so we'll try to build the test image locally too.
,
Apr 4 2017
Reverted 3 CLs: https://chromium-review.googlesource.com/467888 https://chromium-review.googlesource.com/467889 https://chromium-review.googlesource.com/467890 Tested without these three CLs on a chell in the lab and did not see the rollback behavior.
,
Apr 4 2017
Sorry about the break, let me check what happened. Thanks for the reverts.
,
Apr 5 2017
Marking this as fixed as the last chell-release build was green. https://uberchromegw.corp.google.com/i/chromeos/builders/chell-release/builds/984 |
||||
►
Sign in to add a comment |
||||
Comment 1 by shchen@google.com
, Apr 3 201744.8 KB
44.8 KB View Download