New issue
Advanced search Search tips

Switching from normal mode to dev mode causes reboot loop on M73

Project Member Reported by matthewjoseph@chromium.org, Jan 11

Issue description

Boards:   eve, nocturne, nautilus, grunt
chromeOS: 11558.0.0

What steps will reproduce the problem?
(1) Recover device to 11558.0.0 and switch to normal mode
(2) Once in normal mode, switch back to dev mode
(3) After the dev mode transition process completes, the device will again reboot into the 'Preparing system for Developer Mode' screen

What is the expected result?
Device should reboot to the OS after successfully transitioning to dev mode

What happens instead?
Device gets stuck in a loop where it is continuously performing the dev mode transition

*Notes: 
 -- Once the device is in this state, you must recover the device again for it to be usable.
 -- Unable to reproduce issue on M71 or M72
 -- dev -> normal transition completes without issue
 
I am able to reproduce this on M73 builds as least as far back as 11538.0.0, which also exhibits this issue.
Labels: -ReleaseBlock-Dev ReleaseBlock-Beta
Cc: kmshelton@chromium.org hungte@chromium.org apronin@chromium.org
Cc: abod...@chromium.org dhadd...@chromium.org sdantul...@chromium.org
Any logs from the devices that experience this?
Do we know the first version when it started?
re#5

1) Couldn't get any logs from the device once it is in this state.  It is in a infinite dev-mode transition loop.  Only way to get the device into a usable state is to recover again.

2) Still trying to figure that out.  So far, I've repro'd on builds as far back as 11538.0.0
Cannot reproduce this issue on 11516.0.0, able to reproduce on 11522.0.0

https://crosland.corp.google.com/log/11516.0.0..11522.0.0
Cc: fletch...@chromium.org
There's not much in the diff. One of the more suspicious CLs is https://crrev.com/c/1387208. It deals with clobber, including some actions that happen when transitioning to dev mode.

E.g.
  # Tag that we're in developer mode otherwise we may get wiped again.
  if crossystem "devsw_boot?1" && ! crossystem "mainfw_act?recovery"; then
    touch /mnt/stateful_partition/.developer_mode
  fi

is turned into

  if (cros_system_->GetInt(CrosSystem::kDevSwitchBoot, &dev_mode_flag) &&
      dev_mode_flag == 1 &&
      cros_system_->GetString(CrosSystem::kMainFirmwareActive,
                              &firmware_name) &&
      firmware_name != "recovery") {
    return base::WriteFile(stateful_.Append(".developer_mode"), "", 0) == 0;
  }

Any chance that doesn't work as expected?

I don't see anything obvious at the 1st glance in that CL (aside from the fact that base::WriteFile creates .developer_mode with umask 0666, while normally we have 0644 iiuc). 

But still, just in case,
+fletcherw, can you please check the behavior of clobber-state during normal->dev transitions?
Cc: vapier@chromium.org
I'm seeing this too. I've attached logs (with the right FW switches, I can get the device to boot from USB).
shutdown_stateful_umount_failure
55.6 KB View Download
umount-encrypted.log
587 bytes View Download
clobber.log
499 bytes View Download
clobber-state.log
21.9 KB View Download
Owner: fletch...@chromium.org
Status: Started (was: Untriaged)
I had an issue with boot-looping when the file to request clobber (/mnt/stateful_partition/factory_install_reset) wasn't getting deleted but that happened after I'd converted more things to C++.

I'm taking a look now at what might be causing this.
Think I've solved this. I had a bug in my crossystem library causing it to incorrectly report a failed read, so the create file code wasn't running.

I was having trouble getting a system in verified mode flashed to M73 so this hasn't been tested on device but the logic lines up.

Fix is here: https://crrev.com/c/1409723

As a workaround for anyone hitting this bug, you should be able to boot from a USB image, mount the device's stateful partition to some folder (should look something like 'mount /dev/mmcblk0p1 [mount folder]', or maybe /dev/sda1, or /dev/nvme0n1p1), and then 'touch [mount folder]/.developer_mode' to stop the bootlooping.
Project Member

Comment 12 by bugdroid1@chromium.org, Jan 15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/9c9ba2369244657f65054570d11ff1d7c664b9e6

commit 9c9ba2369244657f65054570d11ff1d7c664b9e6
Author: Fletcher Woodruff <fletcherw@chromium.org>
Date: Tue Jan 15 10:54:54 2019

init: Fix error-handling logic in crossystem

CrosSystemImpl::GetString was incorrectly returning errors due
due to confusion about return values of underlying Crossystem calls.

Fix the code so that it correctly reports an error only when
::VbGetSystemPropertyString returns NULL.

BUG= chromium:921161 
TEST=none

Change-Id: I4d4f8d42024cb6e50854fbfcb49727cc97b56d16
Reviewed-on: https://chromium-review.googlesource.com/1409723
Commit-Ready: Fletcher Woodruff <fletcherw@chromium.org>
Tested-by: Fletcher Woodruff <fletcherw@chromium.org>
Reviewed-by: Fletcher Woodruff <fletcherw@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/9c9ba2369244657f65054570d11ff1d7c664b9e6/init/crossystem_impl.cc

Cc: abodeti@google.com aashuto...@chromium.org songsuk@chromium.org cindyb@chromium.org
 Issue 921800  has been merged into this issue.

Comment 14 by fletch...@chromium.org, Jan 17 (5 days ago)

Status: Verified (was: Started)
Seems fixed.

I recovered to verified mode on 73.0.3673.0 (Platform 11599), entered developer mode, and after clobber successfully booted to dev mode OOBE

Comment 15 by jamescook@chromium.org, Today (13 hours ago)

Cc: lprazdnik@chromium.org xiy...@chromium.org r...@chromium.org jamescook@chromium.org
 Issue 922086  has been merged into this issue.

Sign in to add a comment