Sorry, here are the files.

Deleted: crossystem.txt 5.0 KB

crossystem.txt 5.0 KB View Download

Deleted: Screenshot 2016-02-28 at 14.57.47.png 39.0 KB

	Screenshot 2016-02-28 at 14.57.47.png 39.0 KB View Download

Deleted: Screenshot 2016-02-28 at 14.57.36.png 36.0 KB

	Screenshot 2016-02-28 at 14.57.36.png 36.0 KB View Download

I have been experiencing the same thing on my Acer C710 (parrot).

I had something similar happen to this device in the past but it ( issue 428041  - Cannot remount root as rw) was fixed - for awhile at least.

I'm not sure when this started happening again but I know that ever since the new 'Enable Debugging features' was released it has never worked and I get the same bootloop as the OP. 

I've been reluctant to report it again but since it now has been, I thought I mention it here so we know it's not an isolated experience. It may be just happening to the 'parrot's though.

Part of my reluctance is due to the fact that I have custom partitions (ChrUbuntu & CROUTON) that I don't want to jeopardize.

Well, I hope that it will be fixed. I too have a crouton install, but
it causes the same problem without it.

+abodenha@, 

I think this would be the same as running make_dev_ssd.sh in VT2 ? 

see https://www.chromium.org/chromium-os/poking-around-your-chrome-os-device for more detail.

does that script need to be run in VT2?

Does this need to be restricted?

Adding people who might know more about this.

Yeah, I have no idea what's wrong. It should work just fine.

Here is the doc on this feature http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/debugging-features

The doc says:
 To enable debugging features, do the following:
1. Use the powerwash process or the recovery process to wipe your hard drive. After the device reboots and you sign in again, you’ll see the first screen.
2. Set the device to Developer Mode (see Developer Information for Chrome OS Devices). The system reboots, then displays the “OS Verification is OFF" screen.
3. Press Ctrl+D to dismiss this screen. The device reboots and shows the first screen.
4. Click the Enable debugging features link. A confirmation dialog displays.
5. Click Proceed. The system reboots and displays a dialog with password prompts.
6. [OPTIONAL] Set the new root password. 
Note: If you leave the root password blank, the password defaults to test0000.
7. Click Enable.  The screen displays messages indicating success or failure.
8. Click OK. You'll see the first screen again.
9. Follow the remaining prompts to configure your Chrome device

Following these steps exactly, the process fails at Step 5, when the device reboots but gets stuck in a bootloop instead of displaying a dialog with password prompts.

dzhioev@ can you dig into this?

Targeting to 50 for now. If we can find the fix quickly (and it's safe) we'll cherry pick to 49.

dzhioev@ Do we have a fix for this yet? Can you please update this issue with the latest?

dzhioev@ This bug has not updates since march 17th. Can you please update or reassign as needed. This is currently a stable blocker.

dzhioev@ Would you please update this issue if you are not the right owner?

dzhioev@ Please respond if you are not the right owner so we can get this assigned to the correct owner. This is a stable blocker and we really need to get this investigated soonest possible. abodenha@ thoughts on anyone else who could look into this issue?

I was not able to reproduce this on kip and rikku devices. Turning on debugging features works as expected on R48-R50. Probably this issue is specific for parrot devices.
I don't have any Parrot devices in the SF office. Can somebody from MTV confirm that the issue reproduces on Parrot?

dhaddock@/rookrishna@ can you please repro this on parrot devices and update this thread?

Yes I can repro this on M50 on parrot. It is stuck in a boot loop after step 5

dhaddock@ thanks for helping repro. Are there any logs you can share so Pasha can take a deeper look?

Nope, since I can't get it to boot to view any logs ;)

Why is this RVG? 

dzhioev@ Do you need any more information to investigate further? it looks like there is a repro on parrot devices although we have no logs.

dzhioev@ how should we proceed on this one?

dzhioev@ is out for a bit.

warx@ you wanted something higher priority.  Could you dig into this ASAP and see what you can find?

anyone can spare me a parrot device to investigate on? Chromestop doesn't have parrot devices available. I am in mtv43. dhaddock@, could you?

I'm WFH today but if you drop by my desk one of my testers will give you one :)

I tried manually enabling debugging features on parrot

1. # dbus-send --system --type=method_call --print-reply --dest=org.chromium.debugd /org/chromium/debugd org.chromium.debugd.RemoveRootfsVerification

method return sender=:1.6 -> dest=:1.48 reply_serial=2

2. # dbus-send --system --type=method_call --print-reply --dest=org.chromium.debugd /org/chromium/debugd org.chromium.debugd.EnableChromeDevFeatures string:"test0000"

Error org.chromium.debugd.error.DevFeatures: Rootfs verification must be removed first

So I guess rootfs is not removed even dbus methodcall returns success.

One more sentence: after I typed these two method calls, device goes to bootloop also.

Verification: tried QueryDevFeatures called on parrot after RemoveRootfsVerification call, the return of query is 4, which shows DEV_FEATURE_ROOTFS_VERIFICATION_REMOVED bit is not set (set status is 2).

while tested on cyan after RemoveRootfsVerification called, the Query returns 34, which indicates DEV_FEATURE_ROOTFS_VERIFICATION_REMOVED is set.

So RemoveRootfsVerification dbus method is not effective on parrot device, AFAIK.

cc David for inspiration of some debugd side debugging details

update: communicated with dpursell@ today.

Confirmed that it is the rootfs removal itself causing the bootloop, not enabling debugging features oobe, not debugd side,

by doing following steps:

1) Recover device
2) enable developer mode
3) run # /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification
4) reboot

parrot enters bootloop exactly. cced hungte@, if you can look at how make_dev_ssd.sh behaves on parrot?

The script worked fine and no one has changed it for a long time.

I don't have a parrot to debug on this, but I think you can start by checking:
1. is this problem only on parrot? will you see same issue on link, samus?
2. from what version (bi-sect) do you start seeing this error?

Then we can have more idea for what has go wrong.

Oh, one more thing, check if you can reproduce on devices that people NEVER run ChrUbuntu, CROUTON, or something like that. We've seen several wrong things those 3rd party distro done to break default settings and cause unexpected issues.

Wait, I saw the flag again in attachment of #c2.

dev_boot_signed_only   = 1

ChrUbuntu or CROUTON or something like that tends to set that flag and does not allow booting changed kernel.

I'd recommend when enabling debug features you should really turn off that option (i.e., crossystem dev_boot_signed_only=0)

My parrot is no longer testable for this issue (custom coreboot firmware w/ Ubuntu Linux), but when I originally installed Crouton, it says:

WARNING: Signed boot verification is disabled; enabling it for security.
You can disable it again using: sudo crossystem dev_boot_signed_only=0.

I was not fond of the idea of this script setting crossystem flags "for my security" so I immediately disabled it after running the Crouton installer. This does not seem to have affected the issue, however, I cannot do any tests on my Parrot any more.

@hungte, it seems parrot only, at least cyan doesn't have this issue. My version is 7834.70.0 stable-channel parrot. I can try some version older than 7647.84.0 to find the bi-sect. It takes some time though.

My device has no Crouton installed, and dev_boot_signed_only = 0, so no need to turn off.

I tried on 7390.69.0 (M46), still having this bootloop issue. I think this is not a Chrome UI bug, possibly need to reassign to system/firmware people.

Update: never mind the above comment. Fixed and tested on my parrot device. Bootloop is now not happening when enabling debugging features.

review link: https://chrome-internal-review.googlesource.com/#/c/256855/
https://chromium-review.googlesource.com/#/c/340890/

thanks to the some help from wad@

In a way of problem summary (assuming these fixes are enough!), warx@  tracked down the issue to removing rootfs verification. However, parrot is unique in that many parrot devices shipped with spinning disks. To offset the boot impact, they have an additional device mapper target, dm-bootcache, paired with verity. (essentially readahead on steroids)

The scripts for disabling verity also by necessity check if bootcache is disabled and if it is explicitly enabled, they fail but only late in the game.  On all devices without bootcache as a platform feature, everything works fine.  On platforms/boards where it _is_, there is a file that changes the default in the scripts from false to true.  When cros_make_image_bootable is called with no bootcache argument, it uses the default.  On parrot, it turned on bootcache even when verity was disabled.  This triggered the script error checking and it failed.  By adding a check of the verity flag state in the board specific defaults file, it can avoid forcing a change that is not support -- bootcache=true, verity=false.

Here's hoping that there aren't other dragons :)  Nice work, warx@!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/aaf6746108ab91c171ba28bc538880df0a8e62aa

commit aaf6746108ab91c171ba28bc538880df0a8e62aa
Author: warx <warx@chromium.org>
Date: Wed Apr 27 00:17:59 2016

Fixing bootloop when enabling debugging features on parrot 2

BUG= chromium:590606 

TEST=tested on my parrot device, bootloop is fixed.

Change-Id: I89243b25c74b5166e336703eed682d6139bbb9ae
Reviewed-on: https://chromium-review.googlesource.com/340890
Commit-Ready: Qiang Xu <warx@chromium.org>
Tested-by: Qiang Xu <warx@chromium.org>
Reviewed-by: Will Drewry <wad@chromium.org>

[modify] https://crrev.com/aaf6746108ab91c171ba28bc538880df0a8e62aa/overlay-parrot/scripts/board_specific_setup.sh

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-parrot-private/+/87dbdb07a8018139dbdf2cc0314b280754371b14

commit 87dbdb07a8018139dbdf2cc0314b280754371b14
Author: warx <warx@chromium.org>
Date: Wed Apr 27 00:15:45 2016

The commit in c44 and c45 seem to be changing the build script. Will that fix device trying to enable debugging features (which will disable rootfs_verification AFTER build_image completed)?

If none of the other scripts rely on any board-specific override, then the make_dev_ssd.sh looks like it should properly drop dm="..." and set root=PARTUUID=%U/PARTNROFF=1.  If that boot-loops, then it'd be good to see if anyone can grab any logs (boot loop then a recovery log dump) to check both kernel and early boot logs (assuming anything actually makes it to disk anywhere..). 

Maybe the first step would be dumping the kernel partition after make_dev_ssd.sh is called (before getting stuck in a boot loop!) and ensure that it looks like it should and the long-line dm="", or some other variation, isn't breaking the sed statements in that script unexpectedly!

hth -will

@#46, it really makes remove_rootfs_verification even though the commits are fixing build_image failure with --noenable_rootfs_verification flag on parrot.

[Automated comment] Request affecting a post-stable build (M50), manual review required.

IS the merge also needed in M51? 

Yes

Your change meets the bar and is auto-approved for M51 (branch: 2704)

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Approving merge to M50 7978.B branch.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

You guys really sound like you know what you're talking about, but for those of us who don't is there something I can do now to fix it or do I just have to wait? 

Yes, this fix will appear in an upcoming version of ChromeOS. What version are you on?

I know the question Commenter#60 asked was directed at Commenter#59 but, since I have a vested interest in the resolution, I'll respond if I may.

I have the 'PARROT PLOVERCREST A-C 4619' with an upgraded 128GB SSD.

I'm was on:
    Version 52.0.2743.19 dev (64-bit)
    Platform 8350.14.0 (Official Build) dev-channel parrot

but I switched to:
    Version 53.0.2758.0 canary (64-bit)
    Platform 8415.0.0 (Official Build) canary-channel parrot

In hopes of catching the fix earlier.

I'm not interested in all of the Debugging Features necessarily but I would like to remove rootfs verification (via the make_dev_ssd.sh script) so that I can add things to /etc/init, etc.

Hopefully the fix will land soon, if it hasn't already.

Thanx,
Denny

Still able to reproduce the issue in 8172.56.0, 51.0.2704.103 on Parrot. 
The device keeps rebooting infinitely after "OS Verification is Off" dev screen.

Re-opening. 

ok. Let me grab a parrot device and re-investigate into it.

The reason is cherry picking was failed so it was not merged to 8172.B. I will fix that.

now the fix is successfully merged into 8172 branch, should appear in the next build. @helenzhang, please let me know the updated status. Thanks!

Marking fixed since this should be in the branch.

Hi warx@, is the fix merged to M51-STABLE-5 8172.60.0 / 51.0.2704.103 ? I still see the issue in the build. Thanks! 

It should be merged beginning from 8172.58.0 according to this https://crosland.corp.google.com/log/8172.57.0..8172.60.0

With these two CLs, I checked that it can fix the bug. Feel free to reopen the bug if you still see it on 8172.60.0.

Hello,

I have a Parrot device (Acer C710-842G32 / PARROT NENE A-D 9512) and am still running into this issue.

My chromebook is running:

52.0.2743.116
8350.68.0
Google_Parrot.2685.37.0

warx@ can you take another look and see if this has regressed again?

Yes, sure. According to comment 67, I am worried that the fix is actually not enough, although it should not be based on my local build os image. I will take a look.

ping: anyone could please borrow me a parrot device to investigate on? Thanks

the fix landed earlier was only for build time code -- creating an initial image w/rootfs verification turned off.  it had no impact on runtime which means it wouldn't have fixed this bug.

It seems there is a delay posting the commit here. https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/a1001da56512fdcd3bd648a5a04da03ffea3e91b

The first build is 8895.0.0.

I've got an Acer C710 (parrot) currently using:

    Version 55.0.2882.0 canary (64-bit)
    Platform 8894.0.0 (Official Build) canary-channel parrot
    Firmware Google_Parrot.2685.37.0

Since I have custom partitions setup on this device, I want to be sure the fix is in place before I attempt to remove rootfs verification. I'm unable to tell exactly what version the fix will land on so I've been running the following after each update:

    grep bootcache /usr/share/vboot/bin/make_dev_ssd.sh
  
I'm assuming that once that command succeeds, I can safely remove rootfs verification.

What's interesting is that I've replaced the original 320GB hard drive with a 128GB SSD so the the bootcache feature isn't really an issue at all any more. I understand it would be impractical and maybe a little difficult to implement logic that detects an SSD and bypass enabling bootcache for this one device.

I've been anxiously awaiting a fix since I saw this:
    Comment 55 by keta...@chromium.org, May 9
    ⚐
    Labels: -Merge-Review-50 Merge-Approved-50
    Approving merge to M50 7978.B branch.

I would really like to know when the fix will be in place but until then I'll just keep using the above test. 

Comment#75 mentions: The first build is 8895.0.0.
So since I'm on 8835.0.0 I hoping I won't be waiting much longer. :)

Thanx for your attention,
Denny

there are no plans on making the bootcache logic dynamic.  the parrot boards are done and shipped, and the number of users who would be customizing things is pretty low.  the bootcache logic wouldn't really add overhead either in the SSD case.

Just wanted to report that the fix is in place and works on my Acer C710.

HWID:PARROT PLOVERCREST A-C 4619
OS Track: canary-channel
OS Board: parrot-signed-mp-v4keys

from: 8894.0.0 / 55.0.2882.0 (broken)
to  : 8955.0.0 / 56.0.2907.0 (fixed)

I had waited a bit before I updated to be sure that build 8895.0.0 had landed.

Thanx very much everyone for the fix.
Denny

This EXACT issue is back. Pixelbook running 69.0.3497.21

please file a new bug.  this one is ancient.