Logs can be found here: https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cr/733014/

Have you reproduced this problem on more than one stumpy device? Does it happen every time?

1) 2nd device I was working simultaneously had similar issues mentioned in the notes section and it loops back to the recover screen (insert USB stick to install)

2) Tried on 3rd device with builds 59.0.3071.33/9460.20.0 & 60.0.3112.31/9592.21.0 and both the builds works fine ( I am able to resume the device)

Tried these builds 60.0.3112.31/9592.21.0 and 59.0.3071.91/9460.60.0 stable on the device reported and see the issue.

+sontis found that if you press the on screen power button, the update screen have a better chance to show up.

Chromeos version: 9765.45.0 / 61.0.3163.67

Noticed "Your system is applying a critical update. Please do not turn it off" message after soft reboot.

Repo steps:
1. sign in to the device.
2. click on uber tray - > shut down button to power of the device.
3. wait for few secons (~60 seconds).
4. Power on the device.

Result:
Getting "Your system is applying a critical update. Please do not turn it off" message.
Need to power off and power on again to fix this issue.

Do we know where this critical update screen is coming from? I did not think a Stumpy had any peripherals that would require a special sort of update (like a PD controller or touch interface controller), moreover given the age and stability of this it is even less likely we should have an update for one. 

Googlers, I looked thru all the commits in the commit window for suspect CL. This one May 18th was all I could find. https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/505976 

logs(For comment # 8) are present at https://storage.cloud.google.com/chromiumos-test-logs/bugfiles/cr/733014/9765.45.0/debug-logs_20170830-175509.tgz?_ga=2.223747154.-859875196.1426784702

Note:Logs are taken after hard reboot.

If this is the same as the "Your system is applying a critical update. Please do not turn it off" issue, then I've confirmed that this is overwhelmingly affecting Samsung Chromebox Series 3 (Stumpy) devices running M60.

If it helps, here's a link to the user reports from the past month in Listnr: https://listnr.corp.google.com/product/208/issue/0d:69647b14:5321261?collapseIssueChart=true&dateRange=30

Probably a red herring: "Your system is applying a critical update" on Samsung Series 3. Was rather coincident after applying Android OS update to Samsung Chromebook Plus. I am GDG running both machines and noticed both events happened at end of release 59 on beta channel.     

The critical update screen should only show up when applying for EC/PD firmware. +duncane to confirm. 

 Since this is a chromebook, that screen should not appear. 

W/ regard to stumpy 'Critical update'

I believe its being displayed via this call stack

chromeos_startup -> firmware-boot-update

Which on legacy machines is:

https://chromium.git.corp.google.com/chromiumos/platform/firmware/+/7b044f15360fa0a1e91004ecd2272b2176d291d0/sbin/firmware-boot-update.3

And in the case of repro I'm seeing,

crossystem fwupdate_tries
6

So that allows firmware-boot-update.3 to continue and call

chromeos-boot-alert update_firmware

Now as to what makes fwupdate_tries non-zero I'm not sure.

Looks like '6' comes from updater3.sh,

  if [ "${FLAGS_update_ec}" = "${FLAGS_TRUE}" ]; then
    cros_set_startup_update_tries 6
    verbose_msg "Done (EC update will occur at Startup)"
  fi

Actually #17 isn't the path setting fwupdate_tries ... it looks like its,

chromeos-setgoodfirmware

  if [ "$(crossystem ecfw_act || true)" = "RO" ]; then
    # TODO(hungte) We need to handle one special case in future: user boot with
    # recovery button pressed and then never hard-reset device, and then EC will
    # always be in RO until we rewrite the EC firmware (i.e., next startup
    # update).
    echo "EC was boot by RO and may need an update/recovery."
    crossystem fwupdate_tries=6
  fi

As I see the log message w/in /var/log/update_engine.log

Looks like CL:508509 landed in 9568.0.0 

Why does $(crossystem ecfw_act) return "RO" on a Chromebox?

Seems to be the case for all Intel based chromeboxes that I surveyed.  For arm I see,

$ crossystem ecfw_act
Unable to open FDT property active-ec-firmware

Looks to be determined via,

src/platform/vboot_reference/host/arch/x86/lib/crossystem_arch.c

	} else if (!strcasecmp(name,"ecfw_act")) {
		if (ReadFileInt(ACPI_BINF_PATH ".2", &value) < 0)
			return NULL;
		switch(value) {
			case 0:
				return StrCopy(dest, "RO", size);
			case 1:
				return StrCopy(dest, "RW", size);
			default:
				return NULL;
		}
	}

crossystem ecfw_act return RO on Sridhar's chromebox.

crossystem ecfw_act return RO on my Stumpy device.

Device: Stumpy
Model Code: XE300M22-A02US

localhost / # crossystem 
arch                   = x86                            # Platform architecture
backup_nvram_request   = 1                              # Backup the nvram somewhere at the next boot. Cleared on success.
battery_cutoff_request = 0                              # Cut off battery and shutdown on next boot.
block_devmode          = 0                              # Block all use of developer mode
clear_tpm_owner_request = 1                              # Clear TPM owner on next boot
clear_tpm_owner_done   = 0                              # Clear TPM owner done
cros_debug             = 1                              # OS should allow debug features
dbg_reset              = 0                              # Debug reset mode request (writable)
debug_build            = 0                              # OS image built for debug features
dev_boot_usb           = 0                              # Enable developer mode boot from USB/SD (writable)
dev_boot_legacy        = 0                              # Enable developer mode boot Legacy OSes (writable)
dev_boot_signed_only   = 0                              # Enable developer mode boot only from official kernels (writable)
dev_default_boot       = disk                           # default boot from legacy or usb (writable)
devsw_boot             = 1                              # Developer switch position at boot
devsw_cur              = (error)                        # Developer switch current position
disable_dev_request    = 0                              # Disable virtual dev-mode on next boot
ecfw_act               = RO                             # Active EC firmware
fmap_base              = 0x00670000                     # Main firmware flashmap physical address
fwb_tries              = 0                              # Try firmware B count (writable)
fw_vboot2              = 0                              # 1 if firmware was selected by vboot2 or 0 otherwise
fwid                   = Google_Stumpy.2.102.0          # Active firmware ID
fwupdate_tries         = 6                              # Times to try OS firmware update (writable, inside kern_nv)
fw_tried               = A                              # Firmware tried this boot (vboot2)
fw_try_count           = 0                              # Number of times to try fw_try_next (writable)
fw_try_next            = A                              # Firmware to try next (vboot2,writable)
fw_result              = unknown                        # Firmware result this boot (vboot2,writable)
fw_prev_tried          = A                              # Firmware tried on previous boot (vboot2)
fw_prev_result         = unknown                        # Firmware result of previous boot (vboot2)
hwid                   = STUMPY YELLOW A-C 7520         # Hardware ID
inside_vm              = 0                              # Running in a VM?
kern_nv                = 0x00000006                     # Non-volatile field for kernel use
kernkey_vfy            = sig                            # Type of verification done on kernel key block
loc_idx                = 0                              # Localization index for firmware screens (writable)
mainfw_act             = A                              # Active main firmware
mainfw_type            = developer                      # Active main firmware type
nvram_cleared          = 0                              # Have NV settings been lost?  Write 0 to clear
oprom_needed           = 0                              # Should we load the VGA Option ROM at boot?
phase_enforcement      = (error)                        # Board should have full security settings applied
recovery_reason        = 0                              # Recovery mode reason for current boot
recovery_request       = 0                              # Recovery mode request (writable)
recovery_subcode       = 0                              # Recovery reason subcode (writable)
recoverysw_boot        = 0                              # Recovery switch position at boot
recoverysw_cur         = (error)                        # Recovery switch current position
recoverysw_ec_boot     = 0                              # Recovery switch position at EC boot
ro_fwid                = Google_Stumpy.2.102.0          # Read-only firmware ID
tpm_attack             = 0                              # TPM was interrupted since this flag was cleared
tpm_fwver              = 0x00010003                     # Firmware version stored in TPM
tpm_kernver            = 0x00030002                     # Kernel version stored in TPM
tpm_rebooted           = 0                              # TPM requesting repeated reboot (vboot2)
try_ro_sync            = 0                              # try read only software sync
tried_fwb              = 0                              # Tried firmware B before A this boot
vdat_flags             = 0x0000025e                     # Flags from VbSharedData
vdat_timers            = LFS=0,12320 LF=558274552,1094131816 LK=4827354932,10740005788 # Timer values from VbSharedData
wipeout_request        = 0                              # Firmware requested factory reset (wipeout)
wpsw_boot              = 1                              # Firmware write protect hardware switch position at boot
wpsw_cur               = (error)                        # Firmware write protect hardware switch current position

So my guess is we can't change the return value of ecfw_act in the short term for all x86 based chromeboxes.

Would appropriate workaround be to modify chromeos-firmwareupdate to:

  if [ "$(crossystem ecfw_act || true)" = "RO" ] && [ "$has_ec" = "1" ] ; then

and if so, whats the _right_ way to determine 'has_ec'?  

I've seen 
1. 'mosys | grep -i "ec info"' 
- seems a bit fragile relying on regexp

2. use 'ectool' presence
- won't account for lumpy other devices that don't have cros ec.

 

 Issue 735789  has been merged into this issue.

Add M60, should target for chromeboxes only.

how about "ectool version"  ?  

As:

https://bugs.chromium.org/p/chromium/issues/detail?id=735789

Has been merged with this, do we have any news of a fix for users of Samsung i5 Chromeboxes, please?
This page is far more technical than we were used to over on the other issue and hopefully is of more help to Google but as most of us on the other page are more "Regular" users, then it might be a little beyond some of us....It is for me.

I hope that you Guys can shed some light on this problem and I thank you for your efforts as it is causing a huge problem for those of us who are not as "computer savvy" as you are and I imagine that many people have discarded their Chromeboxes, thinking that they are broken.

Many thanks again for your efforts.

tbroch@ thanks for adding me. Please let me know when we have a fix.

+tbroch, I think checking for ectool should be the right fix.  
+hungte to confirm.

Re#29

As mentioned in #c24, checking ectool won't work because (1) there are devices without cros_ec, so they don't have ectool as well; (2) ectool is actually not available by default in release images; (3) some images may explicitly add ectool, even if it does not have ec.

Using mosys is probably still the right thing to do.

Can you try how the command runs on chromeboxes?

 mosys ec info -s name >/dev/null 2>&1; echo $?

I'd expect this returning 1 or other values. (On devices with EC this returns 0)

An alternative is to try "flashrom -p ec" but mosys should be more portable.

Made a CL https://chromium-review.googlesource.com/651611

Can someone having Chromebox try if this solves the problem?

@32.  Tested on both stumpy & non-chromebox and had the desired affect. Thanks Hung-te

Great. Let's get it merged.

ok, please add the merge labels once CL is landed in ToT 

I assume this is needed in all branches, right? M61 and M62

Note that there may not be another M60 but can opportunistically merge there too in case there is one  

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/5ba9d88d5b2a0958930c834d3c3d2ab87b3c4cb8

commit 5ba9d88d5b2a0958930c834d3c3d2ab87b3c4cb8
Author: Hung-Te Lin <hungte@chromium.org>
Date: Wed Sep 06 08:05:35 2017

sbin: Check if EC exists by running mosys instead of trusting crossystem.

The 'crossystem' will return ecfw_act=RO on x86 Chrome OS devices
without EC, which leads to kicking the boot time update.

Since changing the return value of all x86 Chromeboxes may have other
impact, for short term (and since this is only needed for device running
vboot1) we want to check first if system really has EC by running `mosys
ec info`.

BUG= chromium:733014 
TEST=None

Change-Id: Ic8f7970e761e42a2bce6f889c1cb0f4ce890ec92
Reviewed-on: https://chromium-review.googlesource.com/651611
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Tested-by: Todd Broch <tbroch@chromium.org>
Reviewed-by: Todd Broch <tbroch@chromium.org>

[modify] https://crrev.com/5ba9d88d5b2a0958930c834d3c3d2ab87b3c4cb8/sbin/chromeos-setgoodfirmware

We need this starting from M60.

This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), ketakid@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Approving merge to M61 and M62.

@comment #31
On my Stumpy device.
Command : mosys ec info -s name >/dev/null 2>&1; echo $?
Output: 38

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/d2257805ea820c9726cb5ebc66f28aa762f1d913

commit d2257805ea820c9726cb5ebc66f28aa762f1d913
Author: Hung-Te Lin <hungte@chromium.org>
Date: Wed Sep 06 19:24:39 2017

sbin: Check if EC exists by running mosys instead of trusting crossystem.

The 'crossystem' will return ecfw_act=RO on x86 Chrome OS devices
without EC, which leads to kicking the boot time update.

Since changing the return value of all x86 Chromeboxes may have other
impact, for short term (and since this is only needed for device running
vboot1) we want to check first if system really has EC by running `mosys
ec info`.

BUG= chromium:733014 
TEST=None

Change-Id: Ic8f7970e761e42a2bce6f889c1cb0f4ce890ec92
Reviewed-on: https://chromium-review.googlesource.com/651611
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Tested-by: Todd Broch <tbroch@chromium.org>
Reviewed-by: Todd Broch <tbroch@chromium.org>
(cherry picked from commit 5ba9d88d5b2a0958930c834d3c3d2ab87b3c4cb8)
Reviewed-on: https://chromium-review.googlesource.com/652127
Commit-Queue: Todd Broch <tbroch@chromium.org>

[modify] https://crrev.com/d2257805ea820c9726cb5ebc66f28aa762f1d913/sbin/chromeos-setgoodfirmware

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/45aedbee41b5d369df6164d83b897f19b3d38b1d

commit 45aedbee41b5d369df6164d83b897f19b3d38b1d
Author: Hung-Te Lin <hungte@chromium.org>
Date: Wed Sep 06 19:24:42 2017

sbin: Check if EC exists by running mosys instead of trusting crossystem.

The 'crossystem' will return ecfw_act=RO on x86 Chrome OS devices
without EC, which leads to kicking the boot time update.

Since changing the return value of all x86 Chromeboxes may have other
impact, for short term (and since this is only needed for device running
vboot1) we want to check first if system really has EC by running `mosys
ec info`.

BUG= chromium:733014 
TEST=None

Change-Id: Ic8f7970e761e42a2bce6f889c1cb0f4ce890ec92
Reviewed-on: https://chromium-review.googlesource.com/651611
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Tested-by: Todd Broch <tbroch@chromium.org>
Reviewed-by: Todd Broch <tbroch@chromium.org>
(cherry picked from commit 5ba9d88d5b2a0958930c834d3c3d2ab87b3c4cb8)
Reviewed-on: https://chromium-review.googlesource.com/652128
Commit-Queue: Todd Broch <tbroch@chromium.org>

[modify] https://crrev.com/45aedbee41b5d369df6164d83b897f19b3d38b1d/sbin/chromeos-setgoodfirmware

Don't believe the merge to M60 can happen given that its been stable for ~5wks.

Josafat can confirm though.

Discussed with Todd, since M61 is still few weeks away ... let's get as dev or beta out to make sure no side effects and then we can merge this to M60 in a week or so 

Marking as fixed to get verification going in 61/62

So sorry all, 

Can someone explain in short what the solution is moving forward as this has been marked as fixed but I don't quite understand what the solution is.

Many Thanks

@ Comment 45

I feel exactly the same, how can something be fixed if it is still faulty?  I too do not know what the solution is and do not exactly understand all the talk of "Merging".

Many thanks for any help.

Nothing happening, you still have restart yours chromebox after you start
it. Just press power button for 5-10s.
And wait with hope for next version.


Regards,
Linas

@46

This is the Chromium development forum, and not a support forum (note the @chromium email addresses). The Google Product Forum is the actual support but it does such an incredibly bad job of responding to or updating issues that frustrated people come here.

https://productforums.google.com/forum/#!topic/chromebook-central/qw0QcK30jBc

To translate what the developers on here are saying is they want to push a fix in the next scheduled update, but acknowledge the next scheduled update release is a few weeks away when a fix is needed now.

So what they're doing is releasing a fix in beta so they can do final tests and get the fix out ASAP. So it's not out yet but will likely be in the coming weeks.

The talk of "merging" in this particular thread instance is referring to merging all reports of the this issue into the same thread, so multiple developers aren't working on different fixes at the same time.

In any case, you'll have to keep using the restart workaround for the coming week at least.

Thanks for the clarifying Linas,

I am quite surprised that google have not rolled out an update yet - they are normally pretty quick at resolving these kind of issues

@ Comment 48

Thank you so much, that was most helpful to me and (I suspect) to many others.

Thank you for your time, it is much appreciated.

@ comment 48

Thank you for a clear description for the non technically minded that just love these boxes and would hate to see them become obselete  

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Verified on M62 dev build 9901.11.0
Not able to reproduce this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/firmware/+/6838fe0534de75de80bcc6eb1c1a929beb8a5fef

commit 6838fe0534de75de80bcc6eb1c1a929beb8a5fef
Author: Hung-Te Lin <hungte@chromium.org>
Date: Tue Sep 12 16:30:35 2017

sbin: Check if EC exists by running mosys instead of trusting crossystem.

The 'crossystem' will return ecfw_act=RO on x86 Chrome OS devices
without EC, which leads to kicking the boot time update.

Since changing the return value of all x86 Chromeboxes may have other
impact, for short term (and since this is only needed for device running
vboot1) we want to check first if system really has EC by running `mosys
ec info`.

BUG= chromium:733014 
TEST=None

Change-Id: Ic8f7970e761e42a2bce6f889c1cb0f4ce890ec92
Reviewed-on: https://chromium-review.googlesource.com/651611
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Tested-by: Todd Broch <tbroch@chromium.org>
Reviewed-by: Todd Broch <tbroch@chromium.org>
(cherry picked from commit 5ba9d88d5b2a0958930c834d3c3d2ab87b3c4cb8)
Reviewed-on: https://chromium-review.googlesource.com/652126
Commit-Queue: Todd Broch <tbroch@chromium.org>

[modify] https://crrev.com/6838fe0534de75de80bcc6eb1c1a929beb8a5fef/sbin/chromeos-setgoodfirmware

Verified on M61 beta build 9765.61.0
Not able to reproduce this issue.

Verified on M60 build 9592.94.0
Not able to reproduce this issue.

 Issue 761642  has been merged into this issue.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot