I'm able to reproduce the issue after installing the OS with USB-10895.28.0/	69.0.3497.45 recovery image - Clapper.  I'm not able to see the login page after installing the recovery image.

Attached the recovery_logs file.

Deleted: recovery_logs_clapper.zip 62.4 KB

recovery_logs_clapper.zip 62.4 KB Download

Issue needs an owner and prioritized, if it is still occurring. Please verify latest version.

+dhaddock can you assist in routing this bug? 

Still reproduced from M68-10718.88.2(stable) => M69-10895.42.0(beta)

+aaboagye does this fall into your domain?

No, not really. What's the repro rate? How many devices run into it?

abodeti@ can you provide repro rate details? 

Reproduced 100% on M69 builds. 

aaboagye@ is this similar to crbug/878876 or crbug/874981?

No, it's not. issue 878876 is very board specific and I don't even think clapper has USB-C ports.

Issue 874981 sounds like it wasn't as a result of an AU plus it wasn't even the same major milestone...

Right clapper has no USB-C ports.

If this happens on clapper, does it happen on multiple clappers or just one particular unit?

Has this been seen on glimmer or any of the other similar BayTrail devices?

FWIW, we have not changed anything specific to these devices in some time, and the BayTrail systems are fairly homogeneous. 

I suspect this may be specific to this unit. Looking at Beta updates, most clapper have updated to M69 already -> 10895.40.0.0

https://omahadashwiki.corp.google.com/generate/?active_days=7&app=%7BBBCEB6C1-5567-09B4-1619-DAD125AC892D%7D&app_pretty=%7BBBCEB6C1-5567-09B4-1619-DAD125AC892D%7D&asterisk_date=2018-09-10&breakout=tag_and_version&completion_ratio=1.00000&count=unique&data_category=update_checks&display_name=7-day+Update+Checks+by+Channel&height=350&is_crx=false&limit=10&line_filter=.*beta.*&num_days=548&only_formula=true&sort_order=recent_high&start_day=*&style=gviz&width=800

Seems like this unit is not able to mount ext3

[    4.863516] EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
[    4.876617] EXT4-fs (dm-0): mounting ext2 file system using the ext4 subsystem
[    4.883179] EXT4-fs (dm-0): mounted filesystem without journal. Opts: (null)
[    4.891107] EXT4-fs (sda1): couldn't mount as ext3 due to feature incompatibilities
	
+Gwendal, for any thoughts on that 

Reproduced on two Clapper PVT devices but not reprod on Glimmer.

Note: AU from M68-10718.71.0 => M69-10895.56.0

Clapper HWID: CLAPPER D6A-D9K-L6A-A6R

Stable AU failed from M68-10718.88.2 => M69-10895.56.0

Is this only clapper or on others also?

Seen only on clapper but not reprod on other rambi devices(Glimmer,Banjo, chil, Enguarde and Expresso)

What is the latest status of the issue? This has not been updated in nearly two weeks and is marked RBS. Need to prioritize a fix.

Hi, can you please provide status.

There's a firmware update: 5216.199.7 -> 5216.199.174

    src/platform/depthcharge
      239330 - Depthcharge: Enable eMMC HS200 to enhance boot time.

Will that be related?

Thanks Hung-Te. That's certainly a possibility. That CL landed on 5216.199.9 so is it possible to keep the same R69 but updating the firmware to some version prior to 5216.199.9 (say, 5216.199.8)?

We're currently seeing this issue in the autotest labs. The majority of Clappers are in a failed state and unable to repair. They show "ChromeOS is missing or damaged" screen with recovery reason: No bootable kernel found on disk. Reinstalling from a USB stick does not help. 

We have a bug here:
https://bugs.chromium.org/p/chromium/issues/detail?id=897342

Able to reproduce this issue with M71 11151.18.0 build. 

Device broken after recovery installation.
Recovery completed successfully and Getting "ChromeOS is missing or damaged" screen after reboot.


Recovery logs are present at https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cros/871967/?debugUI=CLOUD

If the problem is due to HS200 change, then any OS version from 10764.0.0 onward (where it updates the FW to 5216.199.174 from 5216.199.7) will have this problem.

Issue 897342 has been merged into this issue.

<Bulk edit> Reminder M71 Stable is approaching. Please review this bug and assess if this is indeed a RBS. If not, please remove the RBS label. If so, please make sure any planned work will be tested in Beta and verified before the Stable date. Thanks

Moving to M71, window is closed for M-70

Peeps: All (I lied, most) the clapper devices in the lab are now hosed because of this bug.
b/117991928

cros-infra basically can't support clapper HWTest until this bug is fixed.

Also, a bug that is "assigned" without an owner ain't really assigned, ya know.

puneetster -> further triage

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/ece712e727b4aae6848cca527a2f99f24c1c82a4

commit ece712e727b4aae6848cca527a2f99f24c1c82a4
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Nov 15 00:47:48 2018

chromeos_config: Temporarily drop hwtests from clapper-release

BUG= chromium:871967 
BUG=b:117991928
TEST=unittests

Change-Id: Ib32e907c713502cd8198a58f741a2a6771ff61fc
Reviewed-on: https://chromium-review.googlesource.com/1336587
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/ece712e727b4aae6848cca527a2f99f24c1c82a4/config/chromeos_config.py
[modify] https://crrev.com/ece712e727b4aae6848cca527a2f99f24c1c82a4/config/config_dump.json
[modify] https://crrev.com/ece712e727b4aae6848cca527a2f99f24c1c82a4/config/chromeos_config_unittest.py

Probably the straightforward and immediate solution is to revert clapper to the previous firmware.

More generally, it would be good if ATL could test firmware update, which would prevent this issue in the future. See crbug.com/851063

We have reverted the CL, updated firmware version 5216.199.187. We'd need to give it a try.

regards to c#35, why the update failed?

The firmware was qual here https://cros-goldeneye.corp.google.com/chromeos/console/firmwareQualEditDetails?firmwareQualId=470, is there something missing during the qual?

based on c#26, looks like a drive incompatibility issue? 

#34 #49 i am wondering so. new emmc storage SKU on clapper failed units? 
if yes, there is an unmerge code review for HS200 timing tuning in depthcharge to achieve the best data sampling timing inside host logic during coreboot stage, https://chromium-review.googlesource.com/c/chromiumos/platform/depthcharge/+/277374

taking a look at the autotest lab devices, I see 1.7G in df -h for all devices (6 total) that are reachable and running the Google_Clapper.5216.199.174 RW AP fw (suggestive of the 4G SKU being affected and the 2G SKU not being affected)

RE: #50

https://chromium-review.googlesource.com/c/chromiumos/platform/depthcharge/+/277374 looks to be adding the support for auto-tuning but not necessarily enables it. To enable it we'd need to enable it (likely per device) with separate CLs?

one of the failed unit boots up after close and open the lid couple of times. 

feedback report: https://listnr.corp.google.com/product/208/report/85791023031

eventlogs: 
99 | 2018-11-16 11:08:53 | Wake Source | Power Button | 0
100 | 2018-11-16 11:08:54 | Chrome OS Recovery Mode | No bootable kernel found on disk | 0x5b
101 | 2018-11-16 11:30:07 | EC Event | Lid Closed
102 | 2018-11-16 11:30:07 | ACPI Enter | S5
103 | 2018-11-16 11:30:11 | System boot | 639
104 | 2018-11-16 11:30:11 | ACPI Wake | S5
105 | 2018-11-16 11:30:11 | Wake Source | Power Button | 0
106 | 2018-11-16 11:30:12 | Chrome OS Developer Mode
107 | 2018-11-16 11:30:25 | System boot | 640
108 | 2018-11-16 11:30:25 | SUS Power Fail
109 | 2018-11-16 11:30:25 | Wake Source | Power Button | 0
110 | 2018-11-16 11:30:26 | Chrome OS Recovery Mode | No bootable kernel found on disk | 0x5b
111 | 2018-11-16 11:30:48 | EC Event | Lid Closed
112 | 2018-11-16 11:30:48 | ACPI Enter | S5
113 | 2018-11-16 11:30:52 | System boot | 641
114 | 2018-11-16 11:30:52 | ACPI Wake | S5
115 | 2018-11-16 11:30:52 | Wake Source | Power Button | 0
116 | 2018-11-16 11:30:52 | EC Event | Lid Closed
117 | 2018-11-16 11:30:52 | ACPI Enter | S5
118 | 2018-11-16 11:30:58 | System boot | 642
119 | 2018-11-16 11:30:58 | Last post code in previous boot | 0x75 | Device Initialize
120 | 2018-11-16 11:30:58 | Extra info from previous boot | PCI | 00:02.0
121 | 2018-11-16 11:30:58 | ACPI Wake | S5
122 | 2018-11-16 11:30:58 | Wake Source | Power Button | 0
123 | 2018-11-16 11:31:00 | Chrome OS Developer Mode
124 | 2018-11-16 11:31:16 | EC Event | Lid Closed
125 | 2018-11-16 11:31:16 | ACPI Enter | S5
126 | 2018-11-16 11:31:18 | System boot | 643
127 | 2018-11-16 11:31:18 | ACPI Wake | S5
128 | 2018-11-16 11:31:18 | Wake Source | Power Button | 0
129 | 2018-11-16 11:31:19 | Chrome OS Developer Mode
130 | 2018-11-16 11:37:28 | Kernel Event | Clean Shutdown
131 | 2018-11-16 11:37:28 | System boot | 644
132 | 2018-11-16 11:37:28 | System Reset
133 | 2018-11-16 11:37:30 | Chrome OS Developer Mode
---------- END ----------

reg c#46.

5216.199.187 build bucket is empty(https://pantheon.corp.google.com/storage/browser/chromeos-releases/canary-channel/clapper/5216.199.187)

https://screenshot.googleplex.com/bfGwSiXU1gX.png

No such object: chromeos-releases/canary-channel/clapper/5216.199.187/ChromeOS-firmware-R34-5216.199.187-clapper.tar.bz2
You are getting this message because you have a Debug IP address. External clients only get "No such object: chromeos-releases/canary-channel/clapper/5216.199.187/ChromeOS-firmware-R34-5216.199.187-clapper.tar.bz2"

com.google.net.rpc3.RpcException: cloud.bigstore.ResponseCode.ErrorCode::OBJECT_NOT_FOUND: No such object: chromeos-releases/canary-channel/clapper/5216.199.187/ChromeOS-firmware-R34-5216.199.187-clapper.tar.bz2
	at com.google.cloud.bigstore.common.BigstoreException.toRpc3Exception(BigstoreException.java:143)
	at com.google.cloud.bigstore.common.BigstoreException.throwRpc3OnError(BigstoreException.java:317)
	at com.google.cloud.bigstore.api.json.handlers.objects.GetObject.handleRequestReceived(GetObject.java:296)
	at com.google.cloud.bigstore.api.json.handlers.objects.GetObject.handleRequestReceived(GetObject.java:71)
	at com.google.cloud.bigstore.api.json.framework.RequestHandler.handle(RequestHandler.java:294)
	at com.google.cloud.bigstore.api.json.ObjectsDelegator.get(ObjectsDelegator.java:81)
	at com.google.cloud.bigstore.isolation.RpcReceiver.lambda$processRequestAsync$4(RpcReceiver.java:203)
	at com.google.cloud.bigstore.isolation.AsyncExecutor.lambda$submit$0(AsyncExecutor.java:253)
	at com.google.common.context.ContextRunnable.runInContext(ContextRunnable.java:50)
	at com.google.common.context.ContextRunnable$1.run(ContextRunnable.java:39)
	at io.grpc.Context.run(Context.java:565)
	at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:166)
	at com.google.tracing.GenericContextCallback.runInInheritedContextNoUnref(GenericContextCallback.java:72)
	at com.google.tracing.GenericContextCallback.runInInheritedContext(GenericContextCallback.java:64)
	at com.google.common.context.ContextRunnable.run(ContextRunnable.java:36)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.cloud.bigstore.common.BigstoreException: No such object: chromeos-releases/canary-channel/clapper/5216.199.187/ChromeOS-firmware-R34-5216.199.187-clapper.tar.bz2
	at com.google.cloud.bigstore.common.BigstoreException.throwOnError(BigstoreException.java:304)
	at com.google.cloud.bigstore.common.BigstoreException.throwRpc3OnError(BigstoreException.java:315)
	... 16 more

With the re-work of firmware builder the artifacts are placed in different location. Anyway, it can be found here, gs://chromeos-image-archive/clapper-firmware/R34-5216.199.187/firmware_from_source.tar.bz2
Thanks.

Device boot up after updating the RW region with 5216.199.187 firmware on one of the failed unit.

Device also boot up when update RO and RW regions with 5216.199.187.

Firmware 5216.199.187 has passed qualification. Uprev is in progress.
Thanks.

Uprev to 5216.199.187 has been merged into ToT/R72, cherry-picked to R71 as https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-clapper-private/+/720331.

This bug requires manual review: We are only 12 days from stable.
Please contact the milestone owner if you have questions.
Owners: benmason@(Android), kariahda@(iOS), kbleicher@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Approved for M71 ChromeOS

Thanks!! The uprev CL has been merged to ToT/R72 and R71. I think it's too late to make it into R69/R70 therefore marking the bug fixed for now. Please re-open if in need.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot