New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 141 users

Comments by non-members will not trigger notification emails to users who starred this issue.
Status: Fixed
Owner:
Closed: Oct 2
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug

Blocked on: View detail
issue 766347
issue 773474
issue 741019
issue 762979

Blocking:
issue 710290
issue 731943
issue 763887


Show other hotlists

Hotlists containing this issue:
Hotlist-1


Sign in to add a comment
device losing enrollment - Lenovo devices
Project Member Reported by marcore@chromium.org, Aug 29 Back to list
I'm opening a new bug because the HW is different from https://bugs.chromium.org/p/chromium/issues/detail?id=738895

Chrome Version: 59.0.3071.134, 60.0.3112.112
OS: ChromeOS
HW: Lenovo N22 REKS C25-E4E-U5A-Q6E

- How many devices are affected?
So far they have reported more than 50

What steps will reproduce the problem?
(1) enroll device
(2) Leave Chromebook charging on a cart during the night
(3) Turn the device on

What is the expected result?
device keeps enrollment
What happens instead?
Devices goes straight to enrollment screen (looks like device was wiped)

debug logs: https://drive.google.com/open?id=0B01ZVp8vDQocckJ1VHFzVDg5RDA

in the logs I see:
ui/ui.20170822-192054:[1016:1016:0822/192059.441158:ERROR:device_cloud_policy_store_chromeos.cc(238)] Device policy read on enrolled device yields no DM token! Status: 4.
as written in https://bugs.chromium.org/p/chromium/issues/detail?id=738895#c6
Value 4 means:
STORE_INVALID_POLICY,    // Invalid settings blob (proto parse failed).

messages.1
2017-08-22T08:18:32.624821-04:00 INFO kernel: [    0.992990] mmc0: BKOPS_EN bit is not set
2017-08-22T08:18:32.624822-04:00 INFO kernel: [    1.009039] mmc0: new HS200 MMC card at address 0001
2017-08-22T08:18:32.624824-04:00 INFO kernel: [    1.009657] mmcblk0: mmc0:0001 AGND3R 14.5 GiB 
2017-08-22T08:18:32.624826-04:00 INFO kernel: [    1.009751] mmcblk0boot0: mmc0:0001 AGND3R partition 1 4.00 MiB
2017-08-22T08:18:32.624833-04:00 INFO kernel: [    1.009832] mmcblk0boot1: mmc0:0001 AGND3R partition 2 4.00 MiB
2017-08-22T08:18:32.624836-04:00 INFO kernel: [    1.009921] mmcblk0rpmb: mmc0:0001 AGND3R partition 3 4.00 MiB
2017-08-22T08:18:32.624838-04:00 INFO kernel: [    1.013375]  mmcblk0: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12
...
2017-08-22T08:18:32.625772-04:00 CRIT kernel: [    2.564271] EXT4-fs error (device mmcblk0p1): ext4_mb_generate_buddy:757: group 6, block bitmap and bg descriptor inconsistent: 24036 vs 24090 free clusters
2017-08-22T08:18:32.625774-04:00 CRIT kernel: [    2.564501] EXT4-fs error (device mmcblk0p1): ext4_mb_generate_buddy:757: group 0, block bitmap and bg descriptor inconsistent: 18985 vs 18984 free clusters
...
2017-08-22T08:18:37.249574-04:00 WARNING cryptohomed[1041]: Could not load the device policy file.
2017-08-22T08:18:37.872329-04:00 ERR session_manager[959]: [ERROR:session_manager_impl.cc(576)] Failed to retrieve policy data.
2017-08-22T08:18:37.953048-04:00 CRIT kernel: [    7.985788] EXT4-fs error (device mmcblk0p1): ext4_mb_generate_buddy:757: group 10, block bitmap and bg descriptor inconsistent: 25580 vs 25579 free clusters
2017-08-22T08:18:37.953078-04:00 CRIT kernel: [    7.986365] EXT4-fs error (device mmcblk0p1): ext4_mb_generate_buddy:757: group 14, block bitmap and bg descriptor inconsistent: 23675 vs 23780 free clusters
2017-08-22T08:18:37.873942-04:00 ERR session_manager[959]: message repeated 2 times: [ [ERROR:session_manager_impl.cc(576)] Failed to retrieve policy data.]
2017-08-22T08:18:38.212705-04:00 INFO cryptohomed[1041]: Attestation: Valid attestation data exists.
2017-08-22T08:18:38.239117-04:00 INFO cryptohomed[1041]: Cannot read boot lockbox files.

 
Owner: josa...@chromium.org
Josafat, could you help us triage this issue ?
Comment 2 Deleted
Support have referenced me to this page to work with you to get this issue resolved. Please let me know If I can do anything on this end. 
The issue is affecting many devices everyday at our schools(reports from teachers) but I haven't been able to reproduce the issue. 
I am having the same exact problem, this is a huge issue as I have purchased close to 3000 Lenovo N22 models in our school district!
Labels: -Pri-1 Pri-0
Same issue here but on Lenovo N23 hardware. Multiple users affected in our district as well. We deployed over 500 in a partial refresh this year and have had affected students coming by daily. These are primarily 1:1 take home devices. Device appears wiped and does not auto-re-enroll though that setting is enforced. But students are unable to connect to domain provisioned PSK wireless networks so they stop by the support desk for help. 
Cc: choonc@google.com katierh@chromium.org
Labels: -M-59 -M-60 M-61 ReleaseBlock-Stable
Owner: gwendal@chromium.org
Cc: mnissler@chromium.org atwilson@chromium.org abodenha@chromium.org
We need more help on this bug
Summary: device losing enrollment - Lenovo devices (was: device losing enrollment)
I assume all customers reporting this on this bug are using Lenovo. If not, please list your hardware here and create a support ticket so that we can work with you to get logs.
It was reported a while back [I missed it]
https://bugs.chromium.org/p/chromium/issues/detail?id=514700#c34

The first critical error message is in messages.1 at 2017-08-22T08:18:32.625772-04:00.
Given that file is corrupted, I am guessing there was a FS corruption earlier.

Let's collect more info from other device.



Cc: igorcov@chromium.org dskaram@chromium.org cyrusm@chromium.org
+igor/cyrus/dskaram

As noted above, looks like FS corruption which is causing us to lose our policy data. While we could probably find a way to preserve "enrollment" by storing a DMToken in NVRAM or something, that wouldn't be sufficient to keep the device functioning since we also need things like device-wide network config, etc.

We're seeing this throughout our school district as well on acer c740's. We've got about 2500 deployed devices and I've had at least 10-15 with the issue so far.
Matt: Please file an enterprise support case and mention this bug. Please send device logs from chrome://net-internals#chromeos -> "store debug logs" and upload it to the case. We'd like to review logs from all similar incidents to understand patterns.
Lenovo N23 chromebooks have appeared to spontaneously un-enroll. The student opens the chromebook and the chromebook is asking for the wi-fi password so they bring it to me. Once I enter the password, the chromebook goes to the accept & continue screen. After that, it still says it's enrolled, but it actually isn't because the log in page has the @xxxx.org already filled in. I have to deprovision it and wipe/enroll it again. 
More files here : https://drive.google.com/corp/drive/folders/0Bx12NVyIouI0d3lmSmJGZ0xWWHc
Cc: maxkirsch@chromium.org
gwendal@ Do we have a resolution on this one? This is currently a stable blocker.
Cc: frankhu@chromium.org
Gwendal, here is another instance that might give us some hints. In CFM, we managed extract log from one of the units which we were suspecting of unenrollment. Fortunately the enrollment is still there, just the CFM mode failed to load, and we traced it down to ext4-fs error

https://b.corp.google.com/issues/65178298

For Mountain Stuart, I believe the error happens here:

2017-08-22T10:03:18.541045-07:00 CRIT kernel: [    7.554737] EXT4-fs error (device sda1): ext4_iget:4044: inode #129809: comm MountThread: bad extra_isize (1157 != 256)
2017-08-22T10:03:18.541065-07:00 CRIT kernel: [    7.554893] EXT4-fs error (device sda1): ext4_iget:4044: inode #129810: comm MountThread: bad extra_isize (1161 != 256)
2017-08-22T10:03:18.676079-07:00 INFO kernel: [    7.689904] tpm_tis tpm_tis: command 0x65 (size 18) returned code 0x0
2017-08-22T10:03:18.718079-07:00 INFO kernel: [    7.732617] tpm_tis tpm_tis: command 0x21 (size 14) returned code 0x0
2017-08-22T10:03:18.737066-07:00 CRIT kernel: [    7.751491] EXT4-fs error (device sda1): ext4_iget:4044: inode #129809: comm MountThread: bad extra_isize (1157 != 256)
2017-08-22T10:03:18.737089-07:00 CRIT kernel: [    7.751661] EXT4-fs error (device sda1): ext4_iget:4044: inode #129809: comm MountThread: bad extra_isize (1157 != 256)
2017-08-22T10:03:18.737094-07:00 CRIT kernel: [    7.751842] EXT4-fs error (device sda1): ext4_iget:4044: inode #129809: comm MountThread: bad extra_isize (1157 != 256)
2017-08-22T10:03:18.737239-07:00 ERR cryptohomed[1700]: Couldn't create vault path: /home/.shadow/018d3d7b52d14446467603feed3ae1ee9f421c07/vault
2017-08-22T10:03:18.737294-07:00 ERR cryptohomed[1700]: Error creating cryptohome.
2017-08-22T10:03:18.737536-07:00 WARNING cryptohomed[1700]: PKCS#11 initialization requested but cryptohome is not mounted.

The device rebooted and the cryptohome couldn't be mounted anymore.
Cc: puneetster@chromium.org snanda@chromium.org adlr@chromium.org
Received 4 other sample log files.
I corrected all info and save it below drive.
https://drive.google.com/open?id=0B01YYztUbOCuZDV6WHRnYUdxc0E

Model:	Chromebook 11 Model 3180
Google Chrome Version:60.0.3112.112	
Platform Version:9592.85.0 (Official Build) stable-channel kefka	
Firmware Version:Google_Kefka.7287.337.0						

Similar error message that I found.
2017-08-24T10:49:41.555047-07:00 CRIT kernel: [    3.584656] EXT4-fs error

I also reported a similar case on crbug.com/759946 few days ago, however it is not Lenovo device.

ChromeOS version: 59.0.3071.134 
ChromeOS device model: lars (Chromebook 14 for work (CP5-471)) 

Labels: M-60
Status: Available
Similar symptoms were observed in older builds as reported in issue 514700

gwendal, any additional info needed to evaluate this case?
Comment 23 Deleted
Some of us less knowledgeable on the script behind the Chrome OS have a thread on here too.  I have been working with Google support for 3 weeks now over this issue.  It is just Lenovo for us but this thread has many other devices affected.

https://productforums.google.com/forum/?utm_medium=email&utm_source=footer#!msg/google-education/mFSODuzRdgE/CmLYd55AAgAJ

Here's some more log files from a Lenovo N22. 

Averaging 10 per school each day.  

debug-logs_20170831-100208.tgz
4.8 MB Download
CTL J5, happening a percentage of our fleet per day at our K-8.
Attached is post-re-enrollment.
debug-logs_20170831-122211.tgz
3.8 MB Download
Adding another example here.
Affected devices - Acer Chromebook 15 (CBS-571) running Chrome 60.0.3112.80.
debug logs and additional information: 
https://drive.google.com/open?id=0By9xEkAUmm_cQVVJbWd3MXhxbTg
Just did another four reenrollments.  Something to note is the ones affected so far are all CTL J5 from only two carts.  We have other groups of J4 and J5 books that are not affected.  All, affected and unaffected, share the same wifi network and there is really no difference between the groups of affected hardware and the unaffected hardware as far as I note.  NOthing fdifferent about how they are used or by whom.

The pool of serial numbers (two carts' worth) that keep experiencing unenrollment are:

JM01N8E0H00YA6850177
JM01N8E0H00YA6850135
JM01N8E0H00YA6850221
JM01N8E0H00YA6850336
JM01N8E0H00YA6850342
JM01N8E0H00YA6850133
JM01N8E0H00YA6850267
JM01N8E0H00YA67S0144
JM01N8E0H00YA6850140
JM01N8E0H00YA6850234
JM01N8E0H00YA6850337
JM01N8E0H00YA6850331
JM01N8E0H00YA6850324
JM01N8E0H00YA6850247
JM01N8E0H00YA6850332
JM01N8E0H00YA6850192
JM01N8E0H00YA67S0120
JM01N8E0H00YA6850085
JM01N8E0H00YA6850151
JM01N8E0H00YA6850193
JM01N8E0H00YA6830428
JM01N8E0H00YA6850092
JM01N8E0H00YA6850216
JM01N8E0H00YA6850273
JM01N8E0H00YA6850233
JM01N8E0H00YA6850401
JM01N8E0H00YA6850349
JM01N8E0H00YA6850347
JM01N8E0H00YA68K0474
JM01N8E0H00YA68K0045
JM01N8E0H00YA68K0037
JM01N8E0H00YA68K0040
JM01N8E0H00YA68K0043

JM01N8E0H00YA6AU1033
JM01N8E0H00YA6AU0983
JM01N8E0H00YA6AU0925
JM01N8E0H00YA6AU1031
JM01N8E0H00YA6AU0985
JM01N8E0H00YA6AU1030
JM01N8E0H00YA6AU0926
JM01N8E0H00YA6AU0011
JM01N8E0H00YA6AU0851
JM01N8E0H00YA6AU1048
JM01N8E0H00YA6AU1051
JM01N8E0H00YA6AU1066
JM01N8E0H00YA6AU0942
JM01N8E0H00YA6AU0880
JM01N8E0H00YA6AU1070
JM01N8E0H00YA6AU1063
JM01N8E0H00YA6AU0940
JM01N8E0H00YA6AU0087
JM01N8E0H00YA6AU1062
JM01N8E0H00YA6AU0052
JM01N8E0H00YA6AU0146
JM01N8E0H00YA6AU0162
JM01N8E0H00YA6AU0421
JM01N8E0H00YA6AU1039
JM01N8E0H00YA6AU0558
JM01N8E0H00YA6AU0688
JM01N8E0H00YA6AU0729
JM01N8E0H00YA6AU1008
JM01N8E0H00YA6AU0591
JM01N8E0H00YA6AU0616
JM01N8E0H00YA6AU0051
JM01N8E0H00YA6AU0927
Forgot to add that after the unenrolled welcome screen, and after joining the wifi network, a minority of re-enrollees go right to enrollment opportunity without my having to ctrl alt e.  
Received another sample log files. 
Saved it on below linked drive with details.
https://drive.google.com/open?id=0B01YYztUbOCuNVNOUlcxQTdCQTQ

ChromeOS version:
59.0.3071.134 
60.0.3112.112	

Device model:HP Chromebook 11 G5 EE relm.

Sample error message.
CRIT kernel: [    2.499660] EXT4-fs error  [Reason: ALERT!!:Filesystem error]  (device mmcblk0p1): ext4_mb_generate_buddy:757: group 4, block bitmap and bg descriptor inconsistent: 11693 vs 11652 free clusters

Cc: tnagel@chromium.org sonnyrao@chromium.org
Some pointer of related issues in the past:
- reks had an issue with TPM reset (that would trigger a recovery in the bug)
https://docs.google.com/document/d/1ZpS2-tXqaXh4HY_h6umMJAn-ZOQypIyS8C4J4weQS0A/edit#

- more info why corruption can happen on panic in crbug/502898

- being in the same cart reminds me issue crbug/431030 where the cart would power cycle devices a lot and auto rollback would be initiated and fail due to a change in file layout.
Blocking: 710290
Cc: apronin@chromium.org
Adding Andrey, who fixed a major issue when testing ARC++ on ecryptfs (b:36092409)

tl;dr: Corrupted data is aligned to 4k blocks and has a pattern that repeats every 512 Bytes which itself consists of a repeating pattern of 16 Bytes. Does that ring a bell for anyone?

I went through 5CD71137D2debug-logs_20170829-143125.tgz, 5CD7185VCHdebug-logs_20170829-134839.tgz and 5CD7187H20debug-logs_20170829-135935.tgz and analyzed file corruptions in the log files themselves. There is plenty of corruption in the log files (many files affected) and on all three devices it follows a similar scheme.

Very often corruption starts at the beginning of the file, but I've also seen offsets of 4000h, 50000h, 15000h, e000h, 21000h ==> it seems that corruption is aligned to 4k boundaries. The corruption pattern is always the same: 16 Bytes of varying data followed by a 16 Bytes pattern of identical data that is repeated 31 times:

00000000  6d a7 04 30 49 54 b2 15  32 10 6b 50 f6 49 53 c0  |m..0IT..2.kP.IS.|   <-- varying data
00000010  09 86 7c f3 07 cf 8e 17  4d f6 83 eb d5 ba 98 1c  |..|.....M.......|   <-- identical data
*
00000200  4c 9a 66 33 19 bb d1 27  a4 0d 11 a5 5b bd 10 cd  |L.f3...'....[...|   <-- varying data
00000210  09 86 7c f3 07 cf 8e 17  4d f6 83 eb d5 ba 98 1c  |..|.....M.......|   <-- identical data
*
00000400  18 b3 66 b6 f0 d9 eb 00  de 07 16 e2 e1 4f 12 7d  |..f..........O.}|   <-- varying data
00000410  09 86 7c f3 07 cf 8e 17  4d f6 83 eb d5 ba 98 1c  |..|.....M.......|   <-- identical data
*
00000600  c7 6d 80 55 73 2c 6b eb  1a f3 8f 0a 7d 84 b2 46  |.m.Us,k.....}..F|   <-- varying data
00000610  09 86 7c f3 07 cf 8e 17  4d f6 83 eb d5 ba 98 1c  |..|.....M.......|   <-- identical data
*
[and so on]

This form of corruption is the same across all three devices, except that each device has a different value for "identical data" (which however is stable for all observed corruption on the same device). A few more details are described in [1]. There's evidence that update_engine config files contain the same type of corruption.

[1] https://docs.google.com/document/d/1wM3qYnE9-Af_SSC6PTGvtecYNr9anGnGkyHMX6Vs1dw/edit#
Same issue here with Lenovo N42-20 (non-touchscreen) reks devices. About half, after giving them wifi info, will go straight to enrollment screen; the other half will go to a login screen with "managed by" on it. 
I'd be interested if writing zeroes to the backing store of encrypted stateful could lead to the observed pattern when reading from encrypted stateful i.e. whether decryption of zero blocks could produce this pattern.
#37, you are on something.

On a sane system, run debugfs on /dev/mapper/encstateful:

debugfs:  ex /var/log/messages
Level Entries       Logical        Physical Length Flags
 0/ 1   1/  1     0 -   231 230396             232
 1/ 1   1/  5     0 -     9 254516 - 254525     10
 1/ 1   2/  5    10 -    11 233039 - 233040      2
 1/ 1   3/  5    12 -    45   1476 -   1509     34
 1/ 1   4/  5    46 -   127   1372 -   1453     82
 1/ 1   5/  5   128 -   231   1920 -   2023    104

Corrupt the same blocks on /dev/loop0 where /var/log/messages is mounted:

dd if=/dev/zero of=/dev/loop0 seek=254516 bs=4k count=100

messages is still valid, but after reboot, it is corrupted with the pattern you observed:
od -x /var/log/messages | less

0000000 ee6e 7ac6 3e69 feb0 66f4 58f9 c58b 788d
0000020 a48a 25a3 60b5 1c74 113a 8be4 f8f8 8c21
*
0001000 0705 9626 2927 97fe d01d 21d3 cf06 ded7
0001020 a48a 25a3 60b5 1c74 113a 8be4 f8f8 8c21
*
0002000 cfc0 666b cf32 e426 f898 569f c262 7f5a
0002020 a48a 25a3 60b5 1c74 113a 8be4 f8f8 8c21
*
0003000 b3d1 feb9 c0b7 fa47 3967 2474 3770 866e
0003020 a48a 25a3 60b5 1c74 113a 8be4 f8f8 8c21
*
0004000 5b70 fea4 e1c3 246b c3d3 2752 b98f a4bb

However, as expected no corruption is reported by ext4 in mmc0blockp1.
Weird pattern in encstateful is an indication that some LBAs have been zero'ed out.

Going further, creating a directory /var/log/test_dir and a file in it.

mkdir /var/log/test_dir
truncate /var/log/test_dir/test --size 2G
sync

debugfs:  blocks /var/log/test_dir     
230503 

zero'ing that block and reboot
[ dd if=/dev/zero of=/dev/loop0 seek=230503 bs=4k count=1]

lead to 'ls /var/log/test_dir/test' to return empty and the message:
2017-09-01T14:51:38.652006-07:00 CRIT kernel: [   21.638743] EXT4-fs error (device dm-1): htree_dirblock_to_tree:976: inode #57215: block 230503: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=319288257, rec_len=31578, name_len=120

So we explain the dm-1 messages and corruption: from the loop device, we try to access blocks that have been zero'ed, or never used.

As Ted pointed out in https://www.spinics.net/lists/linux-ext4/msg55509.html, the fs errors we see on mmcblk0p1, like "ext4_mb_generate_buddy:757: group 9, block bitmap and bg descriptor inconsistent: 31834 vs 31917 free clusters" do not have to be in ext4 but can be elsewhere in the system.





Comment 39 Deleted
I received another log file and I saved it on below linked drive site. 
https://drive.google.com/open?id=0B01YYztUbOCuVHNvWXFSNTRkWXM
Device model:Kefka.7287.337.0
ChromeOS version:
59.0.3071.91
60.0.3112.112
I've updated [1] with further results from 5CD7187H20debug-logs_20170829-135935.tgz. Key findings:

* All corrupted data result from a single session with an uptime of 21 seconds.
* There is no evidence of a single uncorrupted fresh 4k block being written during that session.
* Very curiously, all partial writes of 4k blocks during that session seem to have succeeded: Log files contain valid data of that session until a 4k boundary is reached, at which point the turn to garbage.

[1] https://docs.google.com/document/d/1wM3qYnE9-Af_SSC6PTGvtecYNr9anGnGkyHMX6Vs1dw/edit#
5CD7185VCHdebug-logs_20170829-134839.tgz shows a similar pattern: 22 seconds uptime, all full blocks written seem corrupt, all appended-to blocks seem ok.

5CD71137D2debug-logs_20170829-143125.tgz is different though: Here the uptime is 47 minutes, and there is less corruption and the corruption patterns are somewhat different.
There are at least two, possibly three issues which are getting conflated in this bug report.  First we have the reports of the file system corruption.  These are the ones that involve EXT4-fs error reports.  And there we have two different sorts of metadata corruption.  The one in the original bug report is an inconsistency between the free block counts in the block group descriptor and the block allocation bitmap:

2017-08-22T08:18:32.625772-04:00 CRIT kernel: [    2.564271] EXT4-fs error (device mmcblk0p1): ext4_mb_generate_buddy:757: group 6, block bitmap and bg descriptor inconsistent: 24036 vs 24090 free clusters

This can be caused by a kernel bug or a hardware issue.  (For example, the Debian Jessie kernel for a while had a bug where if it was used as a Host OS in a virtualization setup, with a specific Intel CPU model, it would cause this precise corruption in the *Guest* OS.  This took almost a year to track down, since Debian Jessie used 3.16, and the bug was fixed upstream in 3.17, and the Debian kernel maintainers missed the bug fix in 3.17, and since it was fixed in 3.17, the bug fix didn't show up in the 3.18.x long-term stable kernels --- and it took forever for people to realize it was the host OS kernel that was critical to repro'ing the bug.)   With flash storage, many flash storage devices are not power-fail rated, so it could be that a flash write to the allocation bitmap or the block group descriptor is getting lost after a power failure.

The other metadata corruption that has been discussed in this bug is a corrupted inode.  An example of this is in comment #18:

2017-08-22T10:03:18.737089-07:00 CRIT kernel: [    7.751661] EXT4-fs error (device sda1): ext4_iget:4044: inode #129809: comm MountThread: bad extra_isize (1157 != 256)

This is a completely different syndrome, and it's most likely caused by the entire block in the inode table getting corrupted.  The way to tell would be to use debugfs to examine adjacent inodes in the same 4k block, and see if they look simiarly insane.  This can be caused by any number of things, including FTL hijinks after a power failure, or a write getting sent to wrong block, etc.  (Or a kernel bug, of course, but there's a reason why we run xfstests aggressively upstream to try to shake down file system errors before a release.  This won't save us from kernel bugs coming from a device driver, or some wild pointer dereference in a GPU code, but we can at least try to find the ext4 bugs before we do upstream releases.)

The other issue being discussed here is the data block corruptions, and that's probably a completely different thing.   It's possible this also caused by FTL hijinks, but this is the least likely to be caused by a hardware issue.  First of all, the contents of data files after a crash are completely unguaranteed unless userspace successfully completes a fsync(2) or msync(2) with MS_SYNC system call.   If the file system is mounted with the data=writeback mount option, this is completely what is expected.  If the file system is mounted with data=ordered, then having freshly written blocks containing garbage shouldn't have happened --- the file should have been sparse (e.g., no physical block mapping for the newly written logical block), so that the read of that logical block should have gotten zeros, not an actual block of zeros that then got decrypted to be zeros.  So that's almost certainly a kernel bug, but it might have been caused by device mapper allowing trims, and falsely allowing the "discard zeros blocks" flag for dm-crypt to be passed through from the hardware.   There are some cases where will try to use discard to gaurantee that a block will be zero'ed out --- but it seems unlikely they would show up in a log file write.   So if the log file was getting written using mmap (some C libraries use mmap to do standard I/O writeS), it could be that we're seeing some wierd combination of a crash plus delayed allocation and data file writes using mmap that is causing this.  It would still be wrong, but it might be why we're seeing garbage instead of the proper sparse (unwritten and unassigned) block.

We could try to debug the data log write, but a completely different issue than the file system metadata corruption problem, which is probably the much bigger deal.  Whether the last bit a of log file is properly left sparse after a power failure is probably not that big of a deal.  For any kind of properly written application journal file, presumably fsync() or msync() is properly used to guarantee that critical data is safely on stable store even after a power failure.
Cc: msnoxell@chromium.org
Blocking: 731943
I've added a new log in this share:
https://drive.google.com/open?id=0B01ZVp8vDQocQ0NEb2l3QnlzTkU
if you have new logs please add to this share (google only access read/write)
Thanks a lot, Ted, for your detailed comment!

I'll summarize the kinds of corruptions that I have observed:
1. Files containing repetitive garbage aligned to block boundaries, identified as the result of decrypting zero blocks.
2. Files containing data that belongs to a different file, aligned to block boundaries and zero-padded to the next boundary (e.g. a log file containing contents that belong to metrics files).
3. Files containing runs of zeroes that end at a block boundary, to be followed by valid data.

All observed types of corruption are consistent with (some of) the fs metadata blocks being written to disk but failing to write (some of) the data blocks. Being the simplest explanation, I'd suggest to treat this as working hypothesis and to focus further investigation on the question why and where these blocks are lost (or have their write ordering changed). Encrypted stateful is mounted as data=ordered, thus the blocks *should* be pushed down the stack in the correct order.

(I'm signing off here since since I'm lacking familiarity with the kernel.) 

Sample 1: debug-logs_20170822-141304/messages.1
0001cfd0  73 61 6c 74 3d 32 64 34  32 33 33 65 66 38 66 39  |salt=2d4233ef8f9|
0001cfe0  34 34 63 33 30 30 35 61  37 38 34 39 36 31 34 38  |44c3005a78496148|
0001cff0  61 35 36 37 66 30 35 34  62 33 31 30 39 66 62 30  |a567f054b3109fb0|
0001d000  8a 1f 7d 3b de da 68 a0  c6 9e cd 83 63 07 ca 25  |..};..h.....c..%|
0001d010  f7 f6 19 7d 16 5c 27 55  45 13 ff 6a 80 b5 8a 2f  |...}.\'UE..j.../|
*
0001d200  2f 66 83 79 c0 28 72 c2  95 23 5c 8b af 6d 3b 69  |/f.y.(r..#\..m;i|
0001d210  f7 f6 19 7d 16 5c 27 55  45 13 ff 6a 80 b5 8a 2f  |...}.\'UE..j.../|
*
0001d400  fb 00 ae 74 a9 ae 35 d8  c0 1d 13 bf a7 19 70 94  |...t..5.......p.|
0001d410  f7 f6 19 7d 16 5c 27 55  45 13 ff 6a 80 b5 8a 2f  |...}.\'UE..j.../|
*

Sample 2: debug-logs_20170822-141304/net.1.log
00007fe0  20 49 4e 46 4f 20 73 68  69 6c 6c 5b 31 31 31 31  | INFO shill[1111|
00007ff0  5d 3a 20 5b 49 4e 46 4f  3a 64 65 76 69 63 65 2e  |]: [INFO:device.|
00008000  32 2e 34 36 20 32 2e 36  35 0a 00 00 00 00 00 00  |2.46 2.65.......|
00008010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00009000  36 2e 35 37 20 33 2e 30  33 0a 00 00 00 00 00 00  |6.57 3.03.......|
00009010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0000a000  34 2e 39 38 20 32 2e 37  36 0a 00 00 00 00 00 00  |4.98 2.76.......|
0000a010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0000b000  35 2e 30 31 20 32 2e 37  36 0a 00 00 00 00 00 00  |5.01 2.76.......|
0000b010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0000ba40  00 32 30 31 37 2d 30 38  2d 32 31 54 31 33 3a 33  |.2017-08-21T13:3|
0000ba50  34 3a 35 30 2e 30 32 39  38 35 39 2d 30 37 3a 30  |4:50.029859-07:0|
0000ba60  30 20 4e 4f 54 49 43 45  20 77 70 61 5f 73 75 70  |0 NOTICE wpa_sup|
0000ba70  70 6c 69 63 61 6e 74 5b  34 39 36 5d 3a 20 53 75  |plicant[496]: Su|
0000ba80  63 63 65 73 73 66 75 6c  6c 79 20 69 6e 69 74 69  |ccessfully initi|

Sample 3: debug-logs_20170822-141304/messages
00001400  49 4e 46 4f 3a 70 6f 6c  69 63 79 5f 73 65 72 76  |INFO:policy_serv|
00001410  69 63 65 2e 63 63 28 32  30 32 29 5d 20 50 65 72  |ice.cc(202)] Per|
00001420  73 69 73 74 65 64 20 70  6f 6c 69 63 79 20 74 6f  |sisted policy to|
00001430  20 64 69 73 6b 2e 0a 00  00 00 00 00 00 00 00 00  | disk...........|
00001440  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000  32 30 31 37 2d 30 38 2d  32 32 54 31 34 3a 31 30  |2017-08-22T14:10|
00002010  3a 32 33 2e 38 39 34 32  36 34 2d 30 37 3a 30 30  |:23.894264-07:00|
00002020  20 49 4e 46 4f 20 6b 65  72 6e 65 6c 3a 20 5b 20  | INFO kernel: [ |
00002030  20 20 20 30 2e 30 30 30  30 30 30 5d 20 49 6e 69  |   0.000000] Ini|

To be clear: The observed corruption is also consistent with *random* blocks being dropped from the write queue. (The distinction between data and metadata merely stems from what is observable by looking at a log tarball and what isn't.)

The observed corruption is *not* consistent with only the tail of the write queue being dropped. We definitely violate block write ordering in some way.
The corrupted inode table block report in #18 can definitely *not* be explained by blocks getting dropped by the write queue.    The discrepancy in block group descriptors and allocation bitmaps can be so explained, but given that there are many more data blocks written than metadata blocks (for most workloads), it would imply that there should be far fewer of the metadata corruptions compared to the data block corruptions.  (Of course the metadata corruptions are easier to detect...)

What may be useful to do is to preallocate a fixed large file, and write a userspace program which writes into the file using an order which is driven by a pseudo-random generator with a fixed seed, and based on the RNG, write a 4k block which contains a sequence number and timestamp in the data block.   This might be an easy way to determine whether or not writes are being dropped, in an easy to reproduce way.

Forgot to add: do the writes using O_DIRECT Writes.   Such a program is also useful in checking to see whether or not the storage device does the right thing in power fail scenarios when the battery level goes to 0%, soft/hard bus resets, push and hold the power button for eight seconds, etc.
Cc: tytso@google.com
> The corrupted inode table block report in #18 can definitely *not* be explained by blocks getting dropped by the write queue.

Interesting. I'm not seeing any "bad extra_isize" errors in my sample of log files [1].

But your comment reminds me of another type of corruption that I haven't reported yet (probably subconsciously rejected because I couldn't make sense of it):

I'm observing 4k blocks that are *almost* fine but are interspersed with 16 Bytes random corruption aligned to 512 Byte boundaries. For example the block 6000h-7000h in 5CD71137D2debug-logs_20170829-143125/messages.2 is fine except for these 8 corruptions (uncorrupted data is omitted):

00006000  2c 31 ff 73 b3 8c 41 be  0d c2 95 98 a4 4a f2 12  |,1.s..A......J..|
00006200  4a b9 c1 35 34 ca bf 29  81 46 e0 07 0e 51 73 3b  |J..54..).F...Qs;|
00006400  38 c0 4c df 3d 22 3b f2  1b a3 dc 36 fd bc 9b db  |8.L.=";....6....|
00006600  dd 18 0c 5b 19 0f 89 7d  63 fe f0 8c 1d 45 59 08  |...[...}c....EY.|
00006800  f1 f7 09 36 e4 38 31 1f  b1 26 12 a0 3e 3c 97 ca  |...6.81..&..><..|
00006a00  42 74 21 50 76 36 45 02  79 b4 a8 de 39 83 c5 b6  |Bt!Pv6E.y...9...|
00006c00  18 94 d3 83 e4 59 7c 53  de 6f 7c 0f 9c 2c 45 df  |.....Y|S.o|..,E.|
00006e00  1e a2 85 28 c1 ca 4e a6  aa a2 83 61 df 4a ce 01  |...(..N....a.J..|

What could be the source of this kind of corruption?

[1] My sample of logs:
  5CD71137D2debug-logs_20170829-143125
  5CD7185VCHdebug-logs_20170829-134839
  5CD7187H20debug-logs_20170829-135935
  debug-logs_20170822-140808
  debug-logs_20170822-141304
  debug-logs_20170824-135332
  debug-logs_20170829-104337
  debug-logs_20170830-104356
  debug-logs_20170830-111345
  debug-logs_20170831-185132

Add Jackson County School District to the list of affected school districts.  We have just under 4000 chrome devices.  We use a xirrus wireless network.  The devices are mostly Lenovo N22's with some Lenovo N23's.  Attached is one example:

debug-logs_20170905-104005.tgz
2.8 MB Download
Note that the "16 out of 512" corruption seems to be rare, I'm only seen it in two files: debug-logs_20170830-104356/ui/ui.20170828-104806 and 5CD71137D2debug-logs_20170829-143125/messages.2.

However I found some interesting new corruption pattern in debug-logs_20170830-104356/messages.2. Note how the fixed letters are increasing: a, b, c, ... 

00035f00  8f bf 60 f4 c8 71 60 f9  87 2d 60 fd 16 ef 60 fd  |..`..q`..-`...`.|
00035f10  d6 8d 60 ff 47 71 61 04  8b a1 61 04 d2 7e 61 08  |..`.Gqa...a..~a.|
00035f20  b6 af 61 0a 3d 52 61 0a  f8 15 61 0a f9 fa 61 0b  |..a.=Ra...a...a.|
00035f30  6d ee 61 14 61 01 61 1c  89 c1 61 1c ae a2 61 22  |m.a.a.a...a...a"|
00035f40  af fb 61 3c 00 c4 61 3f  5f b8 61 42 a5 20 61 44  |..a<..a?_.aB. aD|
00035f50  7f bd 61 46 ec bf 61 50  07 b2 61 52 6e 46 61 53  |..aF..aP..aRnFaS|
00035f60  55 4e 61 55 ec 00 61 58  18 f9 61 59 58 40 61 59  |UNaU..aX..aYX@aY|
00035f70  a8 d6 61 5d 44 4c 61 63  3b a0 61 66 40 a9 61 6d  |..a]DLac;.af@.am|
00035f80  60 d4 61 75 e5 5f 61 8e  cb 5a 61 93 46 c0 61 97  |`.au._a..Za.F.a.|
00035f90  53 c8 61 a4 36 b7 61 a9  22 1a 61 aa b1 6c 61 ac  |S.a.6.a.".a..la.|
00035fa0  7f 67 61 ae 19 c3 61 b4  8e 36 61 c0 09 75 61 c0  |.ga...a..6a..ua.|
00035fb0  6c 16 61 c1 74 7b 61 c4  ab 7c 61 c9 0b 2d 61 ce  |l.a.t{a..|a..-a.|
00035fc0  5f 49 61 cf a3 d2 61 dc  88 1a 61 dc 94 70 61 e1  |_Ia...a...a..pa.|
00035fd0  19 c4 61 e8 f8 d1 61 e9  7d 36 61 ec 4b 84 61 fc  |..a...a.}6a.K.a.|
00035fe0  ac c2 61 ff 83 99 62 06  59 41 62 06 82 ea 62 0a  |..a...b.YAb...b.|
00035ff0  95 a1 62 0a da 16 62 0b  b8 9f 62 21 45 11 62 21  |..b...b...b!E.b!|
00036000  8d df 42 43 a3 54 32 75  08 5a 50 70 f3 3d fc db  |..BC.T2u.ZPp.=..|
00036010  5d 6b 62 43 b3 71 62 47  3c 02 62 4a 5d bc 62 4a  |]kbC.qbG<.bJ].bJ|
00036020  8b 1c 62 4a bc 0d 62 4d  bb 9d 62 4e d8 10 62 4e  |..bJ..bM..bN..bN|
00036030  dc eb 62 58 88 48 62 5a  d9 75 62 5c 6d 27 62 5c  |..bX.HbZ.ub\m'b\|
00036040  c3 10 62 69 4c 06 62 6a  51 39 62 6a 6b 09 62 6e  |..biL.bjQ9bjk.bn|
00036050  bc 4f 62 7a 50 e9 62 85  4c 8a 62 86 29 b3 62 89  |.ObzP.b.L.b.).b.|
00036060  db ff 62 8a 78 0e 62 94  33 e0 62 9e e0 4e 62 a2  |..b.x.b.3.b..Nb.|
00036070  b3 5c 62 ae 9a c2 62 b1  c9 bf 62 b4 25 dd 62 b6  |.\b...b...b.%.b.|
00036080  54 70 62 b8 f2 87 62 b9  4d ee 62 c1 6a 8b 62 c5  |Tpb...b.M.b.j.b.|
00036090  a8 b5 62 cb b6 67 62 cd  71 db 62 d0 13 91 62 d2  |..b..gb.q.b...b.|
000360a0  50 34 62 da 16 af 62 e5  97 54 62 eb 82 06 62 f6  |P4b...b..Tb...b.|
000360b0  4b 3c 62 f7 ef 90 62 fd  8f 8f 63 04 73 67 63 04  |K<b...b...c.sgc.|
000360c0  d6 b2 63 09 ca 0f 63 0a  23 94 63 0a 30 58 63 0c  |..c...c.#.c.0Xc.|
000360d0  01 c8 63 0e 7d db 63 0f  43 2b 63 10 3b 36 63 25  |..c.}.c.C+c.;6c%|
000360e0  ab 16 63 27 36 37 63 2b  28 d0 63 2f d9 06 63 34  |..c'67c+(.c/..c4|
000360f0  56 d0 63 3a 17 b0 63 42  aa d9 63 55 c6 96 63 58  |V.c:..cB..cU..cX|
00036100  20 4c 63 5c 41 0c 63 5c  7e e1 63 6b 5e b4 63 6e  | Lc\A.c\~.ck^.cn|
00036110  06 24 63 76 26 89 63 7a  19 29 63 80 d4 65 63 84  |.$cv&.cz.)c..ec.|
00036120  6a 73 63 91 9d 7f 63 91  b9 f8 63 92 ee de 63 99  |jsc...c...c...c.|
00036130  2c cd 63 99 b6 be 63 9d  7d 9f 63 a0 2d 02 63 ab  |,.c...c.}.c.-.c.|
00036140  41 11 63 ad df d9 63 b4  d9 d4 63 cb aa 60 63 cc  |A.c...c...c..`c.|
00036150  1e 9b 63 d0 13 b8 63 d0  aa 0d 63 d6 a5 f1 63 dc  |..c...c...c...c.|
00036160  43 18 63 e0 08 8b 63 e5  c7 1c 63 fa 04 5e 63 fb  |C.c...c...c..^c.|

gwendal@ can you please share next steps on this stable blocker?
Noticed the same problem 4 out of 7 asus chromebox.
The common thing is that the electric power is turned off by a timer every day.
Looking at #31 logs and my test machine, I notice mmcblk0p1 and/or dm-1 are not properly unmounted/remounted read only as
"EXT4-fs (mmcblk0p1): recovery complete" appears on every reboot in messages. file. Filed 761911

#49, I wrote a similar script, tracking a similar issue at https://bugs.chromium.org/p/chromium/issues/detail?id=693439#c58

Enclosed a modified version to start with:
Run a fio script in a file created by truncate (the older version was using the block device directly).
In parallel, run fstrim on the filesystem in a tight loop.


stress_760007.sh
341 bytes View Download
stress_760007.fio
489 bytes Download
more devices reported as affected - CTL J5/J2 Chromebook (Jerry)

- Chrome OS version. 
60.0.3112.112 

Drive link to Debug logs and additional information: https://drive.google.com/open?id=0By9xEkAUmm_cczJtQk5oX2l4akU
Hot Springs School District can also report this issue with Dell 11 3180's that we just purchased.
Prosper ISD has seen this issue on over 1,000 of our 15k devices, mostly Acer C740s and C731s.  Logs available if needed.
Our Devices are Acer C740 Chromebooks. In our environment, it looks like the following:

Units are "self updating " to Chrome OS v 60. ( Many before were v. 45, v.49 or v. 57)
After they self update, they are thrown out of Management and lose the Wifi SSID info. 

Our fixes thus far:
1. Put them back on their WiFi SSID
2. Re-enroll the device
3. Device self checks to determine it has " the latest Chrome OS and is up to date". This goes by very quickly, and checking after step 4 verifies it is at Chrome OS v60.
4. Ready for student use.

We have filed a Case with Google as well.

Terry Jackson
Yukon Public School District
We have Lenovo N22,N23's
Same as everyone else, they are deprovisioning themselves and losing the network key. 
We re-provision them but the same Chromebook will fall off again.
We just started battling this, any idea on the percentage or is it random?


Blockedon: 762979
Restricted bug for internal discussion: issue 762979.
Everyone may already know this, but this is only happened after upgradeding to OS 60.  Unfortunately, we skipped from 53 to 60.  No Chromebooks running 53 experienced this issue at Jackson County School District.

Similar case here at my school District (Support Case #13602725) with Lenovo N23 cart. We have two carts in different OUs, but the same network, chrome version, etc. 1 cart is working fine, but the other prompts for re-enrollments every morning.
From looking at metrics, 58 and 59 up to and including 59.0.3071.91 are safe. 59.0.3071.134, 60 and 61 very likely are broken. (Not entirely sure about 59.0.3071.113.)

If your deployment is affected by this problem I'd suggest to pin to 59.0.3071.91 or lower.
we don't allow pinning to minor release versions, only major. Customers would need to pin to 58 to avoid.
Actually most recovery images for boards are currently at 9460.60.0 / 59.0.3071.91. So disabling updates entirely for these devices and then restoring back to 9460.60.0 (current image for most boards right now, version shows in recovery tool when selecting image) should be one way to mitigate short-term.
I see how to pin my fleet in the admin console.  How can I run a report on which of my books is using which OS?
You can use Chromebook Inventory or GAM to get a report:
https://chrome.google.com/webstore/detail/chromebookinventory/iifofihmahmlokfnajjngehfnilmdmei?hl=en
https://git.io/gam

Note that pinning to 59 alone will not suffice. You'll need to either pin to 58 and rollback to 58 or rollback to 9460.60.0/59 and disable updates entirely to avoid getting 9460.73.0.
Is this a final fix, rolling back to 58? All my devices are past that, mostly on 60 now.
@nwilson: absolutely not a final fix, eng teams is still actively investigating. Just sharing as a possible mitigation short-term.
Enrollment wiped today. Has occurred on multiple devices.
debug-logs_20170907-103943.tgz
1.5 MB Download
more devices reported as affected:

- Chrome OS version. 
59.0.3071.91 
60.0.3112.112 

- Devices model 
ZAKO HP Chromebox CB1-(000-099) / HP Chromebox G1/ HP Chromebox for Meetings 

Drive link to Debug logs and additional information: https://drive.google.com/open?id=0B9JNylCOuXiuNkhDbUpmbWRveW8
We have had issues since last week with our Acer C740's appearing to wipe all on their own. We will re-enroll up to 30-40 a day in our environment. Most devices are moving from version 57 to version 60 as they have been turned off over the summer. Maybe an update issue? Please keep us posted on a fix for this and if you need any logs feel free to contact me.
 I am having the same issue has the original poster also with the Acer740's 
Another log analysis. https://docs.google.com/a/google.com/document/d/1PnQLAyOSMU2hNvfCRm76MeF1B34Y54_Fj6bs4cvOTiA/edit?usp=sharing

Looking at the event log, there are a lot of power cycles. Other logs show a lot of short power cycles as well. As is, that's not enough to trigger the issue, short cycles seems to be the fact of life in carts.

Do we have a log of a machine where the issue happened outside a cart? 
Blockedon: 741019
Cc: puthik@chromium.org
Customers experiencing this isssue, can you please let us know if you are using a cart with the devices or not and if not please get logs?

https://support.google.com/chrome/a/answer/3293821
We are using carts and experiencing the issue.
sorry, can you please also mention which cart type/brand you are using if you are?
Anywhere Cart, ac-plus something, I'd have to look up the exact model. Notably, perhaps, it has a switching power system that does A/B charging swaps every 15 minutes.
more devices reported as affected:

- Chrome OS version. 
59.0.3071.91 
60.0.3112.112 

- Devices model 
HP Chromebook 11 G5 EE relm

more devices reported as affected: https://drive.google.com/open?id=0B01YYztUbOCuRm14eTQybUhueDQ
another devices are affected.

- Chrome OS version. 
59.0.3071.134 
60.0.3112.112 

- Devices model 
 Lenovo N22 (Touch) Chromebook

more devices reported as affected: https://drive.google.com/open?id=0B01YYztUbOCuQ0ltSkVzSWFENGc
We do use power cycling carts.  I posted this question on the education forum for this issue (the one posted in this thread) as many people on there were mentioning that they either had a power cycling cart or some sort of timer system. I believe my logs were used to open this bug but happy to provide them again if I am mistaken or you would like to have them.
Here's a shared folder for 3180's with error logs.  As we get more we'll keep adding to this folder.
https://drive.google.com/open?id=0B1yGKNmKS8f9VWw1UWpmeTdoSFk
We are using Ergotron Anthro Yes carts.  I believe they have a 30 minute Upper/Lower cycle.
Experiencing issue in 50 or so known in fleet of 5000+. Appears to be specific to Acer C720 (not seeing on HP11 G4/G4, Lenovo N21, at this time). Devices have been unused all summer and are being used for first time since. Using cart: Spectrum Cloud32 Chromebook Cart - cart - with 5" Balloon Wheels, eLogix Ti
https://www.cdwg.com/shop/products/Spectrum-Cloud32-Chromebook-Cart-cart-with-5in-Balloon-Wheels-eLogix-Ti/4471324.aspx?pfm=srh

We're new to the 1 to 1 world; trialing them this year with four classrooms of students.  Asus C202SA here; all purchased this summer...all came pre-loaded with version 60+...and many of them are exhibiting the same issues you all are seeing.

I've shown our trial teachers how to fix each, so it's less work for me, but aggravating for them since they have to take a few minutes per period to make sure all the the devices are ready to go.  They are recording the serials of the devices they have to re-manage.  Some of the devices this has happened to over 3 times.  
Same here... 15-20 per day having to be re-enrolled. We appreciate the efforts you're making to resolve this. CDI eduGear M4 models mostly.
Acer R731 - problem keeps returning on device after re-enrolling. 
TL;DR: trying to reproduce the fs corruption by aggressively power cycling.

We are using the eMMC with Power off notification enabled: to improve performance, boot up time, we have to tell the eMMC that we are about to shut it down. When we don't the eMMC firmware will have to work harder to present the data, if it can.

After adding log in mmc_core.c, I notice we sent the GSMI notification before power off notification to the eMMC:

[  119.058507] gsmi: Log Shutdown Reason 0x00
[  119.100024] mmc0: Power Off Notification Sent, 600
[  119.119442] ACPI: Preparing to enter system sleep state S5
[  119.125702] reboot: Power down      

kernel_shutdown_prepare()
  --> call chain kernel_shutdown_prepare --> include GSMI code
  --> device_shutdown : shutdown all device including eMMC.

There is small window where power can fail after gsmi and before Power Off notification where we would still see a clean shutdown in the event log, but the eMMC would not have receive its notification.

Looking at the logs, I notice there are unclean power sequence on the units. 
For instance, in 5CD71137D2debug-logs_20170829-143125, 
ext4 reports the first critical error at:
2017-08-28T14:18:31

In event log, there is an unclean shutdown (33):
243 | 2017-08-24 11:08:48 | System boot | 32
248 | 2017-08-24 12:04:08 | Kernel Event | Clean Shutdown
250 | 2017-08-25 13:10:53 | System boot | 33
258 | 2017-08-25 13:10:59 | System boot | 34
263 | 2017-08-25 13:58:01 | Kernel Event | Clean Shutdown
265 | 2017-08-28 14:18:28 | System boot | 35


I tried to just disable the final flush and power off notification, but I could not trigger the error; the kernel just notices the 'needs_recovery' flag, does some cleanup and goes on.

I am looking at reproducing the ext4 error message locally by forcing a corruption, hard powering the device while doing creating and writing files in dm-1.

In the meantime, until the file system corruption is fixed, we could merge CL/567510 (and friends) in M60. That CL prevent writing the policy at every shutdown when not needed, so when a FS corruption happens at shutdown, it will not necessarily trigger an unenrollement.

Including test files as reference, keeping tweaking them.
power_cycle_stress1.sh
318 bytes View Download
stress_760007.sh
194 bytes View Download
stress_760007.fio
514 bytes Download
Cc: -dskaram@chromium.org
I just had this same issue on an ASUS C202SA.
last policy fetch time: sunday 9/10/17 3:58pm
tome : monday 9/11/17 7:30am the devive had lost it's enrollment and could not attach to wifi and was on the welcome screen.
we just had a few similar incidents being reported on ASUS N62 boxes
(originally CFMs, being used as Kiosks), suspect related to this.  tbc


*  •  **Peter Kakasi*
*  •  Sales Engineer*
*  •  *Android <https://www.google.com/work/android/> & Chrome
<https://www.google.com/work/chrome/>

*  •  pkakasi@google.com <pkakasi@google.com>*
  •  +61 410 136086
gwendal@ thanks for the update. Please keep us posted.
We are also having this issue with all of our ASUS C202S chromebooks. We purchased just over 1,000 of them. It appears to happen when the chromebook updates to 60. I stopped updates at 59 and the chromebook that was at 58 updated to 59 with no problem. I can't say for sure that this happens to any chromebook more than once, but it seems to happen a lot (possibly always) during the upgrade from 59 or 58 to 60.
Having the same issue with Acer 740, maybe 720 too. 
Just want to note this is happening equally as often to some of mine that do not use carts and are not returning to use just now after a long period of disuse.
Can anyone post log for the machine that do not use carts?
This is happening in our district too.  We are re-enrolling on a daily basis, very frustrating.
This is happening in our district also (Acer chromebooks - C720, C730, C740).  It appears to occur in groups - 5 or 10 from a single classroom at a time.  When I put the wifi credentials back in it appears to finish updating and then I must re-enroll the device.  All of our chromebooks were probably on 58 before school started so I assume they are updating to 59 or 60. 
  I did have a device today that I fixed... logged in with a test login and it worked.  Powered it down and handed it back to the student - they powered back up and it was un-enrolled again.

ejohnson@ please share logs from C720 unit:

https://support.google.com/chrome/a/answer/3293821?hl=en

are all of these devices on carts? Have they been untouched until now or were they opened just before school started and allowed to update (any details you can share about how devices are being used, if students log off after use, etc may help).
We have Lenovo N22, N21 and N23. The N22s are take home and has been happening to them since the start of school 7-31-2017. The N23 and N21 stay at school in charging cabinets that do not have a timer to alternate which bank is charging.
My poor techs are going insane.  Is there any way to create a USB to load 58?  I've capped our devices, but we have entirely too many already impacted.  
#106: use the Chromebook Recovery Utility to burn a flash drive with a theoretically working 59. If you want 58, you'll have to hunt around for it on the google recovery site. (For Lenovo N22/42 Chromebooks, it's https://dl.google.com/dl/edgedl/chromeos/recovery/chromeos_9334.72.0_reks_recovery_stable-channel_mp-v2.bin.zip. Unzip that, use it as a "local image" in the recovery utility.)
I have been in the habit of archiving Chrome OS ISOs each month or so. I somehow missed 58, but could share the files for v57 for the Dell CB 11 and Dell CB 3189 if it would help anyone. Although I think the newer versions of Chrome get cranky if you try a USB stick with an older version...
"you'll have to hunt around for it on the google recovery site"

Mind elaborating on how you do this :)  I just get 404 errors accessing 
anything other than your direct link.

On 9/12/2017 11:36 AM, jsm… via monorail wrote:
#109: attempted to send you email via your website as addresses are obscured here.
Please send to me as well.  TY
Let's try this: email me at jsmith@stillwaterschools.com (for instructions on how to find older recovery images)
We have seen this since the school year began, and we are seeing several new cases every day. The only way to reconnect and re-provision is to connect the CB to a cellular hotspot and re-provision. Very disruptive and time consuming.
Cc: ssmadan@chromium.org
My carts are not special and do not cut off. They have simple surge protectors. Please use my earlier log. 

Please don't take this as a criticism but why not look at the changes that occurred from 58 to 59 specifically around filesystem residency?  Even the carts didn't experience the issue with that version?

If nothing else, send an update that resembles upgrade 58 as 61.  I have over 4000 chromebooks all of which are succeptible. This is putting us in a huge bind.

Respectfully,
David
Here's another debug log from an affected Lenovo N42 that came out of a cart (but not a fancy auto-switching one).
debug-logs_20170912-151223.tgz
2.0 MB Download
several issues noted while debugging the issues:
- on one system, filesystems are not unmounted properly:

Both dm-1 and stateful are not unmounted properly.

The logs do not say exactly why. [enclosed]
On some logs, I can see it happening on older images as well.

- After the migration from a release that does not enable ext4 crypto to one that supports it, we set the encrypt flag in the stateful superblock in chromeos-startup. However, since we are recovery stateful at mount just after, the flag is removed. Preventing the removal is addressed with cl/541176, but this CL is neither in R59 nor R60.

- cryptohomed still uses ext4 crypto even if the flag is not set in the superblock on 3.18 or 3.14 kernels.
This is addressed with cl/549046 for 3.18 (pending). On the surface, this is harmless because we have formatted the filesystem properly (ordered, 4k pages) already. 

I have check the cleanup path, but the code does not check if encrypt filesystem superblock flags are set properly, ext4 is directly looking at the inodes data structures.

Working on finding the culprit(s) of the mount failures. We are giving 1s grace for unmounting to work, I am suspecting that may not be enough on slower systems.



shutdown_stateful_umount_failure
2.5 KB View Download
umount-encrypted.log
189 bytes View Download
Very interesting find! It seems that encrypted stateful unmounts ok, thus I don't think there's a problem with the 1s interval. (There's no mention of /var or /dev/mapper/encstateful in shutdown_stateful_umount_failure.)

It's the unmount of stateful that fails. Maybe it would help to add "-r" to these lines: 

umount -n /usr/local /home /mnt/stateful_partition
Just checked: "-r" doesn't help.

I have documented repro steps for how an umount failure could come about in issue 764643. (This may or may not be the mechanism that is causing umount failures in the wild.)

I'm surprised that the encrypt flag in the superblock doesn't stick since chromeos_shutdown calls sync of which the man page says: "Force changed blocks to disk, update the super block."
Status: Started
I'm having the same issue with Lenovo N42s. Our school district has about 500 CBs and are having to re-enroll 4 or 5 a day. This has been going on far too long. I hope this is a top priority for Google and Chromium engineers.
Has anyone found a way to identify devices that have dropped their enrollment from the control panel (or GAM/Chrome Gopher/chromebookInventory)?

With a bunch of devices out in MS and HS students' hands...and an easily accessible guest network...my fear is that devices could un-enroll and we might never find out.
This is happening at our school district with our ACER C740 chromebooks, on V60. I have to re-enroll 5-10 of these per day now in my school of a few hundred chromebooks, and it never did this before this summer. The issue doesn't jump over to our ACER C710 chromebooks or our Chromium computers, however.
Our district has been having this issue also.  We have Acer C740.  We have to re-enroll several chromebooks each day.
Cc: dskaram@chromium.org jayhlee@chromium.org
Issue 759946 has been merged into this issue.
This is happening with our Dell Chromebook 11 3180 Chromebooks. We purchased 120 over the summer and it's happened with about 30~40 of them. Only thing we can do is reset and re-enroll them into the domain. Hasn't happened with the older models we have here, just these ones. 
continuing #117:

in chromeos_shutdown, we are exiting the loop at the first iteration:
"mount-encrypted umount" succeeds, even if "dmsetup remove /dev/mapper/encstateful" fails. 
It fails because ioctl DM_DEV_REMOVE, flags DM_EXISTS_FLAG returns EBUSY.

We are neither specfying -f (force) nor --deferred (not implemented in the version of lvm2 we are using, forked from lvm2-2.02.97)

Trying 10 times does not help.

Tearing down the loopback device succeed, but it is not destroyed (confirmed by adding output of losetup -a after). This is expected if the refcount on the loop device is greater than 1, the tear down (ioctl LOOP_CLR_FD) is deferred (aka AUTOCLEAR) until the refcount is 0.

So umount dm-1 has succeed but the cleanup is incomplete.

jbd2/dm-1 is still running after unmount, which is unexpected.

Looking around, chromeos-3.18 kernel needs jdb2 patches:
> 44d77496ba72 jbd2: fix FS corruption possibility in jbd2_journal_destroy() on umount path
> 8fecc1e2c4b4 ext4, jbd2: ensure entering into panic after recording an error in superblock
> ffb04e43c08a jbd2: avoid infinite loop when destroying aborted journal
> 1215720518ca jbd2: fix ocfs2 corrupt when updating journal superblock fails

I still don't know how to put a device in a state so that dm-1/stateful will always need to be recovered.

The Lenovo N22s and N23s in our school district are being hit by this bug. They appear to be running versions 59+ and from my encounters, those not fully affected by the re-enroll issue do not seem to be getting the Admin console changes they should be getting.  
The biggest problem we have is trying to get the problem to replicate.  It happens at random across over 2000+ chromebooks with no notable pattern.  
We do use chromecarts that have timers built in to load balance, but we also have chromebooks that are not on a timer system of any sort.  Some are in a kiosk mode for our K-3 students, while others are for our 4-8 students.  So far, our high school has been lucky enough to not be hit.  

A working theory was that they were dropping out on an age base, but the N23s were new, so that theory went out.  Every pattern I have looked for has given the same results: random.  
I'm seeing the same issue of Chromebooks losing enrollment with new Dell Chromebook 11-3189.
It appears to be like all other comments here, random. So far these are our newest Chromebooks and the only ones being affected. Need a resolution!
If the jbd thread is still running, could it be that some process still has the file system mounted in another mount namespace? 
We are having this issue at our school. At first I thought it was kids wiping the devices, but I pulled a Chromebook and the key combinations that would do this, do not work. We have Asus Chromebooks. 
A dumb question, the ChromeBooks should be both cable and battery powered. But in the eventlogs attached in #1, #25, #52 and #73, I see lots of SuS Power Fail. How could that happen?
"SUS Power Fail" is *not* indicative of a problem. I don't remember the exact details, but it's just one of the on-board systems saying that power to it has been cut. This happens every time the device powers down.
Comment 134 Deleted
We are having this issue with our Levovo N22 devices. 5 of the devices with issues are on Chrome OS: 53, 60, 51, 58, 60, 59
-We are now pushing a new Chrome OS build, 9592.94.0 to devices that have upgraded to 9460.61.0 or newer.

-Devices on 9460.60.0 or older will not receive this build at this time, they'll stay on 9460.60.0 which is not showing this issue.

-We believe the new build will minimize unenrollment occurrences. We are still working hard to understand and solve the root cause (filesystem corruption).

-If you see devices unenrolling on 9592.94.0, please grab logs and attach them here.

https://support.google.com/chrome/a/answer/3293821?hl=en
We are having a much more severe version of this issue.  Our devices are stuck in a reboot loop when connected to wifi after upgrading to 59.

Anyone else having boot problems?
#133: SUS power fail means the DRAM lose power in SUSpend state and should not be relied upon, which is expected when we shutdown.

#127, #130: the leak is due to imageloader, which normally release the 3 mounts for encstateful (encstateful, /var and /home), but only 2 on my systems.
With cl:668108, on a working systems:
[    5.274457] imageloader clone mount 3                                                                
[    5.280166] imageloader clone mount 4                                                                
[    5.284369] imageloader clone mount 5                                                                
[    5.304725] imageloader deactivate 6                                                                 
[    5.311713] imageloader deactivate 5                                                                 
[    5.318650] imageloader deactivate 4 
then at shutdown:
[   31.255749] mount-encrypted deactivate 3
[   31.266740] mount-encrypted deactivate 2
[   31.275739] mount-encrypted deactivate 1
[   31.280144] mount-encrypted deactivate locked 1

While on my systems, I have:
[    5.198967] imageloader clone mount 3
[    5.198996] imageloader clone mount 4
[    5.199007] imageloader clone mount 5
[    5.205333] platform vpd: Driver vpd requests probe deferral
[    5.206496] platform vpd: Driver vpd requests probe deferral
[    5.224986] platform vpd: Driver vpd requests probe deferral
[    5.253329] device-mapper: verity: Argument 0: 'payload=/dev/loop6'
[    5.253347] device-mapper: verity: Argument 1: 'hashtree=/dev/loop6'
[    5.253358] device-mapper: verity: Argument 2: 'hashstart=29784'
[    5.253369] device-mapper: verity: Argument 3: 'alg=sha256'
[    5.253380] device-mapper: verity: Argument 4: 'root_hexdigest=c36ba182e0a1a533dd079ea00bbb9289d0300ae5e8ab2d03feeb25be6a4e29c0'
[    5.253397] device-mapper: verity: Argument 5: 'salt=90b9b5d4b7fda62e70011df97b403095330ae3dbe9f03501e726d79879c01439'
[    5.253413] device-mapper: verity: Argument 6: 'error_behavior=eio'
[    5.305022] imageloader deactivate 6
[    5.305040] imageloader deactivate 5

Then at shutdown:
[  120.242911] mount-encrypted deactivate 4
[  120.248912] mount-encrypted deactivate 3
[  120.254905] mount-encrypted deactivate 2
We are not at refcount 1, so umount does not complete.


re #138 - I see the same thing on my Caroline

I dumped the stack there at deactivate and it looks like this:

[    2.196633] deactivate_super: ffff880075e26800 s->s_id = dm-1 s_active = 5 comm = imageloader
[    2.196642] CPU: 3 PID: 564 Comm: imageloader Tainted: G        W      3.18.0 #78
[    2.196649] Hardware name: Google Caroline/Caroline, BIOS Google_Caroline.7820.263.0 01/26/2017
[    2.196658]  0000000000000000 00000000e39e1f3a ffff88017595fe28 ffffffff8c6a1140
[    2.196670]  0000000000000000 ffff880075e26800 ffff88017595fe48 ffffffff8c155226
[    2.196681]  ffff880176142280 ffffffff8ce71720 ffff88017595fe68 ffffffff8c16bd35
[    2.196693] Call Trace:
[    2.196698]  [<ffffffff8c6a1140>] dump_stack+0x4e/0x71
[    2.196705]  [<ffffffff8c155226>] deactivate_super+0x60/0x96
[    2.196712]  [<ffffffff8c16bd35>] cleanup_mnt+0x59/0x78
[    2.196719]  [<ffffffff8c16bd94>] __cleanup_mnt+0x12/0x14
[    2.196726]  [<ffffffff8c07bf78>] task_work_run+0x7e/0xab
[    2.196734]  [<ffffffff8c065268>] do_exit+0x43b/0x964
[    2.196745]  [<ffffffff8c0e0386>] ? seccomp_phase1+0x48/0x95
[    2.196760]  [<ffffffff8c066657>] do_group_exit+0x43/0xb1
[    2.196776]  [<ffffffff8c0666d9>] SyS_exit_group+0x14/0x14
[    2.196785]  [<ffffffff8c6a651c>] system_call_fastpath+0x1c/0x21

so what is happening there is that when the task exits we run through some tasks in the task_work_run() function.  Tasks are added to the list of tasks by task_work_add() and one of the users of task_work_add() is mntput_no_expire() which is adding the "__cleanup_mnt" callbacks.

So it seems like somewhere we're missing a call to mntput() somewhere in the kernel, which might be a little hard to track down quickly.  

This is all on kernel 3.18, so I think we should verify whether same thing is happening on the other kernels or not.

I'm also having the same issue, but with our Acer chromebooks. I'm not sure exactly which log file you guys are looking at, but here's some logs from one of mine.

https://drive.google.com/file/d/0B8a8QPPAhXpSOFhUNkI1d3llNGc/view?usp=sharing
Here's another dump from one of my acer chromebooks.

https://drive.google.com/file/d/0B8a8QPPAhXpSb3YzeU5DaHhvVVk/view?usp=sharing
Comment 142 Deleted
We are also seeing this on hp chromebook 11 ee g5.

So far about 100+ chromebooks have had this problem. 
What will trigger devices to take the new build?

Jeremy Smythe


Director of Information Technology

Capital City Public Charter School

www.ccpcs.org ☎ (202)-808-9755 <//(202)-808-9755>

On September 14, 2017 at 4:21:03 PM, jayh… via monorail (
monorail+v2.1523987270@chromium.org) wrote:
Comment 145 Deleted
We are experiencing this as well.  We have ASUS C202's.  950 units, I have had probably 10 the last 2 weeks.
Blocking: 763887
We are seeing this too. We have HP 11 G5 EE and we are starting to see couple of them unenrolling. 
We have more than 30000 Chromebooks and starting to see the number of unenrolled Chromeboosk creeping up. Even though we have force enrollment on, they are still coming up as though they were not enrolled. 
Regarding Comment 136,
I have Dell 3180, and the latest version is 9460.60.0 (ChromeOS v59( and we ARE having issues with it automatically unenrolling!
confirmed that if I disable imageloader (by renaming /etc/init/imageloader.conf) I am now able to get clean shutdowns for encstateful and stateful

so something in the way imageloader is running is causing the kernel to leak a reference to encstateful
re #151 -- I tried the same mitigation on kevin (4.4) kernel  but I'm still seeing unclean shutdowns of encstateful -- so there may be multiple causes for unclean shutdown, still investigating
Uploaded cl/670041: with this CL, we remove PepperFlashPlayer mount point before unmount encstateful: therefore, we are able to umount cleanly, preventing recovery at reboot.

This ensures we reboot with clean filesystems, reducing the risk of corruption at reboot. We still have to fully understand why ext4 crypto introduction prevents unmounted filesystems to be cleanup properly at reboot sometimes, but the proposed fix should reduce the rate of unenrollement.
Project Member Comment 154 by bugdroid1@chromium.org, Sep 16
Labels: merge-merged-release-R60-9592.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/4bc568d96a6fa3dbc0a188f1d7e1c4a304bd8ea1

commit 4bc568d96a6fa3dbc0a188f1d7e1c4a304bd8ea1
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Sat Sep 16 15:19:41 2017

Add script to unloaded loaded images

Add script to unload PepperFlashPlayer, its device mapper and loop devices.
It ensurses we can unmount encstateful and stateful properly.

BUG=chromium:760007
CQ-DEPEND=CL:669704
TEST=On cyan, check that dm-1 and mmcblk0p1 are mounted without repairs.

Change-Id: I3f065890968d59484acc74e3aed0ea8e307d752b
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/669910
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

[add] https://crrev.com/4bc568d96a6fa3dbc0a188f1d7e1c4a304bd8ea1/imageloader-shutdown.conf
[modify] https://crrev.com/4bc568d96a6fa3dbc0a188f1d7e1c4a304bd8ea1/imageloader.conf

This went to the wrong bug:

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/c68db285a46d62f2eeef4e1f677018e0a4ffee1b

commit c68db285a46d62f2eeef4e1f677018e0a4ffee1b
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Sat Sep 16 15:20:02 2017

image-loader: Add shutdown conf file

Add conf file to undo what has been done at load time.
In particular, be sure that PepperFlashPlayer resources are freed.

BUG= chromium:76007 
CQ-DEPEND=CL:669705
TEST=On cyan, check that dm-1 and mmcblk0p1 are mounted without repairs.

Change-Id: Ib7b883255ca71aea80b31785a2b8454f75189299
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/669869
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/c68db285a46d62f2eeef4e1f677018e0a4ffee1b/chromeos-base/imageloader/imageloader-9999.ebuild
Project Member Comment 156 by bugdroid1@chromium.org, Sep 16
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/c549b5e1b3cf7ddde35d6961f784e3720b3d7f78

commit c549b5e1b3cf7ddde35d6961f784e3720b3d7f78
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Sat Sep 16 19:15:04 2017

Add script to unloaded loaded images

Add script to unload PepperFlashPlayer, its device mapper and loop devices.
It ensurses we can unmount encstateful and stateful properly.

BUG=chromium:760007
CQ-DEPEND=CL:669704
TEST=On cyan, check that dm-1 and mmcblk0p1 are mounted without repairs.

Change-Id: I3f065890968d59484acc74e3aed0ea8e307d752b
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/669705
Commit-Ready: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/c549b5e1b3cf7ddde35d6961f784e3720b3d7f78/imageloader-shutdown.conf
[modify] https://crrev.com/c549b5e1b3cf7ddde35d6961f784e3720b3d7f78/imageloader.conf

Project Member Comment 157 by bugdroid1@chromium.org, Sep 18
Labels: merge-merged-release-R61-9765.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/7f84bd47d48bac90d5d39c4ec98a9ae9ed7f5869

commit 7f84bd47d48bac90d5d39c4ec98a9ae9ed7f5869
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Mon Sep 18 03:00:06 2017

image-loader: Add shutdown conf file

Add conf file to undo what has been done at load time.
In particular, be sure that PepperFlashPlayer resources are freed.

BUG=chromium:760007
CQ-DEPEND=CL:669705
TEST=On cyan, check that dm-1 and mmcblk0p1 are mounted without repairs.

Change-Id: Ib7b883255ca71aea80b31785a2b8454f75189299
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/669868

[modify] https://crrev.com/7f84bd47d48bac90d5d39c4ec98a9ae9ed7f5869/chromeos-base/imageloader/imageloader-9999.ebuild

Project Member Comment 158 by bugdroid1@chromium.org, Sep 18
Labels: merge-merged-release-R62-9901.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/72183321a8edbbe55e732fcfb2c0706a69069828

commit 72183321a8edbbe55e732fcfb2c0706a69069828
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Mon Sep 18 03:01:49 2017

image-loader: Add shutdown conf file

Add conf file to undo what has been done at load time.
In particular, be sure that PepperFlashPlayer resources are freed.

BUG=chromium:760007
CQ-DEPEND=CL:669705
TEST=On cyan, check that dm-1 and mmcblk0p1 are mounted without repairs.

Change-Id: Ib7b883255ca71aea80b31785a2b8454f75189299
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/669867

[modify] https://crrev.com/72183321a8edbbe55e732fcfb2c0706a69069828/chromeos-base/imageloader/imageloader-9999.ebuild

Project Member Comment 159 by bugdroid1@chromium.org, Sep 18
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/391ef9c3d99c8d4313e468d3ebcf3f3bba573796

commit 391ef9c3d99c8d4313e468d3ebcf3f3bba573796
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Mon Sep 18 03:01:49 2017

Add script to unloaded loaded images

Add script to unload PepperFlashPlayer, its device mapper and loop devices.
It ensurses we can unmount encstateful and stateful properly.

BUG=chromium:760007
CQ-DEPEND=CL:669704
TEST=On cyan, check that dm-1 and mmcblk0p1 are mounted without repairs.

Change-Id: I3f065890968d59484acc74e3aed0ea8e307d752b
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/669909

[add] https://crrev.com/391ef9c3d99c8d4313e468d3ebcf3f3bba573796/imageloader-shutdown.conf
[modify] https://crrev.com/391ef9c3d99c8d4313e468d3ebcf3f3bba573796/imageloader.conf

Project Member Comment 160 by bugdroid1@chromium.org, Sep 18
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/ef3ece24f05ab504c9ce62a4b32cf35d4ee8b2df

commit ef3ece24f05ab504c9ce62a4b32cf35d4ee8b2df
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Mon Sep 18 03:01:52 2017

Add script to unloaded loaded images

Add script to unload PepperFlashPlayer, its device mapper and loop devices.
It ensurses we can unmount encstateful and stateful properly.

BUG=chromium:760007
CQ-DEPEND=CL:669704
TEST=On cyan, check that dm-1 and mmcblk0p1 are mounted without repairs.

Change-Id: I3f065890968d59484acc74e3aed0ea8e307d752b
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/669908

[add] https://crrev.com/ef3ece24f05ab504c9ce62a4b32cf35d4ee8b2df/imageloader-shutdown.conf
[modify] https://crrev.com/ef3ece24f05ab504c9ce62a4b32cf35d4ee8b2df/imageloader.conf

Labels: M-62
Cc: ken...@chromium.org
Issue 765340 has been merged into this issue.
Cc: -dskaram@chromium.org
I'm having the same issue here but with Acer C740's. If I reboot the Chromebook a few times i can reproduce the issue. I have to re enroll at least 1-5 a day.
I have acer c740 and the same issue.  Many each day are brought back on the welcome screen with the wifi disconnected.  We push wifi to the device via Google Admin...

Attaching 2 logs....
HS Cart B 18.tgz
1.7 MB Download
HS Cart B 25.tgz
1.5 MB Download
Blockedon: 766347
We are seeing this on Lenovo N42 touch also and upon re-enroll it shows update available. Will post logs later.
Is there any update to this issue?  This has become a major issue for our school. 
Project Member Comment 169 by bugdroid1@chromium.org, Sep 19
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/16d735ce501cbfbfdb50998d1f048455716d11f1

commit 16d735ce501cbfbfdb50998d1f048455716d11f1
Author: Thiemo Nagel <tnagel@chromium.org>
Date: Tue Sep 19 14:11:06 2017

init: Add sync to chromeos_shutdown

Sync before unmounting in chromeos_shutdown to guard against present and
(potential) future unmount failures.

(This is intentionally placed before writing boot stats because the sync
time is legitimately part of shutdown and should be covered by the
stats. Due to the change in coverage, this CL will look like a
regression though it likely isn't one as the subsequent umount will be
sped up by approximately the amount of time that sync takes.)

BUG=chromium:760007
TEST=manual: reboot works

Change-Id: Ic5e577f3facfd3f1b12862b211248c387fb65210
Reviewed-on: https://chromium-review.googlesource.com/671244
Commit-Ready: Thiemo Nagel <tnagel@chromium.org>
Tested-by: Thiemo Nagel <tnagel@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@google.com>

[modify] https://crrev.com/16d735ce501cbfbfdb50998d1f048455716d11f1/init/chromeos_shutdown

We're now pushing 9592.96.0 / 60.0.3112.114 (Chrome build is same as .94 was) with a potential fix for the unenrollment issue.

We'd love to get feedback on 9592.96.0, if you see a device in the field that starts showing the unenrollment issue *after* upgrading to .96, please submit logs here.

We are holding off on pushing this new build to devices on older Chrome OS releases that do not have this issue. Once we further validate that it is solving the issue we'll push to all devices.
jayhlee, if we have devices pinned to 58 in the admin console that are currently sitting on some version of 59 or 60, will they upgrade to 9592.96.0 or do we need to move them to unpinned?
@jsmith: they'll need to be unpinned to get the update.
Has anyone seen 60.0.3112.114 get un=enrolled?
How do you know if they are pinned and how do you go about unpinning them?

By Pinned, they mean that Updates are being prevented.  You would change the setting Restrict Updates in Device Settings to Unrestricted.
60.0.3112.114 has indeed been affected as well, there has however been a firmware update to 60.0.3112.114 that can be seen if you look under Detailed build information. You can see the changes before and after a manual update. 
Not seen any issue with 9592.96.0 yet. Checking each machine as we re-enroll and they are all on a lower version. 
> Checking each machine as we re-enroll and they are all on a lower version.

Thank you for doing that!
Seeing devices with 59.0.3071.91 updating to 60.0.3112.114. Is the older OS no longer pinned or were we supposed to do that manually? 

Also seeing devices hang on white screen on boot after being on 60.0.3112.114. Been told by support that this could be because of remnants of data corruption (assume related to initial issue) and to wipe device to fix. Wiping device when initial issue presents will be our procedure going forward in hopes of avoiding white screen issue after un-enroll/re-enroll. 
We are updating 59.0.3071.91 because over time our metrics showed that this version likely is affected by the bug (contrary to what I wrote in comment #66). In that situation, updating to 60.0.3112.114/9592.96.0 which has a potential fix seems better than sticking around on a known-broken version. (58 and older are still blocked from updating, though.)
Yes, I am testing the update, just re-enrolled 7 Chromebooks with the issue and updated them...now we wait and see

Thanks for clarifying tnagel! 
tnagel@ any further updates on this issue? Have we validate the complete fix? can we close this issue?
I just implemented the fix yesterday, can we give it until Friday before we
jump and close it?
We'll hold the public bug open until we get full confirmation issue is resolved. Thanks.
This is the latest log from a device I work with.
debug-logs_20170921-145140.tgz
1.7 MB Download
How is the 9592.96.0 working for everyone? I have not seen any issues yet. Would love to hear from other users. 

We should not close this ticket until there is 100% certainty this issue is resolved. This has been a major headache for a lot of people. I am happy that it might be fixed but still upset it ever happened.  
jayhlee@ can you please clarify who and what you are looking for confirmation on?
@Ketaki: please see my email
jsmythe: thanks for the logs, looks like that device is on Chrome OS 59 and trying to update to the latest Chrome OS 60 that I mentioned in:

https://bugs.chromium.org/p/chromium/issues/detail?id=760007#c170

until the device is updated and re-enrolled, the issue is likely to continue.
It seems this bug has been corrected via the 9592.96.0 /60.0.3112.114 push.
Enrollment seems to be holding for us.
It seems like updating to 60.0.3112.114/9592.96.0 has been working for us, the ones that are still affected are on 9592.94
Cc: dskaram@chromium.org
Issue 767923 has been merged into this issue.
likewise here. It seems to holding with 9592.96.0

thank you for the quick turnaround 
Project Member Comment 195 by bugdroid1@chromium.org, Sep 26
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/ab5f056ff6a21a77fb0a19e4b61572f0eb8ee8a5

commit ab5f056ff6a21a77fb0a19e4b61572f0eb8ee8a5
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Tue Sep 26 23:14:08 2017

chromeos-init: Add UMA FS error stats

Move send-kernel-errors to its own script,
Reenable unit test and add test for send-kernel-errors.

BUG=chromium:760007
TEST=Unit test, check on ultima the script runs without error.

Change-Id: I0651e6e580153191ea7b1beffabd599794fc5ca9
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/682355
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/ab5f056ff6a21a77fb0a19e4b61572f0eb8ee8a5/chromeos-base/chromeos-init/chromeos-init-9999.ebuild

Project Member Comment 196 by bugdroid1@chromium.org, Sep 26
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/66d7eb52f04781db90792aca2cc73a6e7de0f462

commit 66d7eb52f04781db90792aca2cc73a6e7de0f462
Author: Gwendal Grignou <gwendal@chromium.org>
Date: Tue Sep 26 23:14:08 2017

init: Add UMA FS error stats

Move send-kernel-error code from .conf file.
Add unit tests.
Report to UMA when stateful/encrypted stateful have some errors.

BUG=chromium:760007
CQ-DEPEND=CL:682355
TEST=Unit test, check on gnawty

Change-Id: Ibbc0c25a5d4baa06624cd638e63dc289a37c48c4
Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/682596
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/tests/gather_battery_errors_2.golden
[add] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/tests/test_dmesg_1
[add] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/tests/test_dmesg_2
[modify] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/upstart/send-kernel-errors.conf
[add] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/tests/test_dumpe2fs_1
[add] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/chromeos-send-kernel-errors
[add] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/tests/send-kernel-errors-test.sh
[add] https://crrev.com/66d7eb52f04781db90792aca2cc73a6e7de0f462/init/tests/gather_fs_error_1.golden

Have been seeing the Public Session enrollment reset issue on Samsung Chromebook Plus models. I believe we have seen this issue while running 60.0.3112.114 but I have to verify with my team. 
Project Member Comment 198 by bugdroid1@chromium.org, Sep 27
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/3fc50e258087e49054e0f97bd6c66998673beeac

commit 3fc50e258087e49054e0f97bd6c66998673beeac
Author: Thiemo Nagel <tnagel@chromium.org>
Date: Wed Sep 27 19:57:58 2017

init: Add another sync call to chromeos_shutdown

Add a sync below bootstat in order to have a maximally clean filesystem
before unmount in the hope of minimizing the potential for corruption.
Also clarify the comment as to why sync is needed.

BUG=chromium:760007
TEST=none

Change-Id: I8c8873e56eeac2f0a3475217f7b3410fe1ef4173
Reviewed-on: https://chromium-review.googlesource.com/675688
Commit-Ready: Thiemo Nagel <tnagel@chromium.org>
Tested-by: Thiemo Nagel <tnagel@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/3fc50e258087e49054e0f97bd6c66998673beeac/init/chromeos_shutdown

Project Member Comment 199 by bugdroid1@chromium.org, Sep 28
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/7076091b18ab7d8b14722d224b456409b89070bb

commit 7076091b18ab7d8b14722d224b456409b89070bb
Author: Allen Webb <allenwebb@google.com>
Date: Thu Sep 28 18:18:50 2017

platform_ImageLoader: Added tests for --unmount_all and --unmount.

BUG=chromium:760007
CQ-DEPEND=CL:677792
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: Ic0547aa60a7c3d69bc29a73c0cb698583d366ac4
Reviewed-on: https://chromium-review.googlesource.com/682339
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>

[modify] https://crrev.com/7076091b18ab7d8b14722d224b456409b89070bb/client/site_tests/platform_ImageLoader/platform_ImageLoader.py

Project Member Comment 200 by bugdroid1@chromium.org, Sep 28
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/c1a83d303c0643d8d089d501a29a583268d3f43a

commit c1a83d303c0643d8d089d501a29a583268d3f43a
Author: Allen Webb <allenwebb@google.com>
Date: Thu Sep 28 18:18:50 2017

imageloader: Added an unmount_all flag.

BUG=chromium:760007
CQ-DEPEND=CL:682339
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: I1739d36c086c1a124d2c4c79ed2ec05b1fdd2987
Reviewed-on: https://chromium-review.googlesource.com/677792
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>

[modify] https://crrev.com/c1a83d303c0643d8d089d501a29a583268d3f43a/imageloader.gyp
[add] https://crrev.com/c1a83d303c0643d8d089d501a29a583268d3f43a/verity_mounter_impl.h
[add] https://crrev.com/c1a83d303c0643d8d089d501a29a583268d3f43a/verity_mounter_impl.cc
[modify] https://crrev.com/c1a83d303c0643d8d089d501a29a583268d3f43a/verity_mounter.h
[add] https://crrev.com/c1a83d303c0643d8d089d501a29a583268d3f43a/verity_mounter_unittest.cc
[modify] https://crrev.com/c1a83d303c0643d8d089d501a29a583268d3f43a/verity_mounter.cc
[modify] https://crrev.com/c1a83d303c0643d8d089d501a29a583268d3f43a/imageloader_main.cc

Any reported issues with the .114 build?
Project Member Comment 202 by bugdroid1@chromium.org, Sep 29
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/f4e8671e0ce791c7cc6ad570dfe03d695382fa30

commit f4e8671e0ce791c7cc6ad570dfe03d695382fa30
Author: Allen Webb <allenwebb@google.com>
Date: Fri Sep 29 21:35:33 2017

imageloader: Execute unmount_all flag on stopped ui.

BUG=chromium:760007
CQ-DEPEND=CL:677792
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: I0ec96c441b238ce3b9f3a5df220e60a2b9e1e814
Reviewed-on: https://chromium-review.googlesource.com/690438
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>

[modify] https://crrev.com/f4e8671e0ce791c7cc6ad570dfe03d695382fa30/imageloader-shutdown.conf

Project Member Comment 203 by bugdroid1@chromium.org, Sep 30
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/c745a92832f1f13ffaf32329306eb9c98a21a359

commit c745a92832f1f13ffaf32329306eb9c98a21a359
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Sep 30 03:16:29 2017

imageloader: add libdevmapper dep

BUG=chromium:770386
BUG=chromium:760007
TEST=precq passes

Change-Id: I7d0d208b03db5430f1c7a6f7485d78e4fc853c34
Reviewed-on: https://chromium-review.googlesource.com/693224
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Nicolas Norvez <norvez@chromium.org>
Reviewed-by: Allen Webb <allenwebb@google.com>

[modify] https://crrev.com/c745a92832f1f13ffaf32329306eb9c98a21a359/chromeos-base/imageloader/imageloader-9999.ebuild

So far 60.0.3112.114 looks good; all unenrollment issues we've had have been with older versions.
Status: Fixed
Project Member Comment 206 by bugdroid1@chromium.org, Oct 6
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/e43d8af075ffad0a1a7c690e21f5db7d13cba6c3

commit e43d8af075ffad0a1a7c690e21f5db7d13cba6c3
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Oct 06 23:53:41 2017

imageloader: add libdevmapper dep

BUG=chromium:770386
BUG=chromium:760007
BUG=chromium:772132
TEST=precq passes

Change-Id: I7d0d208b03db5430f1c7a6f7485d78e4fc853c34
Reviewed-on: https://chromium-review.googlesource.com/693224
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Nicolas Norvez <norvez@chromium.org>
Reviewed-by: Allen Webb <allenwebb@google.com>
(cherry picked from commit c745a92832f1f13ffaf32329306eb9c98a21a359)
Reviewed-on: https://chromium-review.googlesource.com/701574
Trybot-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Commit-Queue: Allen Webb <allenwebb@google.com>

[modify] https://crrev.com/e43d8af075ffad0a1a7c690e21f5db7d13cba6c3/chromeos-base/imageloader/imageloader-9999.ebuild

Project Member Comment 207 by bugdroid1@chromium.org, Oct 6
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/06c56d3140a4bb5be69a23a274d0475400b69b1d

commit 06c56d3140a4bb5be69a23a274d0475400b69b1d
Author: Allen Webb <allenwebb@google.com>
Date: Fri Oct 06 23:53:45 2017

platform_ImageLoader: Added tests for --unmount_all and --unmount.

BUG=chromium:760007
BUG=chromium:772132
CQ-DEPEND=CL:701594
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: Ic0547aa60a7c3d69bc29a73c0cb698583d366ac4
Reviewed-on: https://chromium-review.googlesource.com/682339
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>
(cherry picked from commit 7076091b18ab7d8b14722d224b456409b89070bb)
Reviewed-on: https://chromium-review.googlesource.com/690815
Trybot-Ready: Allen Webb <allenwebb@google.com>
Commit-Queue: Allen Webb <allenwebb@google.com>

[modify] https://crrev.com/06c56d3140a4bb5be69a23a274d0475400b69b1d/client/site_tests/platform_ImageLoader/platform_ImageLoader.py

Project Member Comment 208 by bugdroid1@chromium.org, Oct 6
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/69850078c6ab9bd3e46dc2312aa89e883adfb0dc

commit 69850078c6ab9bd3e46dc2312aa89e883adfb0dc
Author: Allen Webb <allenwebb@google.com>
Date: Fri Oct 06 23:53:52 2017

imageloader: Added an unmount_all flag.

BUG=chromium:760007
BUG=chromium:772132
CQ-DEPEND=CL:690815
CQ-DEPEND=CL:701574
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: I1739d36c086c1a124d2c4c79ed2ec05b1fdd2987
Reviewed-on: https://chromium-review.googlesource.com/677792
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>
(cherry picked from commit c1a83d303c0643d8d089d501a29a583268d3f43a)
Reviewed-on: https://chromium-review.googlesource.com/701594
Trybot-Ready: Allen Webb <allenwebb@google.com>
Reviewed-by: Allen Webb <allenwebb@google.com>
Commit-Queue: Allen Webb <allenwebb@google.com>

[modify] https://crrev.com/69850078c6ab9bd3e46dc2312aa89e883adfb0dc/imageloader.gyp
[add] https://crrev.com/69850078c6ab9bd3e46dc2312aa89e883adfb0dc/verity_mounter_impl.h
[add] https://crrev.com/69850078c6ab9bd3e46dc2312aa89e883adfb0dc/verity_mounter_impl.cc
[modify] https://crrev.com/69850078c6ab9bd3e46dc2312aa89e883adfb0dc/verity_mounter.h
[add] https://crrev.com/69850078c6ab9bd3e46dc2312aa89e883adfb0dc/verity_mounter_unittest.cc
[modify] https://crrev.com/69850078c6ab9bd3e46dc2312aa89e883adfb0dc/verity_mounter.cc
[modify] https://crrev.com/69850078c6ab9bd3e46dc2312aa89e883adfb0dc/imageloader_main.cc

Project Member Comment 209 by bugdroid1@chromium.org, Oct 6
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/08941abfd7fe79eb54f51f71e6fae87a3c9c619f

commit 08941abfd7fe79eb54f51f71e6fae87a3c9c619f
Author: Allen Webb <allenwebb@google.com>
Date: Fri Oct 06 23:53:55 2017

imageloader: Execute unmount_all flag on stopped ui.

BUG=chromium:760007
BUG=chromium:772132
CQ-DEPEND=CL:701594
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: I0ec96c441b238ce3b9f3a5df220e60a2b9e1e814
Reviewed-on: https://chromium-review.googlesource.com/690438
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Allen Webb <allenwebb@google.com>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>
(cherry picked from commit f4e8671e0ce791c7cc6ad570dfe03d695382fa30)
Reviewed-on: https://chromium-review.googlesource.com/701595
Trybot-Ready: Allen Webb <allenwebb@google.com>
Reviewed-by: Allen Webb <allenwebb@google.com>
Commit-Queue: Allen Webb <allenwebb@google.com>

[modify] https://crrev.com/08941abfd7fe79eb54f51f71e6fae87a3c9c619f/imageloader-shutdown.conf

Issue 765910 has been merged into this issue.
Blockedon: 773474
Project Member Comment 212 by bugdroid1@chromium.org, Oct 15
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/1e51d8b63530b571c20edc37c0e336637d1f9e00

commit 1e51d8b63530b571c20edc37c0e336637d1f9e00
Author: Allen Webb <allenwebb@google.com>
Date: Fri Oct 13 16:41:10 2017

imageloader: Fix to error logging for --unmount_all.

BUG=chromium:760007
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: Idf22cd6f52c00ce93a511c3ba54e9b9ea190e682
Reviewed-on: https://chromium-review.googlesource.com/713637
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>

[modify] https://crrev.com/1e51d8b63530b571c20edc37c0e336637d1f9e00/imageloader_main.cc

Project Member Comment 213 by bugdroid1@chromium.org, Oct 15
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/imageloader/+/27ca312bb82df59772edfa6b037d661569679f03

commit 27ca312bb82df59772edfa6b037d661569679f03
Author: Allen Webb <allenwebb@google.com>
Date: Fri Oct 13 17:54:53 2017

imageloader: Fix to error logging for --unmount_all.

BUG=chromium:760007
BUG=chromium:773924
TEST=test_that -b amd64-generic <ip:port> platform_ImageLoaderServer

Change-Id: Idf22cd6f52c00ce93a511c3ba54e9b9ea190e682
Reviewed-on: https://chromium-review.googlesource.com/713637
Commit-Ready: Allen Webb <allenwebb@google.com>
Tested-by: Greg Kerr <kerrnel@chromium.org>
Reviewed-by: Greg Kerr <kerrnel@chromium.org>
(cherry picked from commit 1e51d8b63530b571c20edc37c0e336637d1f9e00)
Reviewed-on: https://chromium-review.googlesource.com/718992
Commit-Queue: Greg Kerr <kerrnel@chromium.org>
Trybot-Ready: Greg Kerr <kerrnel@chromium.org>

[modify] https://crrev.com/27ca312bb82df59772edfa6b037d661569679f03/imageloader_main.cc

Sign in to add a comment