> And the reason is /mnt/stateful_partition became readonly (probably caused by chromeos_shutdown itself...).

Yes, the logic there isn't very good [1]; we do this:

umount -n /var /usr/local /home /mnt/stateful_partition

and then later we try to log failures to /mnt/stateful_partitions/...

If the umount for /mnt/stateful_partition succeeded, then we'll now be trying to write to the (read only) rootfs, instead of to the stateful partition.

Gwendal suggested we could possibly make chromeos_shutdown try to remount /mnt/stateful_partition in that case, which could help to reduce the need for "stop the world" debugging here.

[1] I noted this here too:
https://b.corp.google.com/issues/76121632#comment80

Assign to fshao as good-first-bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/01cb2d8522f3b1f872aea7e8a0b3e6cbbd86c24e

commit 01cb2d8522f3b1f872aea7e8a0b3e6cbbd86c24e
Author: Fei Shao <fshao@google.com>
Date: Wed Sep 26 17:32:44 2018

init: Update unmount logic in chromeos_shutdown

Some modifications to ensure /mnt/stateful_partition won't be unmounted
if any of the previous unmounts failed, so that system can write
failure log into it.
Note that mounts like /var, /home, /usr/local are submounts of
/mnt/stateful_partition on regular images, so original logic works. But
on factory images, they can be independent from /mnt/stateful_partition
and system may fail to log because of it.

BUG= chromium:835557 
TEST=Manually tested on DUT.

Change-Id: I73b97e390e5a0110aae2bc691ed3293cf60bfc3e
Reviewed-on: https://chromium-review.googlesource.com/1227868
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Fei Shao <fshao@chromium.org>
Reviewed-by: Brian Norris <briannorris@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/01cb2d8522f3b1f872aea7e8a0b3e6cbbd86c24e/init/chromeos_shutdown

The bug of failing to create log files should be fixed by the CL above.

Regarding youcheng's first concern, instead of stopping shutdown process, one can enable "check_tag_file" flag for shutdown test (refer to /pytests/shutdown.py) and check failure logs after reboot.