s/screenshot/photo/, obviously can't take a screenshot.

Leaving it in this state for the next bit (~1 hr) in case it's useful for debugging. Will reboot to try to make the laptop usable after then.

After ~30 mins aborted using ctrl+c. Everything seems fine (trackpad works), so probably not a high priority issue.

I'm seeing the same while updating from M-67 stable to M-69 dev.

References:
https://groups.google.com/a/google.com/d/topic/chromeos-discuss/iGZrfzpyBK8/discussion
https://www.reddit.com/r/PixelBook/comments/927euk/dev_channel_frozen_on_touchpad_firmware_updating/

Making the bug public as I don't see anything confidential here.

This has been happening to me since the previous dev build(69.0.3497.14) sent my Pixelbook into a bootloop and I had to USB recovery.

ctrl+c aborts fine and it doesn't drop back into the firmware update in future reboots.

I have the same problem and just tried the ctrl-c and it does not drop back into a boot loop. Maybe this issue should be tied to: 
https://bugs.chromium.org/p/chromium/issues/detail?id=867519

I think this is a different reason.  

So I have a pixelbook which was in m66 before I decided to flash it to lastest dev version  (R70-10952.0.0) now,  after flash it and rebooting, it show the screen "Touchpad firmware update in progress" and actually it's there forever, then I found I still can ssh into it and I checked the log, here is some interesting thing in log:
1. It showed something like "update-firmaware-process exited with status 2"
2. In message, it also show "update touchpad firmware success".
So my guess is that: firmware update actually succeed but somehow the update process aborted and then the screen is stucked at "update in progess" for ever.

Just do reboot will bring back it.

The reason I tied it to 867519 is that if I rebooted while it was stuck in the touchpad firmware update, it would get stuck in the boot loop (I only tried on the three 69.0.x dev releases). However, if I did a ctrl-c to get out of the touchpad firmware update, it would go to the login screen. So maybe the boot loop is a result of the touchpad firmware exit issue. 

Re: comment #8

Looks like you're right, if the updater returns non-zero for some reason (even if the update actually succeeded) then the upstart script will not clean up the "critical update" screen afaict

https://chromium.googlesource.com/chromiumos/platform2/+/master/init/upstart/boot-update-firmware.conf

Can we shift this to RBS since the ctrl-c / reboot mitigates the message hang?

I think RBB

RE #12, is this due a recent change?  (https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1156055)

Hi, Shecky, could you take a look?

cc shecky, instead of assign.

mberes@

Could you confirm your touch firmware version?

Sure, if you can point me to instructions on how to get the requested version.

cat /sys/class/chromeos/cros_tp/version

Looking at the script I really can't understand why it could return 2 after successfully updated the FW. Would need to do more research.

https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/master/scripts/chromeos-google-touch-firmware-update.sh
https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/master/scripts/chromeos-touch-update.sh

Try to repro on two devices but no luck.

Steps:
1. AUed from M67 stable to M69 Dev

2. and also installed M67 stable recovery image and updated to M69 dev. 

Is this happening from M68 beta to M69 beta?

Any update to this issue?

A deeper look at the upstart script has shown that this has nothing to do with touch FW updates. "chromeos-touch-update.sh" is called as a child process so any error it got will not affect the parent process. The log also shows that chromeos-touch-update.sh has actually finished successfully.

I believe the error comes from the line 44 to 50 of boot-update-firmware.conf

  if [ "${frecon_pid}" != "$(pgrep frecon)" ]; then
    logger -t "${UPSTART_JOB}" "Restore frecon."
    pkill -9 -f frecon
    if is_developer_end_user; then
      frecon --daemon --enable-osc --enable-vts --pre-create-vts
    fi
  fi

Assigning the bug to the script's owner for further debugging.

Related issues:

https://buganizer.corp.google.com/issues/111450661
https://bugs.chromium.org/p/chromium/issues/detail?id=867519

Since all reports are related to dev channel, I am guessing these few lines have triggered the issue.

    if is_developer_end_user; then
      frecon --daemon --enable-osc --enable-vts --pre-create-vts
    fi

What is the status of this issue? This is blocking beta and needs resolution.  Please provide an update. Thanks.

  if [ "${frecon_pid}" != "$(pgrep frecon)" ]; then
    logger -t "${UPSTART_JOB}" "Restore frecon."
    pkill -9 -f frecon
    if is_developer_end_user; then
      frecon --daemon --enable-osc --enable-vts --pre-create-vts
    fi
  fi

What this snippet does is: after all update are finished, if display_boot_message was called before, i.e. the warning message is shown, then kill the frecon to clear the message. 
If the issue is that the message is not cleared after update, I guess maybe the frecon invoked by display_boot_message has the same PID as the original frecon. Then the message screen won't be cleared. Let me change the condition of that.

BTW, I cannot switch my Eve to stable channel. Now the version is: "10892.0.0 dev-channel soraka-unibuild (soraka) test."
I follow the instruction here,
https://support.google.com/chromebook/answer/1086915?hl=en
but nothing happen after I click "Change channel and Powerwash". Any advice about this?

The version string says "soraka-unibuild", but it should be eve? Maybe try reflashing eve image?

The bug has entered into beta channel now.

I experienced this in dev channel so reverted to beta via recovery but today the beta updated and the bug is back.

I got this when upgrading from m67 stable to m69 beta. I was in developer mode. I hard-rebooted the machine and was unable to log in, so I was forced to wipe by exiting developer mode.

Couldn't repro this issue after I installed M67 stable recovery image (R67-10575.58.0) and updated to M70 dev (R70-10934.0.0). 

Thanks all, I can reproduced it when I switch from stable R67-10575.58.0 to beta R69-10895.33. 

After surveying again, I think the issue should be at chromeos-touch-update.sh because it returns error code 2. I will bypass all the error of firmware update and guarantee the frecon will be restored at the end.

Could you share the log where chromeos-touch-update.sh exits with 2? In all the feedback logs where the user reports this problem, chromeos-touch-update.sh finishes successfully.

Also, since chromeos-touch-update.sh is called as a child process, I think it shouldn't affect the parent script even if it fails.

Seems the upstart stanza is different from a normal shell script. When a command exit with non-zero value, the whole stanza is stopped.

I haven't traced why chromeos-touch-update.sh exits with 2, will investigate now.
Anyway, I created a CL as #38 mentioned. It fixed this issue. Please take a look.
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1195283

I can help debug that if I can see the log where chromeos-touch-update.sh fails. Will review the CL. Thanks a lot for the fix!

After the fix CL landed, should I cherry-pick it to other branch?

I think m69 and beyond should be enough.

Issue 865970 has been merged into this issue.

> #40
FYI the document says: script instead gives shell script code that will be executed using /bin/sh. The -e shell option is used, so any command that fails will terminate the script.

http://upstart.ubuntu.com/getting-started.html

Thanks for the info! Didn't know that before.

Attached another log for the issue. In that log, it also seems that touch updater finishes successfully, but then the main script still exits with 2.

Deleted: debug-logs_20180713-143020.tgz 2.1 MB

debug-logs_20180713-143020.tgz 2.1 MB Download

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/8812126a9786bb7e73cc5ecb94b7dbcb7ad8693d

commit 8812126a9786bb7e73cc5ecb94b7dbcb7ad8693d
Author: Chih-Yu Huang <akahuang@google.com>
Date: Thu Aug 30 04:05:56 2018

init: bypass firmware update error.

boot-update-firmware.conf would call the firmware update script, and
clear the boot message if any of script showed the message. However,
if the script is terminated with error, the boot message won't be
cleared.

This CL bypasses the error. Then we can guarantee the boot message is
cleared no matter the scripts are executed successfully or not.

The verification step:
  1. Install M67 stable test image
  2. Flash local built test image to USB
  3. Boot from USB, then the boot message is shown
With this CL, the boot message is cleared.

BUG= chromium:869155 
TEST=Check the boot message is cleared

Change-Id: I66b07e0d7aca2397d002ade2fc25e00f7d5abf5d
Reviewed-on: https://chromium-review.googlesource.com/1195283
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Chih-Yu Huang <akahuang@chromium.org>
Reviewed-by: Tai-Hsu Lin <sheckylin@chromium.org>

[modify] https://crrev.com/8812126a9786bb7e73cc5ecb94b7dbcb7ad8693d/init/upstart/boot-update-firmware.conf

This bug requires manual review: We are only 4 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), kariahda@(iOS), cindyb@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Hi Cindy,

Please review whether the CL in #47 be able to cherry-pick to R69 or not.
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1196324

looks like the fix just force the script to continue on non-zero exit status.  We might still have a real problem here ? (from c#40)

Re #51,

Yes, we haven't found the reason why touch updater failed in this case. The CL only make sure the boot message is cleared in failure case. I think we should create another issue to track why touch updater failed.

Hi Shecky,

I tried to find where the script return non-zero value. But after I added some logger to print debug message, the script is executed successfully. So I guess maybe it's timing issue about parent/child process?

What kind of timing issue do you mean?

In one of the log I got, it seems that boot-update-firmware was determined to exit with 2 even if chromeos-google-touch-firmware-update reports success. So maybe the return state wasn't passed to boot-update-firmware correctly?

2018-07-13T12:46:10.149504-07:00 NOTICE chromeos-google-touch-firmware-update[1933]: Update FW succeded. Current Firmware: rose_v1.1.8546-ee1861e9e
2018-07-13T12:46:10.153856-07:00 NOTICE chromeos-google-touch-firmware-update[1937]: Attempting to re-bind 'i2c-ACPI0C50:00' to driver '/sys/bus/i2c/drivers/i2c_hid'
2018-07-13T12:46:10.162028-07:00 DEBUG kernel: [   59.190296] i2c_hid i2c-ACPI0C50:00: GPIO lookup for consumer reset
2018-07-13T12:46:10.162040-07:00 DEBUG kernel: [   59.190300] i2c_hid i2c-ACPI0C50:00: using ACPI for GPIO lookup
2018-07-13T12:46:10.162045-07:00 DEBUG kernel: [   59.190303] acpi ACPI0C50:00: GPIO: looking up reset-gpios
2018-07-13T12:46:10.162047-07:00 DEBUG kernel: [   59.190305] acpi ACPI0C50:00: GPIO: looking up reset-gpio
2018-07-13T12:46:10.162048-07:00 DEBUG kernel: [   59.190307] acpi ACPI0C50:00: GPIO: looking up 0 in _CRS
2018-07-13T12:46:10.162049-07:00 DEBUG kernel: [   59.190320] i2c_hid i2c-ACPI0C50:00: using lookup tables for GPIO lookup
2018-07-13T12:46:10.162050-07:00 DEBUG kernel: [   59.190322] i2c_hid i2c-ACPI0C50:00: lookup for GPIO reset failed
2018-07-13T12:46:10.188017-07:00 INFO kernel: [   59.216764] input: ACPI0C50:00 18D1:5028 Touchpad as /devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-8/i2c-ACPI0C50:00/0018:18D1:5028.0004/input/input15
2018-07-13T12:46:10.189018-07:00 INFO kernel: [   59.217118] hid-multitouch 0018:18D1:5028.0004: input,hidraw1: I2C HID v1.00 Mouse [ACPI0C50:00 18D1:5028] on i2c-ACPI0C50:00
2018-07-13T12:46:10.190900-07:00 NOTICE chromeos-google-touch-firmware-update[1946]: Success.
2018-07-13T12:46:10.193018-07:00 WARNING kernel: [   59.220994] init: boot-update-firmware pre-start process (628) terminated with status 2

Approve #50 merge to M69.

Got some new findings today.

It is neither chromeos-google-touch-firmware-update or boot-update-firmware which crash. Instead, chromeos-touch-update (caller of chromeos-google-touch-firmware-update) seems to crash after chromeos-google-touch-firmware-update has finished its job.

In chromeos-touch-update.sh:

  local dev
  local power_path
  for dev in $(get_dev_list); do
    # For devices supported runtime power managment, the "auto" mode would have
    # chance to let device go into runtime_suspend state. Which will prevent
    # touch updater from accessing that device. Here tries to move i2c devices's
    # state from "auto" to "on" then restoring them in the end of update
    # process.
    power_path="${dev}/power/control"
    if [ -e "${power_path}" ] && [ "auto" = "$(cat "${power_path}")" ]; then
      echo on > "${power_path}"
      restore_power_paths="${restore_power_paths} ${power_path}"
    fi

    ( check_update "${dev}" ) &
  done

  ...blah...

  wait

  # To restore runtime power management state from "on" to "auto" if they are
  # changed by this script in the begining.
  for power_path in ${restore_power_paths}; do
    echo auto > "${power_path}"
  done


It appears that the script may crash after the wait statement (which blocks until all updates finish) when restoring the power status of affected devices. I am able to reproduce exactly the same bug and log messages by manually placing an "exit 2" anywhere after the wait statement.

I am trying to figure out why "echo auto > "${power_path}" could crash though...

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The fixed CL is merged in ToT and M69. Close this issue.

Shecky could you create another issue to track the touch updater failure?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/2b2ce9945b68e9547772b45f96231b6d4677a076

commit 2b2ce9945b68e9547772b45f96231b6d4677a076
Author: Chih-Yu Huang <akahuang@google.com>
Date: Tue Sep 04 05:00:26 2018

init: bypass firmware update error.

boot-update-firmware.conf would call the firmware update script, and
clear the boot message if any of script showed the message. However,
if the script is terminated with error, the boot message won't be
cleared.

This CL bypasses the error. Then we can guarantee the boot message is
cleared no matter the scripts are executed successfully or not.

The verification step:
  1. Install M67 stable test image
  2. Flash local built test image to USB
  3. Boot from USB, then the boot message is shown
With this CL, the boot message is cleared.

BUG= chromium:869155 
TEST=Check the boot message is cleared

Change-Id: I66b07e0d7aca2397d002ade2fc25e00f7d5abf5d
Reviewed-on: https://chromium-review.googlesource.com/1195283
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Chih-Yu Huang <akahuang@chromium.org>
Reviewed-by: Tai-Hsu Lin <sheckylin@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1196324
Commit-Queue: Chih-Yu Huang <akahuang@chromium.org>
Trybot-Ready: Chih-Yu Huang <akahuang@chromium.org>

[modify] https://crrev.com/2b2ce9945b68e9547772b45f96231b6d4677a076/init/upstart/boot-update-firmware.conf

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/2b2ce9945b68e9547772b45f96231b6d4677a076

commit 2b2ce9945b68e9547772b45f96231b6d4677a076
Author: Chih-Yu Huang <akahuang@google.com>
Date: Tue Sep 04 05:00:26 2018

init: bypass firmware update error.

boot-update-firmware.conf would call the firmware update script, and
clear the boot message if any of script showed the message. However,
if the script is terminated with error, the boot message won't be
cleared.

This CL bypasses the error. Then we can guarantee the boot message is
cleared no matter the scripts are executed successfully or not.

The verification step:
  1. Install M67 stable test image
  2. Flash local built test image to USB
  3. Boot from USB, then the boot message is shown
With this CL, the boot message is cleared.

BUG= chromium:869155 
TEST=Check the boot message is cleared

Change-Id: I66b07e0d7aca2397d002ade2fc25e00f7d5abf5d
Reviewed-on: https://chromium-review.googlesource.com/1195283
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Chih-Yu Huang <akahuang@chromium.org>
Reviewed-by: Tai-Hsu Lin <sheckylin@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1196324
Commit-Queue: Chih-Yu Huang <akahuang@chromium.org>
Trybot-Ready: Chih-Yu Huang <akahuang@chromium.org>

[modify] https://crrev.com/2b2ce9945b68e9547772b45f96231b6d4677a076/init/upstart/boot-update-firmware.conf

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/2b2ce9945b68e9547772b45f96231b6d4677a076

commit 2b2ce9945b68e9547772b45f96231b6d4677a076
Author: Chih-Yu Huang <akahuang@google.com>
Date: Tue Sep 04 05:00:26 2018

init: bypass firmware update error.

boot-update-firmware.conf would call the firmware update script, and
clear the boot message if any of script showed the message. However,
if the script is terminated with error, the boot message won't be
cleared.

This CL bypasses the error. Then we can guarantee the boot message is
cleared no matter the scripts are executed successfully or not.

The verification step:
  1. Install M67 stable test image
  2. Flash local built test image to USB
  3. Boot from USB, then the boot message is shown
With this CL, the boot message is cleared.

BUG= chromium:869155 
TEST=Check the boot message is cleared

Change-Id: I66b07e0d7aca2397d002ade2fc25e00f7d5abf5d
Reviewed-on: https://chromium-review.googlesource.com/1195283
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Chih-Yu Huang <akahuang@chromium.org>
Reviewed-by: Tai-Hsu Lin <sheckylin@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1196324
Commit-Queue: Chih-Yu Huang <akahuang@chromium.org>
Trybot-Ready: Chih-Yu Huang <akahuang@chromium.org>

[modify] https://crrev.com/2b2ce9945b68e9547772b45f96231b6d4677a076/init/upstart/boot-update-firmware.conf

The touch updater issue will be followed up at https://buganizer.corp.google.com/issues/111450661

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot