Kip fails to correct touchpad product ID
Reported by
charliemooney@chromium.org,
Jun 28 2016
|
|||||||||||||||||||
Issue descriptionWe've gotten reports on Kip of dead touchpads that simply don't recover. The logs on these devices are pretty useless -- they don't contain much information, because the touch_updater simply fails to find an appropriate FW to update the touchpad with. Feedback Link (Google-only) https://hotsauce.corp.google.com/product/208/neutron?rPaneMode=explore&rView=tl&lView=rd&rTagFamily=%23issue&lRSort=1&lROrder=2&lRFilter=1&lReportSearch=user:eroemer@kv.k12.in.us&lReport=10352641910 I worked with the user who reported this to get some more debugging information. I had him switch one of his units into dev mode and run various commands for me on the terminal to try and figure out why his Kips aren't getting updated. 1)localhost / # cat /etc/lsb-release CHROMEOS_AUSERVER=https://tools.google.com/service/update2 CHROMEOS_BOARD_APPID={BBE772C3-5541-6976-C676-54FD72D41B2B} CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1} CHROMEOS_DEVSERVER= CHROMEOS_RELEASE_APPID={BBE772C3-5541-6976-C676-54FD72D41B2B} CHROMEOS_RELEASE_BOARD=kip-signed-mp-v2keys CHROMEOS_RELEASE_BRANCH_NUMBER=38 CHROMEOS_RELEASE_BUILD_NUMBER=8350 CHROMEOS_RELEASE_BUILD_TYPE=Official Build CHROMEOS_RELEASE_CHROME_MILESTONE=52 CHROMEOS_RELEASE_DESCRIPTION=8350.38.0 (Official Build) beta-channel kip CHROMEOS_RELEASE_NAME=Chrome OS CHROMEOS_RELEASE_PATCH_NUMBER=0 CHROMEOS_RELEASE_TRACK=beta-channel CHROMEOS_RELEASE_VERSION=8350.38.0 DEVICETYPE=CHROMEBOOK GOOGLE_RELEASE=8350.38.0 2)localhost / # ls -1 /lib/firmware/elan* lrwxrwxrwx 1 root root 39 Jun 22 01:20 /lib/firmware/elan_i2c_72.0.bin -> /opt/google/touch/firmware/72.0_1.0.bin lrwxrwxrwx 1 root root 39 Jun 22 01:20 /lib/firmware/elan_i2c_75.0.bin -> /opt/google/touch/firmware/75.0_3.0.bin lrwxrwxrwx 1 root root 36 Jun 22 01:20 /lib/firmware/elan_i2c.bin -> /opt/google/touch/firmware/dummy.bin 3)localhost / # ls /opt/google/touch/firmware/ 72.0_1.0.bin 75.0_3.0.bin 4)localhost / # ls /sys/bus/i2c/devices/ 0-0015 1-0010 i2c-0 i2c-1 i2c-10 i2c-2 i2c-3 i2c-4 i2c-5 i2c-6 i2c-7 i2c-8 i2c-9 5)localhost / # for dev in /sys/bus/i2c/devices/*; do if [ -e "${dev}/driver" ] && [ "$(readlink -f ${dev}/driver | xargs basename)" = "elan_i2c" ]; then echo ${dev}; ls -l ${dev}/; for sysfs in firmware_version product_id fw_checksum iap_version; do echo "${sysfs}" -- 'cat ${dev}/${sysfs}'; done; fi; done /sys/bus/i2c/devices/0-0015 total 0 -r--r--r-- 1 root root 4096 Jun 28 08:12 baseline -r--r--r-- 1 root root 4096 Jun 28 08:12 calibrate lrwxrwxrwx 1 root root 0 Jun 28 07:55 driver -> ../../../../../bus/i2c/drivers/elan_i2c -r--r--r-- 1 root root 4096 Jun 28 08:12 driver_version lrwxrwxrwx 1 root root 0 Jun 28 08:12 firmware_node -> ../../../../LNXSYSTM:00/device:00/80860F41:00/ELAN0000:00 -r--r--r-- 1 root root 4096 Jun 28 07:55 firmware_version -r--r--r-- 1 root root 4096 Jun 28 08:12 fw_checksum -r--r--r-- 1 root root 4096 Jun 28 08:12 iap_version drwxr-xr-x 3 root root 0 Jun 28 07:55 input -r--r--r-- 1 root root 4096 Jun 28 08:12 modalias -r--r--r-- 1 root root 4096 Jun 28 08:12 mode -r--r--r-- 1 root root 4096 Jun 28 07:55 name drwxr-xr-x 2 root root 0 Jun 28 07:55 power -r--r--r-- 1 root root 4096 Jun 28 07:55 product_id -r--r--r-- 1 root root 4096 Jun 28 08:12 reinitialize -r--r--r-- 1 root root 4096 Jun 28 08:12 sample_version lrwxrwxrwx 1 root root 0 Jun 28 07:55 subsystem -> ../../../../../bus/i2c -rw-r--r-- 1 root root 4096 Jun 28 07:55 uevent --w------- 1 root root 4096 Jun 28 07:55 update_fw firmware_version -- 255.0 product_id -- 255.0 fw_checksum -- 0x0000 iap_version -- 9.0 ----------------------------- From what I can tell there are a couple things happening here 1. These Kips came with no FW on the touchpads at all -- they're reporting 255.0 for the product ID and the firmware version which is incorrect. Touchpads should always come with a valid firmware on them 2. Even so, the touchpad FW updater should recover them, but it looks like the board name in /etc/lsb-release is different than what I expected. On the effected machines it shows up as: CHROMEOS_RELEASE_BOARD=kip-signed-mp-v2keys but on my devices it shows up as just CHROMEOS_RELEASE_BOARD=kip Solution: It looks like all we need to do is fix the touch FW updater script to also accept "kip-signed-mp-v2keys" as a valid board name for kip. However, I don't understand why the name is different and would really like to reproduce this on a device locally first to confirm this fix works. We should also look into why these devices are leaving the factory without touchpad FW.
,
Jun 28 2016
I've worked up a patch that fixes the name-matching bug in the updater, and have tested it locally as well as sending a Kip image to the customer who's devices were failing who also confirmed it fixed his touchpad. The patch is merging into ToT now. We'll want to merge this back though once it's in
,
Jun 28 2016
,
Jun 28 2016
This is a pretty urgent fix -- the user who brought this to our attention and helped me debug this has several thousand devices they're deploying to a school, so it would be good to get this into 52/beta as soon as possible. Given the simplicity of the patch and the fact that I've tested it locally and the user tested the fix in the field, this merge should be very safe Requesting merge permission for R52
,
Jun 29 2016
We should also consider merging this to M51 given the urgency.
,
Jun 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/3f379a2b8808a435dc14a333f3f7ad82e54e2511 commit 3f379a2b8808a435dc14a333f3f7ad82e54e2511 Author: Charlie Mooney <charliemooney@chromium.org> Date: Tue Jun 28 18:13:57 2016 Match all boards starting with "Kip" This CL fixes a bug in the elan touch fw updater. There is a hack in the updater that will determine the Kip's product ID even if the touchpad's firmware has been corrupted by looking at the device's VPD. To do this, the script checks that the board's name is "kip" (among other things). However, previously the check was too strict, accepting only the exact string "kip", but it's come to light that the board name string often has a key name appended to the end like "kip-signed-mp-v2keys" which means the recovery hack won't work properly on devices like that. This CL changes the check to be more permissive, accepting any board that starts with the name "kip" but ignores the fact that it may have other text stuck on the end. This bug was difficult to track down due to lack of early debug messages in the logs, so it also adds a few more bits of information to the logs that may help debugging future issues. BUG= chromium:624001 TEST=manually tested on a device with an /etc/lsb-release file that matched the offending devices. Without this patch the update failed to recover a corrupted touchpad. With this patch, the update occurred just fine. Change-Id: Iec4064d397fc9fe69f5a9fbcf0a64ce89644d2f1 Signed-off-by: Charlie Mooney <charliemooney@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/356681 Reviewed-by: Dennis Kempin <denniskempin@google.com> [modify] https://crrev.com/3f379a2b8808a435dc14a333f3f7ad82e54e2511/scripts/chromeos-elan-touch-firmware-update.sh
,
Jun 29 2016
This just landed in ToT, but we really should merge it back into 52 & 51 as soon as possible
,
Jun 29 2016
,
Jun 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/1c14d0a9966ac4d602b0e37b8be2eec9ed2f5c7c commit 1c14d0a9966ac4d602b0e37b8be2eec9ed2f5c7c Author: Charlie Mooney <charliemooney@chromium.org> Date: Tue Jun 28 18:13:57 2016 Match all boards starting with "Kip" This CL fixes a bug in the elan touch fw updater. There is a hack in the updater that will determine the Kip's product ID even if the touchpad's firmware has been corrupted by looking at the device's VPD. To do this, the script checks that the board's name is "kip" (among other things). However, previously the check was too strict, accepting only the exact string "kip", but it's come to light that the board name string often has a key name appended to the end like "kip-signed-mp-v2keys" which means the recovery hack won't work properly on devices like that. This CL changes the check to be more permissive, accepting any board that starts with the name "kip" but ignores the fact that it may have other text stuck on the end. This bug was difficult to track down due to lack of early debug messages in the logs, so it also adds a few more bits of information to the logs that may help debugging future issues. BUG= chromium:624001 TEST=manually tested on a device with an /etc/lsb-release file that matched the offending devices. Without this patch the update failed to recover a corrupted touchpad. With this patch, the update occurred just fine. Change-Id: Iec4064d397fc9fe69f5a9fbcf0a64ce89644d2f1 Signed-off-by: Charlie Mooney <charliemooney@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/356681 Reviewed-by: Dennis Kempin <denniskempin@google.com> (cherry picked from commit 3f379a2b8808a435dc14a333f3f7ad82e54e2511) Reviewed-on: https://chromium-review.googlesource.com/357370 [modify] https://crrev.com/1c14d0a9966ac4d602b0e37b8be2eec9ed2f5c7c/scripts/chromeos-elan-touch-firmware-update.sh
,
Jun 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/1c14d0a9966ac4d602b0e37b8be2eec9ed2f5c7c commit 1c14d0a9966ac4d602b0e37b8be2eec9ed2f5c7c Author: Charlie Mooney <charliemooney@chromium.org> Date: Tue Jun 28 18:13:57 2016 Match all boards starting with "Kip" This CL fixes a bug in the elan touch fw updater. There is a hack in the updater that will determine the Kip's product ID even if the touchpad's firmware has been corrupted by looking at the device's VPD. To do this, the script checks that the board's name is "kip" (among other things). However, previously the check was too strict, accepting only the exact string "kip", but it's come to light that the board name string often has a key name appended to the end like "kip-signed-mp-v2keys" which means the recovery hack won't work properly on devices like that. This CL changes the check to be more permissive, accepting any board that starts with the name "kip" but ignores the fact that it may have other text stuck on the end. This bug was difficult to track down due to lack of early debug messages in the logs, so it also adds a few more bits of information to the logs that may help debugging future issues. BUG= chromium:624001 TEST=manually tested on a device with an /etc/lsb-release file that matched the offending devices. Without this patch the update failed to recover a corrupted touchpad. With this patch, the update occurred just fine. Change-Id: Iec4064d397fc9fe69f5a9fbcf0a64ce89644d2f1 Signed-off-by: Charlie Mooney <charliemooney@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/356681 Reviewed-by: Dennis Kempin <denniskempin@google.com> (cherry picked from commit 3f379a2b8808a435dc14a333f3f7ad82e54e2511) Reviewed-on: https://chromium-review.googlesource.com/357370 [modify] https://crrev.com/1c14d0a9966ac4d602b0e37b8be2eec9ed2f5c7c/scripts/chromeos-elan-touch-firmware-update.sh
,
Jun 30 2016
This change is in ToT & 52 now. That means the fix should be in Beta and Dev channels sometime next week. The builds haven't kicked off yet from what I can tell, I'll try to update here if I can get a better estimate in a day or two.
,
Jun 30 2016
#CBC-RS/TC-watchlist
,
Jun 30 2016
[Automated comment] Request affecting a post-stable build (M51), manual review required.
,
Jun 30 2016
Your change meets the bar and is auto-approved for M52 (branch: 2743)
,
Jun 30 2016
[Automated comment] Request affecting a post-stable build (M51), manual review required.
,
Jun 30 2016
the 52 merge is already done
,
Jul 6 2016
Do we have an ETA on 51 merge and stable release ? - This is breaking devices everywhere.
,
Jul 6 2016
Sorry, I know this is a serious issue. Doing my best to get it into 51. The happened right as 52 is going to become stable so it's difficult to get it into a 51 stable build atm. We need to see it working in 53Dev/52Beta first before we can merge into 51stable though. The change just got picked up in a beta build, so once that build it out to users we can confirm that switching to beta restores their touchpads. Once we've confirmed that we'll have a stronger case for patching stable in a hurry.
,
Jul 6 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/53f1c6824ab8f83e51ae38e95376eb674f6cff2d commit 53f1c6824ab8f83e51ae38e95376eb674f6cff2d Author: Charlie Mooney <charliemooney@chromium.org> Date: Tue Jun 28 18:13:57 2016 Match all boards starting with "Kip" This CL fixes a bug in the elan touch fw updater. There is a hack in the updater that will determine the Kip's product ID even if the touchpad's firmware has been corrupted by looking at the device's VPD. To do this, the script checks that the board's name is "kip" (among other things). However, previously the check was too strict, accepting only the exact string "kip", but it's come to light that the board name string often has a key name appended to the end like "kip-signed-mp-v2keys" which means the recovery hack won't work properly on devices like that. This CL changes the check to be more permissive, accepting any board that starts with the name "kip" but ignores the fact that it may have other text stuck on the end. This bug was difficult to track down due to lack of early debug messages in the logs, so it also adds a few more bits of information to the logs that may help debugging future issues. BUG= chromium:624001 TEST=manually tested on a device with an /etc/lsb-release file that matched the offending devices. Without this patch the update failed to recover a corrupted touchpad. With this patch, the update occurred just fine. Change-Id: Iec4064d397fc9fe69f5a9fbcf0a64ce89644d2f1 Signed-off-by: Charlie Mooney <charliemooney@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/356681 Reviewed-by: Dennis Kempin <denniskempin@google.com> (cherry picked from commit 3f379a2b8808a435dc14a333f3f7ad82e54e2511) Reviewed-on: https://chromium-review.googlesource.com/358355 [modify] https://crrev.com/53f1c6824ab8f83e51ae38e95376eb674f6cff2d/scripts/chromeos-elan-touch-firmware-update.sh
,
Jul 7 2016
Removing release block label
,
Jul 12 2016
FYI, this patch is going out to users on stable now. Currently at 5% of Kip users on "stable" branch. If no regressions are detected, it will go out to more and more people until we hit 100%. 100% is currently target before the end of the week. What this means is that if you're effected by this bug, just keep your device on "stable" channel and wait for the update -- it should be here by the end of the week as long as nothing unexpected happens.
,
Jul 12 2016
If you want to try to force an update go to about://help and click the check for updates button, this should get you a stable update to 8172.62.0 which should fix the touchpad.
,
Jul 18 2016
Please reply if you still see this issue with the latest stable build! Calling this Verified due to lack of user alarm.
,
Jul 20 2016
I have five or so units that still have no cursor control with the track pad. All are now in stable channel. All have been updated. I have tried wiping two devices. Reports and debug log has been shared with google support. Some devices are in a TestOrgUnit. Some are not. I have one that the issue was resolved.
,
Jul 20 2016
Hi Michele, I tried looking up your feedback reports using your email, but I'm only seeing two old reports from June 20th and 28th which are before the fix was pushed out. Were these feedback reported with a different email address?
,
Jul 21 2016
I've shared them with yana at google support as instructed. Is there a better way to share that info?
,
Jul 21 2016
Sorry, I'm not sure who that is -- I don't see a Yana on this bug. If this was someone in customer support, they may be using the term "feedback" differently. These bugs are mostly for the engineers working on fixing issues, so sometimes we have different terms for things, sorry for the confusion. When I mean feedback, I was referring to this process where users can send a feedback report to Google that includes some system logs and other debugging information for us: https://support.google.com/chromebook/answer/2982029?hl=en I searched the repository of logs sent in this way for ones associated with your email address but only found two old reports from before the fix was pushed out to everyone -- June 20th and 28th. This could be because the feedback was filed in "guest mode" or logged in as another user, either of which would result in your email not being associated with the report. If you can file feedback like in the instructions I linked to right after booting the Chromebook we should be able to get some insight into your issue.
,
Jul 21 2016
I just saw some new reports from you, thanks! Looking at them now
,
Jul 21 2016
Feedback reports for two of the five devices are submitted. Working on the rest.
,
Jul 21 2016
It looks like one of the devices is on Beta Channel and is still over ChromeOS version 8350.46.0, the fix wasn't included until 8350.48.0. On this device I believe the update will fix it -- you could try switching it to stable channel or manually checking updates with the instructions below: https://support.google.com/chromebook/answer/177889?hl=en On the other feedback, however, it seems that something else in awry. It appears that the touchpad is completely unable to be connected to the driver. I've created a new bug over at Issue 630333 to track it. I'm not sure what caused this, but unfortunately it appears to be a different issue.
,
Jul 21 2016
FYI for anyone who might be following along, the first builds that include the fix are as follows: stable -- 8172.62.0 beta -- 8350.48.0 |
|||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||
Comment 1 by charliemooney@chromium.org
, Jun 28 2016