I found 3 important things.
- Filename of the firmware for touchpad is not correct.
- Customer uses 'scatter update' > this may cause the difference between devices, I think.
- Firmware loading error occurred before updating. (by comparing the first timestamp of update_engine and messages) > but I'm not sure this is related to the root cause.


Here is the detail information for the firmware's filename:

chromeos-elan-touch-firmware-update should load firmware from "elan_i2c_" + version number + ".bin". (e.g. elan_i2c_75.0.bin)
but on the log file of devices with disabled touchpad, the version number is missing. (e.g. elan_i2c.bin)

I confirmed that there are no elan_i2c.bin in the develop mode. (there are elan_i2c_72.0.bin and elan_i2c_75.0.bin)
so I think that causes this issue. (but I'm not sure why the version number is missing)

The log file on the device whose touchpad is working properly.
var/log/messages.1
2016-05-20T15:02:03.182221-07:00 NOTICE chromeos-elan-touch-firmware-update[547]: Attempting to load FW: '/lib/firmware/elan_i2c_75.0.bin'
2016-05-20T15:02:03.184662-07:00 NOTICE chromeos-elan-touch-firmware-update[549]: Hardware product id : 75.0
2016-05-20T15:02:03.190742-07:00 NOTICE chromeos-elan-touch-firmware-update[551]: Updater product id  : 75.0
2016-05-20T15:02:03.198672-07:00 NOTICE chromeos-elan-touch-firmware-update[558]: Current Firmware: 3.0
2016-05-20T15:02:03.201389-07:00 NOTICE chromeos-elan-touch-firmware-update[560]: Updater Firmware: 3.0

The log file on the device whose touchpad is NOT working.
var/log/messages
2016-05-20T15:24:45.482051-07:00 NOTICE chromeos-elan-touch-firmware-update[925]: Attempting to load FW: '/lib/firmware/elan_i2c.bin'
2016-05-20T15:24:45.494917-07:00 NOTICE chromeos-elan-touch-firmware-update[937]: error: No valid firmware for ELAN0000:00 found.
2016-05-20T15:24:45.498285-07:00 NOTICE chromeos-touch-update[939]: ELAN0000:00 firmware update failed.

Does anyone know why the version number is missing on the firmware's filename?

The fix for https://code.google.com/p/chrome-os-partner/issues/detail?id=46266 landed in M51.

Can we reconfirm with the customer that switching to M51 beta and rebooting the machine doesn't fix the problem?  In theory, it should.

The trackpad has stopped working on our HP 14" Chromebooks.  This happened after the upgrade to Version 51.  If I reboot the chromebook the trackpad will work again for anywhere from 10 seconds to maybe 3 minutes.  We are downgrading to version 49 and stopping future updates until this issue is resolved.

Hi Jim,

I think that's a different issue than this specific bug, but I've got good news for you.  We've been working on fixing that issue you've described and already know the root cause (a bad touchpad firmware update).  The patches to fix it are already merged and a fix should be pushed to you shortly. I'm not sure the exact release schedule, but it should be very soon.  It's being tracked over on  Issue 619303 , if you want to follow along there.

Thanks for the report,
-Charlie

We just began processing another order of HP Chromebook 11 G4 EEs today (KIP C3A-G8F-N3A-A6A-A74). I ran the latest USB recovery for this device and found that it is still at ver. 49 (chromeos_7834.70.0_kip_recovery_stable-channel_mp-v2.bin). This update broke the trackpad again. When running the web update to ver. 51(stable), this did not fix the trackpad failure. I have attached a debug log in case you needed this for further investigation. As of now, the only way I have been able to repair this is to complete an RMA SHIM procedure on each of the affected devices, and update the OS over the web.

Can you please provide a downloadable copy of ver.51 that we can use to update the devices via USB?

Thanks

Deleted: debug-logs_20160617-090712.tgz 113 KB

debug-logs_20160617-090712.tgz 113 KB Download

Fix is incoming with our next stable push!
See  issue 619303 

Hi,

FYI
- The issue https://bugs.chromium.org/p/chromium/issues/detail?id=616394 refers to HP Chromebook 11 G4 EEs KIP C3A-G8F-N3A-A6A-A74
- The issue https://bugs.chromium.org/p/chromium/issues/detail?id=619303 refers to HP Chromebook 14 Google_Nyan_Blaze.5771.63.0

Are you going to merge HP Chromebook 11 G4 EEs and HP Chromebook 14 Google_Nyan_Blaze all together?

Ah, I take it back - these are likely not the same issue.  We /are/ going to revert the firmware on Kip back to 3.0 because of that issue, but this is an update problem.  Sorry - I got Duplicate happy :)


Logs from #7:
2021-09-24T17:11:40.097873+00:00 NOTICE chromeos-elan-touch-firmware-update[687]: Attempting to load FW: '/lib/firmware/elan_i2c.bin'
2021-09-24T17:11:40.100361+00:00 NOTICE chromeos-elan-touch-firmware-update[689]: error: No valid firmware for ELAN0000:00 found.
2021-09-24T17:11:40.101602+00:00 NOTICE chromeos-touch-update[690]: ELAN0000:00 firmware update failed.

There is another customer and their logs have "No valid firmware for ELAN0000:00 found" error.

The most important thing is that customer confirmed that this issue persists on both of 51 and 52. So I removed "Needs-Feedback" label requested by comment #4.

Here is the customer's environment:

- Chrome OS version. 
Any version of Chrome OS after updating the 49 using a recovery media. 

- Devices make/model 
Chromebook HP 11 G4 

- Hardware ID: KIP C3A******* 

- Domain managed devices? 
No, customer has not completed the process yet. 

- Network information. 
NA 

- Issue description: 
Touch-pad is not working after updating the device to version 49 using the USB recovery method. 

- Steps to reproduce 
Take a Chromebook HP 11 and update the OS from 41 to 49 using a recovery media and the touchpad will stop working. 

- What's the expected behavior? 
Update the device as normal without affecting the touchpad 

- What's the actual result? 
Touch-pad will stop working after the update 

- Timeframe when issue started: 
As soon as the devices were updated to version 49. Issue was reported 3-4 days ago. 

- Does it affect all devices? 
All devices that were updated using the recovery media are affected. 

- Troubleshooting steps taken: 
Tried to wipe the device several times. 
Power washed the device then power wash and revert. 
Tried using different recovery medias. 
Updated device to version 51 and 52. Issue persists. 

- Is this reproducible in Beta/Dev? 
YES 

- Chrome OS debug logs
https://drive.google.com/a/google.com/file/d/0B8hJbBKk0-c4VzFVUWYtRUFDX1E/view?usp=sharing

I think these lines are related to the root cause.

line #487
2016-06-16T14:14:27.933171+00:00 INFO kernel: [    1.962741] input: Elan Touchpad as /devices/platform/80860F41:00/i2c-0/0-0015/input/input5

line #496-#497
2016-06-16T14:14:27.933208+00:00 ERR kernel: [    1.964243] elan_i2c 0-0015: report id [ff] fail.
2016-06-16T14:14:27.933210+00:00 ERR kernel: [    1.964251] elan_i2c 0-0015: wrong packet format.

line #670-#672
2016-06-16T14:14:29.803357+00:00 NOTICE chromeos-elan-touch-firmware-update[749]: Attempting to load FW: '/lib/firmware/elan_i2c.bin'
2016-06-16T14:14:29.805878+00:00 INFO kernel: [    4.833437] EXT4-fs (dm-1): resized filesystem to 795999
2016-06-16T14:14:29.808511+00:00 NOTICE chromeos-elan-touch-firmware-update[753]: error: No valid firmware for ELAN0000:00 found.

So I'm pretty confused by reading through all these comments.  It sounds to me like:

1. Several Kip users see that when they update their device to R49 via USB recovery stick, their touchpads stop working.
2. If they update to it with normal network updates, their touchpads are fine.

Is that correct or am I misreading that?


Either way, looking at the logs from the failed devices, I agree that it looks like they're trying to load a FW that doesn't exist (elan_i2c.bin instead of elan_i2c_75.0.bin or similar).  The updater scripts will try to load this FW (without the product ID appended to it) whenever the one it's expecting to load doesn't exist.  It's a "fallback" solution for devices that were created before we supported multiple FWS -- their FWs were simply named without a product ID.  This allows us to use the same script on the new-style and old-style devices.

So, if your Kip is looking for "elan_i2c.bin" that means it doesn't think there is a FW with the right product ID attached.

You can look up your product ID if you are in developer mode from the command line like this:
  1. Go do /sys/bus/i2c/devices/
  2. Find the device in that directory that has a "driver" symlink pointing to elan_i2c.
     Try using a command like this that will show you which driver each device uses:

     localhost devices # for f in */driver; do echo $f ' -- ' $(readlink "$f"); done
     i2c-10EC5650:00/driver  --  ../../../../../bus/i2c/drivers/rt5645
     i2c-ELAN0000:00/driver  --  ../../../../../bus/i2c/drivers/elan_i2c      <------- This one!
     i2c-MLFS0000:00/driver  --  ../../../../../bus/i2c/drivers/mip4_ts

  3. go into the device directory for the one you found that uses elan_i2c
     cd ./i2c-ELAN0000:00

  4. "cat" the file product_id.  It should print out a number like this:
     localhost i2c-ELAN0000:00 # cat product_id 
     124.0

I have a strong suspicion that you'll find your "product_id" is 255.0 or 0.0.  My best guess is that your touchpad's FW got corrupted somehow and it's product_id is now reporting incorrectly.  As a result the updater script is looking in the wrong place for the FW.

routing to soushi for now.

gkihumba@ - Could you please help to triage this bug?

We have another customer reporting this issue on ver. 57 and ver. 59.

2017-07-10T14:27:16.385489-04:00 NOTICE chromeos-elan-touch-firmware-update[822]: Board detected as 'kip-signed-mp-v3keys'
2017-07-10T14:27:16.419095-04:00 NOTICE chromeos-elan-touch-firmware-update[842]: Touch controller reported Product ID: 65535.0
2017-07-10T14:27:16.427598-04:00 NOTICE chromeos-elan-touch-firmware-update[851]: Attempting to load FW: '/lib/firmware/elan_i2c.bin'
2017-07-10T14:27:16.430382-04:00 NOTICE chromeos-elan-touch-firmware-update[853]: (which points to '/opt/google/touch/firmware/dummy.bin')
2017-07-10T14:27:16.433528-04:00 NOTICE chromeos-elan-touch-firmware-update[855]: error: No valid firmware for ELAN0000:00 found.
2017-07-10T14:27:16.434635-04:00 NOTICE chromeos-touch-update[693]: ELAN0000:00 firmware update failed.

logs for version 59.0.3071.91:
https://drive.google.com/open?id=0By9xEkAUmm_cNExIOUxFaXVHZXM

I can not reproduce on test device running same version of Chrome OS:

2017-07-11T23:56:08.520195+00:00 NOTICE chromeos-elan-touch-firmware-update[895]: Board detected as 'kip-signed-mp-v3keys'
2017-07-11T23:56:08.561839+00:00 NOTICE chromeos-elan-touch-firmware-update[915]: Touch controller reported Product ID: 72.0
2017-07-11T23:56:08.572491+00:00 NOTICE chromeos-elan-touch-firmware-update[924]: Attempting to load FW: '/lib/firmware/elan_i2c_72.0.bin'
2017-07-11T23:56:08.575403+00:00 NOTICE chromeos-elan-touch-firmware-update[926]: (which points to '/opt/google/touch/firmware/72.0_3.0.bin')
2017-07-11T23:56:08.578417+00:00 NOTICE chromeos-elan-touch-firmware-update[928]: Hardware product id : 72.0
2017-07-11T23:56:08.581369+00:00 NOTICE chromeos-elan-touch-firmware-update[930]: Updater product id  : 72.0
2017-07-11T23:56:08.586622+00:00 NOTICE chromeos-elan-touch-firmware-update[934]: Current Firmware: 3.0
2017-07-11T23:56:08.590243+00:00 NOTICE chromeos-elan-touch-firmware-update[936]: Updater Firmware: 3.0
2017-07-11T23:56:08.597254+00:00 NOTICE chromeos-elan-touch-firmware-update[942]: Firmware up to date.

Also, Product ID: 65535.0 looks weird 

Looking at that latest log, I think I know what's going on now.

The product ID "65535.0" almost certainly means that their touch controller's FW got corrupted in some way.  (That number is stored in the firmware of the touch controller itself)

It looks like this is failing because there are already a lot of "exceptions" in the autoupdater for kip and it's not set up for this.  It expects the corrupted FW version to be "255.0" instead of "65535.0" hence why it give up.

I guess maybe the driver got updated to use a 16-bit number instead of an 8-bit number which is causing this issue.

I've written a CL that I think should handle this situation at the following URL:
https://chromium-review.googlesource.com/c/568459

I can't test it today since I don't have a Kip on hand, but I'll try to get it tested and merged in shortly

@Charlie c#17, thanks for helping in this matter. I'd like to know if you were able to test?

Thanks again!.

I had a chance to test out my fix and it appeared to work (the touch updating script correctly identified that the touch FW was corrupted and replaced it with the correct product_id) but the system was still unable to recover.

The script seemed to be working as intended, but I was getting an error message saying that "LSM" was blocking the driver from accessing the FW.

From what I can tell, this is because for Kip (and Kip alone, I believe) we have a convoluted hack to allow us to replace the product_id when they get corrupted.  Basically we set up a dangling symlink at

/lib/firmware/elan_i2c.bin --> /opt/google/touch/firmware/dummy.bin

(The dummy.bin file doesn't exist)

Then we can trick the driver into updating whatever FW we want by mounting the correct FW overtop of dummy.bin so that /lib/firmware/elan_i2c.bin ends up pointing to the right file.

The code for this hack is found here:
https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/master/scripts/chromeos-elan-touch-firmware-update.sh#108

I believe both of these errors are caused by the new kernel, but I had never heard of LSM until I saw it in the error message, so I have no idea how to fix that part of the issue.

My patch is only half of the battle it appears.  Whoever wants to take on the LSM issue can repro it by:
1. Get a Kip w/ a new image on it
2. Apply my CL to the touch_updater package and deploy onto your machine.
3. Manually corrupt the FW on the touchpad by triggering a FW update from the command line and pressing "Esc + refresh + power" before it finishes to cut power midway through.
4. On reboot check /var/log/messages

(Dtor, I assigned this to you since we worked on that hack together and I thought you might know how to work with LSM.  Feel free to reassign if you're not the right person for this.  I am about to go on leave for a few months, too.  Thanks for taking a look)

Dtor, can we remove "merged", as it's merged into 619303 which is fixed and verified, but this bug still exists. Can we go on working with fix on kip?

Charlie, I think at this time we need to resurrect the CL that you had a while ago allowing overriding firmware name in the firmware loader and rely on it instead of doing symlinks dance, as security team is tightening the symlinks traversal.

Hi Charlie, can you please help to move change

https://chromium-review.googlesource.com/#/c/chromiumos/platform/touch_updater/+/568459/

Have customer with numerous affected devices

Andrew, can you please help with the change, while charliemooney is ooo?

Due to impact in case, rased to P1

CR approved, whoever interested, we need to monitor 
https://chromium-review.googlesource.com/#/c/chromiumos/platform/touch_updater/+/568459/
and once we have Landed-in, try that version on affected device.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/touch_updater/+/8b27e13ee4e70d5ab6065a7bf3d12153d941584e

commit 8b27e13ee4e70d5ab6065a7bf3d12153d941584e
Author: Charlie Mooney <charliemooney@google.com>
Date: Tue Aug 22 02:53:01 2017

Add support for 16-bit invalid Elan product IDs

The Elan touch FW auto-updating script has several exceptions and
caviats for fw corruptions that were possible in early versions of their
products.  These are no longer necessary for modern units, but must be
kept around to support old models.  Recently I've seen reports of
the script failing because when the touch controller gets a corrupted
product ID it's reporting it as a 16-bit number of all 1's (65535)
instead of an 8-bit number like it used to (255).  This is almost
certainly due to driver changes and nothing functional in the FW.

This CL just expands the criteria for a "corrupted product_id" to
include this additional value as well as 255.

BUG= chromium:616394 
TEST=build and test on a Kip.  Try corrupting the FW (cut power abruptly
during a touch FW update) and rebooting to make sure it recovers
cleanly.

Change-Id: I198bd98cb05db0eeac30c193581c77b6c560606e
Signed-off-by: Charlie Mooney <charliemooney@google.com>
Reviewed-on: https://chromium-review.googlesource.com/568459
Commit-Ready: Andrew de los Reyes <adlr@chromium.org>
Tested-by: Andrew de los Reyes <adlr@chromium.org>
Reviewed-by: Andrew de los Reyes <adlr@chromium.org>

[modify] https://crrev.com/8b27e13ee4e70d5ab6065a7bf3d12153d941584e/scripts/chromeos-elan-touch-firmware-update.sh

I assume this fix is in the trunk now ?
Is there a test team which can test this before we request merge ?

Re #28: That would be my team.  The devices in the lab have been quiet since this change went in (and it's a minor change and very similar to stuff we've done before), so this is merge-able (pending TPM approval).  My team doesn't have a device in this state.

Royans, are there some affected users who would be willing to put their devices on Dev or Beta channels?

This bug requires manual review: We are only 5 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), ketakid@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Hi team,

I've been trying to contact my customer with no success. I already sent them the Chrome image they need to install but I haven't receive any updates from their end. I'll try to contact them again tomorrow morning.

Approving merge to M61.

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

This issue hasn't been updated in the last 6 weeks, so removing its merge approval label. Please re-request a merge if needed.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Hello!
This bug is receiving this notice because there has been no acknowledgment of its existence in quite a bit of time
- If you are currently working on this bug, please provide an update.
- If you are currently affected by this bug, please update with your current symptoms and relevant logs.

If there has been no updates provided by EOD Wednesday, 12/12/18 (5pm EST), this bug will be archived and can be re-opened at any time deemed necessary.

Thank you!

Due to lack of action this bug has been Archived. If work is still being done on this issue or you are still experiencing this issue please feel free to re-open with the appropriate information.