Issue metadata
Sign in to add a comment
|
Device state preserving TPM firmware update behaving erratically after interrupted installation |
||||||||||||||||||||||
Issue descriptionRepro: 1. Trigger device state preserving firmware update via chrome://chrome, confirm powerwash dialog after reboot. 2. Interrupt update installation on the progress bar screen 3. Device will reboot into recovery due to inoperable TPM (in bootloader mode) 4. Supply recovery media and let the firmware update retry complete successfully. 5. Reboot in normal mode. Login screen comes up with the previously present user pods visible. 6. Attempt login. Expected result: Failure to login, online login flow gets launched after typing password once. Actual result: Login error dialog. Repeated attempts eventually trigger the cryptohome wipe flow with online login and feedback report prompt. Log tarball attached.
,
Jun 20 2018
Regarding M-68, we need to make a call whether we want to ship with this, fix it for M68, or disable the feature in M68 and punt to M69 to buy time more time for a fix. My current thinking is that we can't ship in this state unfortunately, because the wedged cryptohome state we end up in will prevent user credentials from being bound to the TPM. That brings up the question on whether there's a way to fix M68. I'll need to think some more about this. If I can't figure this out within a day or two, we should probably pull the feature from M68 (which is a one-line CL in Chrome). Marking ReleaseBlock-Stable to be sure we don't accidentally ship this.
,
Jun 20 2018
Here's one thing that I'm not quite sure about: mount-encrypted should have nuked cryptohome state on disk on the first stateful preservation that takes place when rebooting into the installation flow in normal mode. However, cryptohomed's behavior after recovery suggests that it finds state on disk and thinks it already has set up the TPM (SRK, etc.). The best explanation I can come up with right now is that the forced reboot caused the deletion of the files (here: https://chromium.git.corp.google.com/chromiumos/platform2/+/8ae6dcca2b7464a828d7601b5e844eb953b1d52e/cryptohome/mount_encrypted/tpm1.cc#320 ) not to stick on disk (we've seen file writes not surviving hard shutdowns previously). I'll experiment a bit to see whether I can confirm. Anyhow, if the hypothesis in comment #1 is correct (adding apronin@ for a second set of eyes), then I think we'll want to do the following: 1. Force another TPM clear after recovery. The clear operation in recovery will not work due to the TPM being in bootloader mode. I think our best option is to do a crossystem clear_tpm_owner_request=1 also in recovery mode. This is a simple fix. 2. Adjust crypthomed to detect and deal with the situation of the TPM being owned but not initialized properly without relying on file system state. This is more involved (cryptohomed startup logic is convoluted...), so might take longer. For M-68, #1 might be sufficient. So I'll dive into that and report back once I know whether it works out.
,
Jun 20 2018
Regarding the loss of clear_tpm_owner_request across recovery: It looks like older firmware branches do clear the request flag regardless of whether the TPM clear was successful (e.g. when the TPM is in bootloader mode): https://chromium.git.corp.google.com/chromiumos/platform/vboot_reference/+/3d6a7e4ccd3ef706fe8296e18704a3f05093fa60/firmware/lib/vboot_api_init.c#245 So the hypothesis in comment #1 still stands.
,
Jun 20 2018
Here's a CL for the TPM-clear-after-recovery aspect: https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1108117 Let's see what Andrey says before deciding whether it makes sense to try and get this into M68. I'll also need help to verify this with an interrupted life update (I'm unfortunately out of non-updated devices).
,
Jun 20 2018
Makes sense. If the update was interrupted and the tpm entered bootloader mode, clearing the owner would fail. And the request itself would be cleared as shown in #4 (which now also seems to be the cause for issue 778332). I'd say that even if it was not guaranteed to be lost, requesting it again after bootloader mode would make sense. Note that it's still possible for something unrelated to the tpm to go wrong and device rebooting into recovery again with uncleared owner and lost flag. An additional safeguard is a post-update check for old SRK (or just plain post-update reboot-to-clear step, which may also fail, though) - issue 778332 and CLs mentioned there. But CL:1108117 already covers a large set of cases.
,
Jun 20 2018
Thought about this some more: Just patching the recovery code path to request a TPM reset is somewhat risky. If someone uses an old recovery image, they'd not get a TPM clear and still end up in the bad situation. We could prevent existing recovery images from retrying the update by changing the "image" part of the tpm_firmware_update_params VPD key so existing images fail here: https://chromium.git.corp.google.com/chromiumos/overlays/chromiumos-overlay/+/bdaa2d51f894c9ba8526badc779c739f83f366bd/chromeos-base/infineon-firmware-updater/files/tpm-firmware-updater#120 That'd force people to use a recovery image that has the TPM clear request added by CL:1108117. A more robust solution would be to check whether we need another TPM clear after we're back in normal mode though, and as Andrey mentions this would also be useful for issue 778332. We can check whether the TPM is unowned (all good then) or if it's owned whether the SRK is vulnerable (handling only well-known password case should be sufficient as all code paths that trigger firmware updates configure the well-known password if any). The one thing I'm not clear on is how to correctly weave this into the startup logic so it doesn't have to run on every boot. I'll think some more about that.
,
Jun 22 2018
The NextAction date has arrived: 2018-06-22
,
Jun 22 2018
I've thought some more and haven't gotten to a point where I have a good plan of attack. Thus, I've created a CL to disable the feature for now: https://chromium-review.googlesource.com/c/chromium/src/+/1111998 Intention is to merge this to 68, figure out the proper fix, then reenable.
,
Jun 22 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/1f9212ce3222e59aa221cccb97b2ce709c5614da commit 1f9212ce3222e59aa221cccb97b2ce709c5614da Author: Mattias Nissler <mnissler@chromium.org> Date: Fri Jun 22 15:21:27 2018 infineon-firmware-updater: Request TPM clear in recovery mode. When handling a TPM firmware update retry in recovery mode, the existing code that clears the TPM is known to fail because the TPM won't handle the respective commands in bootloader mode. It's still possible a TPM owner is present, and it'll survive if the retry completes successfully. As a result, the SRK from before the update would be carried over, which we must not allow. To address this, request the firmware to preform a TPM clear unconditionally before starting the update so we have reasonable confidence the TPM will be in a good state after update installation and reboot. BUG= chromium:854576 TEST=Manual, see bug. Change-Id: Ief6e065ac7b2e5c987abfc335e13823c4a21702c Reviewed-on: https://chromium-review.googlesource.com/1108117 Commit-Ready: Mattias Nissler <mnissler@chromium.org> Tested-by: Mattias Nissler <mnissler@chromium.org> Reviewed-by: Andrey Pronin <apronin@chromium.org> [rename] https://crrev.com/1f9212ce3222e59aa221cccb97b2ce709c5614da/chromeos-base/infineon-firmware-updater/infineon-firmware-updater-1.1.2459.0-r26.ebuild [modify] https://crrev.com/1f9212ce3222e59aa221cccb97b2ce709c5614da/chromeos-base/infineon-firmware-updater/files/tpm-firmware-updater
,
Jun 25 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bcbd174038035d8c59e45a067b8bea0c16168b40 commit bcbd174038035d8c59e45a067b8bea0c16168b40 Author: Mattias Nissler <mnissler@chromium.org> Date: Mon Jun 25 13:04:47 2018 Disable state-preserving TPM firmware update. The test team found an issue where cryptohomed fails to deal with the state the TPM is in for the case when the update installation gets interrupted and recovery is invoked to retry and complete the update. Disable device-state preserving updates for now until the issue is resolved. BUG= chromium:854576 TEST=Triggering TPM firmware update via chrome://chrome will not invoke the device state preserving update flow. Change-Id: I4ac4d325bfbf953cfa14a501e871a3f9f59c699c Reviewed-on: https://chromium-review.googlesource.com/1111998 Reviewed-by: Steven Bennetts <stevenjb@chromium.org> Commit-Queue: Mattias Nissler <mnissler@chromium.org> Cr-Commit-Position: refs/heads/master@{#570020} [modify] https://crrev.com/bcbd174038035d8c59e45a067b8bea0c16168b40/chrome/browser/ui/webui/settings/browser_lifetime_handler.cc
,
Jun 25 2018
Filing merge request for 68 for the CL to disable the feature: https://chromium-review.googlesource.com/1111998
,
Jun 25 2018
> A more robust solution would be to check whether we need another TPM clear after we're back in normal mode though, and as Andrey mentions this would also be useful for issue 778332. We can check whether the TPM is unowned (all good then) or if it's owned whether the SRK is vulnerable ... Btw, the SRK-check works (and is generally useful for other reasons) for the current update. But if we ever have another fix in the future, that won't catch an incomplete update. To address a general case, I suspect we need to have a special post-upgrade cleanup state, something like: - set this state before starting the update; - on boot, if we are in this state and TPM is owned, request clearing the owner and reboot; I'm not quite sure if we also need a selective data wipe to always go with that (in case this state is faked). - otherwise, switch the state to 'all-done' and continue.
,
Jun 25 2018
sontis@ and pgangishetty@, please confirm no state-preserving TPM FW update is availbale once the build is available(M69 and M68) on GE.
,
Jun 25 2018
mnissler@, should the launch bug issue 846111 be moved to M69?
,
Jun 25 2018
,
Jun 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/927ddd8d02bf73241260f44c529365a4e34f69e3 commit 927ddd8d02bf73241260f44c529365a4e34f69e3 Author: Mattias Nissler <mnissler@chromium.org> Date: Wed Jun 27 13:12:02 2018 Disable state-preserving TPM firmware update. The test team found an issue where cryptohomed fails to deal with the state the TPM is in for the case when the update installation gets interrupted and recovery is invoked to retry and complete the update. Disable device-state preserving updates for now until the issue is resolved. BUG= chromium:854576 TEST=Triggering TPM firmware update via chrome://chrome will not invoke the device state preserving update flow. Change-Id: I4ac4d325bfbf953cfa14a501e871a3f9f59c699c Reviewed-on: https://chromium-review.googlesource.com/1111998 Reviewed-by: Steven Bennetts <stevenjb@chromium.org> Commit-Queue: Mattias Nissler <mnissler@chromium.org> Cr-Original-Commit-Position: refs/heads/master@{#570020}(cherry picked from commit bcbd174038035d8c59e45a067b8bea0c16168b40) Reviewed-on: https://chromium-review.googlesource.com/1116998 Reviewed-by: Mattias Nissler <mnissler@chromium.org> Cr-Commit-Position: refs/branch-heads/3440@{#550} Cr-Branched-From: 010ddcfda246975d194964ccf20038ebbdec6084-refs/heads/master@{#561733} [modify] https://crrev.com/927ddd8d02bf73241260f44c529365a4e34f69e3/chrome/browser/ui/webui/settings/browser_lifetime_handler.cc
,
Jun 27 2018
Feature disabled on 68 per https://chromium.googlesource.com/chromium/src/+/927ddd8d02bf73241260f44c529365a4e34f69e3 and launch bug punted to M69.
,
Jul 5
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/ae52c33eeb4dfc2b04d9db62935fb19719629bb5 commit ae52c33eeb4dfc2b04d9db62935fb19719629bb5 Author: Mattias Nissler <mnissler@chromium.org> Date: Thu Jul 05 22:14:37 2018 infineon-firmware-updater: Force TPM clear after update. We need to make sure the TPM doesn't hold on to an SRK that originates from before the TPM firmware update because that SRK might be weak. This can happen in edge cases, for example when we start the TPM firmware update using owner authorization but then get interrupted and perform a retry via recovery mode. This change adds post-update logic in the tpm-firmware-update.sh script which runs during boot when a TPM firmware update has been requested. It makes sure that we only continue booting if the TPM is clear and requests the TPM to be cleared if necessary. BUG= chromium:854576 , chromium:778332, chromium:859511 TEST=See bug. Change-Id: I091faf438e34c7c05944df1d26840367bc560fb7 Reviewed-on: https://chromium-review.googlesource.com/1123830 Commit-Ready: Mattias Nissler <mnissler@chromium.org> Tested-by: Mattias Nissler <mnissler@chromium.org> Reviewed-by: Mattias Nissler <mnissler@chromium.org> [rename] https://crrev.com/ae52c33eeb4dfc2b04d9db62935fb19719629bb5/chromeos-base/infineon-firmware-updater/infineon-firmware-updater-1.1.2459.0-r27.ebuild [modify] https://crrev.com/ae52c33eeb4dfc2b04d9db62935fb19719629bb5/chromeos-base/infineon-firmware-updater/files/tpm-firmware-update.sh
,
Jul 16
Per comment 18, moving the milestone label to 69.
,
Jul 17
is this fixed ? If so can status be changed to Fixed?
,
Jul 18
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/1fe15bb74804db1a26905f213cd96cec27278e53 commit 1fe15bb74804db1a26905f213cd96cec27278e53 Author: Mattias Nissler <mnissler@chromium.org> Date: Wed Jul 18 01:30:26 2018 infineon-firmware-updater: Force TPM clear after update. We need to make sure the TPM doesn't hold on to an SRK that originates from before the TPM firmware update because that SRK might be weak. This can happen in edge cases, for example when we start the TPM firmware update using owner authorization but then get interrupted and perform a retry via recovery mode. This change adds post-update logic in the tpm-firmware-update.sh script which runs during boot when a TPM firmware update has been requested. It makes sure that we only continue booting if the TPM is clear and requests the TPM to be cleared if necessary. BUG= chromium:854576 , chromium:778332, chromium:859511 TEST=See bug. Change-Id: I091faf438e34c7c05944df1d26840367bc560fb7 Reviewed-on: https://chromium-review.googlesource.com/1123830 Commit-Ready: Mattias Nissler <mnissler@chromium.org> Tested-by: Mattias Nissler <mnissler@chromium.org> Reviewed-by: Mattias Nissler <mnissler@chromium.org> (cherry picked from commit ae52c33eeb4dfc2b04d9db62935fb19719629bb5) Reviewed-on: https://chromium-review.googlesource.com/1134158 Reviewed-by: Andrey Pronin <apronin@chromium.org> Tested-by: Andrey Pronin <apronin@chromium.org> Commit-Queue: Andrey Pronin <apronin@chromium.org> [modify] https://crrev.com/1fe15bb74804db1a26905f213cd96cec27278e53/chromeos-base/infineon-firmware-updater/files/tpm-firmware-update.sh [rename] https://crrev.com/1fe15bb74804db1a26905f213cd96cec27278e53/chromeos-base/infineon-firmware-updater/infineon-firmware-updater-1.1.2459.0-r26.ebuild
,
Jul 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3dad7974c39b645395a82388190469c3f9027669 commit 3dad7974c39b645395a82388190469c3f9027669 Author: Mattias Nissler <mnissler@chromium.org> Date: Fri Jul 27 09:14:20 2018 Revert "Disable state-preserving TPM firmware update." This reverts commit bcbd174038035d8c59e45a067b8bea0c16168b40. Reason for revert: The bug that prompted disabling of the state-preserving firmware update feature has been fixed per https://chromium-review.googlesource.com/1123830 so we can re-enable the feature. Original change's description: > Disable state-preserving TPM firmware update. > > The test team found an issue where cryptohomed fails to deal with the > state the TPM is in for the case when the update installation gets > interrupted and recovery is invoked to retry and complete the update. > Disable device-state preserving updates for now until the issue is > resolved. > > BUG= chromium:854576 > TEST=Triggering TPM firmware update via chrome://chrome will not invoke the device state preserving update flow. > > Change-Id: I4ac4d325bfbf953cfa14a501e871a3f9f59c699c > Reviewed-on: https://chromium-review.googlesource.com/1111998 > Reviewed-by: Steven Bennetts <stevenjb@chromium.org> > Commit-Queue: Mattias Nissler <mnissler@chromium.org> > Cr-Commit-Position: refs/heads/master@{#570020} TBR=stevenjb@chromium.org,mnissler@chromium.org Bug: chromium:854576 Change-Id: Iab59dc64130792b92071d45b82a77ddbd4553aa4 Reviewed-on: https://chromium-review.googlesource.com/1127539 Reviewed-by: Mattias Nissler <mnissler@chromium.org> Commit-Queue: Mattias Nissler <mnissler@chromium.org> Cr-Commit-Position: refs/heads/master@{#578570} [modify] https://crrev.com/3dad7974c39b645395a82388190469c3f9027669/chrome/browser/ui/webui/settings/browser_lifetime_handler.cc
,
Jul 27
This is fixed per "infineon-firmware-updater: Force TPM clear after update." which has been merged to M68. The state-preserving flow has only been reenabled for M70 due to vacation. Sorry for not updating the bug earlier.
,
Jul 30
Tried on Peppy and Skate with build version 70.0.3505.0/10922.0.0 and at step #3 we see login screen instead of recovery screen. 1. Trigger device state preserving firmware update via chrome://chrome, confirm powerwash dialog after reboot 2. Interrupt update installation on the progress bar screen 3. Device rebooted and we see login screen |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by mnissler@chromium.org
, Jun 20 2018