We found out amd64-generic build with 4.14 kernel cannot log in on any of our PCs. We have tested 4.14.3 from R64 and 4.14.12/4.14.13 from master, none of them worked. The same amd64-generic build with 4.4 or 4.12 kernel don't have the login issue.

We checked /var/log/messages and noticed that there are error messages from cryptohomd, like below:

2018-01-14T13:03:44.314899+00:00 ERR cryptohomed[991]: stat() of /home/.shadow/53bf329c613c51252d9eac53bdc09d11a93858c7/mount/user/Downloads failed.: No such file or directory
2018-01-14T13:03:44.314926+00:00 ERR cryptohomed[991]: Couldn't set up group access on directory: /home/.shadow/53bf329c613c51252d9eac53bdc09d11a93858c7/mount/user/Downloads

After adding some more logging code to cryptohome/mount.cc, we can confirm that the error happens in Mount::MountCryptohomeInner(). After CopySkeleton() finished successfully, the SetupGroupAccess() call failed and spit out above errors.

We are still looking into this and trying to find the root cause. We would appreciate if chromeos team could take a look and fix the problem.

#1: Do you see any tpm error messages in the boot log ?

Yes I did see a lot of TPM error messages. However similar TPM error messages was there when running 4.12 kernel, but it does not have the login issue. So I'm inclined to believe this problem is not TPM related.

I've attached messages log from 4.12 and 4.14 kernel.

Deleted: messages-4.12 183 KB

messages-4.12 183 KB Download

Deleted: messages-4.14 194 KB

messages-4.14 194 KB Download

#3: Do I understand correctly that this is a Lenovo Thinkpad X1, and this it is not equipped with a TPM chip ?

This is just for my understanding; I don't question the existence of the bug. It just means that there are really two bugs, one related to TPM and the other to behavior if no TPM is present.

Hi i'll let Kai respond with his findings but just wanted to add that the
issue also presents on the Dell 11 chromebook (wolf) which also is unable
to login.
We can test and supply more logs for chromebooks if required which should
help narrow down any TPM related issues.

The first reported error is:

2018-01-14T13:03:27.913573+00:00 ERR cryptohomed[991]: Couldn't change owner (1000:1000) of destination path: /home/.shadow/53bf329c613c51252d9eac53bdc09d11a93858c7/mount/user/.bash_logout
2018-01-14T13:03:27.913604+00:00 ERR cryptohomed[991]: Couldn't change owner (1000:1000) of destination path: /home/.shadow/53bf329c613c51252d9eac53bdc09d11a93858c7/mount/user/.bashrc
2018-01-14T13:03:27.913642+00:00 ERR cryptohomed[991]: Couldn't change owner (1000:1000) of destination path: /home/.shadow/53bf329c613c51252d9eac53bdc09d11a93858c7/mount/user/.bash_profile

which is just an indication that the path does not exist, ie that some mount failed. It looks like this happens if there is no TPM or the TPM is not instantiated/detected (login works if a TPM is detected and working). No idea yet why the mount fails.

For the Dell chromebook: Please remove "tpm_tis.force=1" from the kernel command line and check if the problem persists.

To clarify #7: You'll see the problem on all Chromebooks which are officially running chromeos-3.8. Primary reason is that the TPM code was changed and no longer works if "tpm_tis.force=1" is on the command line. Secondary reason is that, without TPM, the system fails to come up, and user login fails. I have not been able to track down this second problem.
There is also a variant of the 1st problem, where login fails even with "tpm_tis.force=1" removed. It is possible that the TPM is still not detected or working properly on some chromebooks, and that we hit the second problem again,
but that is just a guess.

#4: yes this is Thinkpad X1 Carbon 1st Gen. This device has a TPM 1.2 device but was disabled in BIOS in this testing. In fact enabling the TPM does not help the login issue.

#7: I can confirm that the tpm_tis.force=1 parameter was not in the amd64-generic build since the beginning. I've also tried manually specifying tpm_tis.force=1 or 0 during boot, didn't change anything on 4.14.


Here are more details and updates regarding why I'm inclined to believe the issue we encountered is not TPM related.

1. No matter enabling or disabling the TPM device on the TP X1C device, the issue persists on 4.14 but never happens on 4.12.

2. I've reverted all TPM related changes between kernel 4.12.14 and 4.14.13, found by command `git log v4.12.14..v4.14.13 --follow drivers/char/tpm`, except two commits, still cannot login. The two commits I didn't revert are b24413180f5600bcb3bb70fbed5cf186b60864bd which just added license information in comments so not relevant, and 94116f8126de9762751fd92731581b73b56292e5 which changed more than just TPM so reverting it requires reverting some other commits.

3. As mentioned in above comments, on the 4.12 kernel there were almost the same TPM error messages from cryptohomed but there was no login problem.

4. As we can see the log, what directly caused login issue is that cryptohome failed to operate on files under /home/.shadow/53bf329c613c51252d9eac53bdc09d11a93858c7/mount/user, which to my understanding is bind mounted from /home/.shadow/53bf329c613c51252d9eac53bdc09d11a93858c7/vault, that was created in earlier calls in crytohome. So I'm guessing some bugs in the ecryptfs or ext4fs or vfs caused this problem.


And yes I do agree that this seems is a different bug than the original one. Shall I re-open a new bug?

#9: Thanks a lot for the details. Let's keep it together in this bug for now; we can file a new bug anytime if necessary.

kraml@, will@: Can you try to revert 88ae4ab9802eaa and check if it makes a difference for you ? Turns out we had that patch reverted in chromeos-4.12, but not in chromeos-4.14. Also see  crbug.com/680333 .

Can login with 4.14 kernel now after reverted 88ae4ab9802eaa. Thanks for pointing that out!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/295b17d7562b9c9ccc7b743b46e18a6a3ba156b9

commit 295b17d7562b9c9ccc7b743b46e18a6a3ba156b9
Author: Guenter Roeck <groeck@chromium.org>
Date: Thu Jan 18 13:09:39 2018

CHROMIUM: Revert "ecryptfs_lookup(): try either only encrypted or plaintext name"

This reverts commit 88ae4ab9802eaa8e400e611f85883143d89d6b61.

Commit 88ae4ab9802e ("ecryptfs_lookup(): try either only encrypted or
plaintext name") introduces a regression preventing logins. Revert it
until the problem has been resolved.

BUG= chromium:680333 ,  chromium:793353 
TEST=Try to log in on graphics screen

Change-Id: I93449548c5e7ab61bc0d9adf842e346610a69abf
Signed-off-by: Guenter Roeck <groeck@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/869771
Commit-Ready: Gwendal Grignou <gwendal@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>

[modify] https://crrev.com/295b17d7562b9c9ccc7b743b46e18a6a3ba156b9/fs/ecryptfs/inode.c

Upstream patch has been rejected by maintainer. Consider this fixed as the revert of the offending patch has landed in chromeos-4.14. No further action planned.