What do you mean by "Remove the user"? Removing it from the login screen?

rsorokin@ Sorry I wasn't clear in the descriptions.  Yes, in the sign-in screen, select the "Remove this user" from the user pod's drop down list.

+alemate@ as this is related to username change

Just to clarify, you're saying that username@newdomain.com is failing, correct? Because username@olddomain.com should not work, based on my understanding (unless the old email address becomes an alias for the new email address, but even then it might not work)

Re #7, yes username@newdomain.com is failing.

Prior to removing username@olddomain.com, I added and signed in username@newdomain.com (while there still exists user pod of username@olddomain.com on the sign-in screen).

In policy page, I still saw the username@olddomain.com being the user policy.

When I signed out, I still saw only the user pod of username@olddomain.com and does not have a user pod for username@newdomain.com.

So I removed the old user pod of username@olddomain.com and then added username@newdomain.com.

I had also checked in VT2 /home/chronos/user/ that the directory for username@olddomain.com was gone after the user pod removal.

Thanks, this sounds like it is related to  crbug.com/727992 , but this bug identifies that the previous workaround is broken as well

+ policy owners.

The problem is this:
[29360:29488:1027/190138.011170:ERROR:cloud_policy_validator.cc(279)] Signature verification failed
[29360:29488:1027/190138.013067:ERROR:cloud_policy_validator.cc(374)] Initial policy root signature validation failed
[29360:29360:1027/190138.013748:ERROR:user_cloud_policy_manager_chromeos.cc(462)] Policy fetch failed for the user. Aborting profile initialization

So basically, when we create new user for renamed domain, user policy verification fails.

Assigning to policy owners for triage.

I was able to reproduce this issue with simply renaming the user and then removing the automatically created username alias for the old username.

1. Created usernameA@domain.com
2. Rather than rename the domain, follow the steps here to rename the username to usernameB@domain.com: https://support.google.com/a/answer/182084
3. Post-rename, usernameB@domain.com has a username alias (i.e. redirect) of usernameA@domain.com. 

At this point I can still log into the user via user pod.

4. Remove usernameA@domain.com as an alias of usernameB@domain.com. 

At this point I can still log into the cached user pod using the old username.

5. Remove usernameA@domain.com user pod
6. Add usernameB@domain.com as a new user from the login screen

I get the same error as the originally posted screenshot.

This is a big concern, because that means G Suite customers can encounter this if they perform a user rename. It doesn't seem like there is a workaround.

re: comment 11 - not clear whether "i can still log into the cached user" means you're doing an online Gaia signin or just typing your password into the device to unlock the cryptohome (no Gaia interaction).

Just to put a stake in the ground:

1) If you rename your user, then try to sign in to an existing user pod, the behavior is undefined. It may work, it may not work, depending on lots of things (whether you are doing an online signin, whether the domain was changed, etc). We won't investigate any bugs about signing in to existing user pods after a rename.

2) If you rename your user, then delete the old pod, you should be able to sign in to the device. If you can't, this is a bug.

I am seeing reports in this bug about both cases - Maksim, please only investigate #2, since we don't officially support #1.

Was able to reproduce the the issue in the "delete-rename-try-to-add" scenario (i.e. #2 from comment 12). From what I've found so far, the profile initialization is aborted due to the system still thinking that the user should have the old e-mail:

  [ERROR:cloud_policy_validator.cc(475)] Invalid user name usernameB@domain.com
  [ERROR:user_cloud_policy_manager_chromeos.cc(464)] Policy fetch failed for the user. Aborting profile initialization

Also, the login screen still displays "usernameA" for me, even after I delete the user pod and re-add it with typing "usernameB@domain.com".

I'll check from where is the old e-mail coming...

Exactly - if CloudPolicyValidator is failing, it's because it's comparing vs existing policy, which shouldn't happen if you've deleted the user cryptohome.

After more investigation, I discovered that removal of the user's pod did not clear all references to it in the Local State. In particular, the user's entry is still preserved as an item under "KnownUsers", with the user's GAIA ID and e-mail:

  // Local State after usernameA@domain.com was removed at the login screen:
  "KnownUsers": [
    <...>,
    {
      "account_type": "google",
      "email": "usernameA@domain.com",
      "gaia_id": "123456789012345678901",
      "wallpaper-files-id": "<...>"
    }
  ]

After I try to add a new user usernameB@domain.com, I see from the Local State clearly that this leftover record gets reused for the new e-mail:

  "KnownUsers": [
    <...>,
    {
      "PasswordTokenHandle": "<...>",
      "TokenHandleStatus": "valid",
      "account_type": "google",
      "arc": <...>,
      "device_id": "<...>",
      "email": "usernameA@domain.com",
      "gaia_id": "123456789012345678901",
      "gaps_cookie": "<...>",
      "profile_ever_initialized": false,
      "wallpaper-files-id": "<...>"
    }
  ]
  "UserDisplayEmail": {
    <...>,
    "usernameA@domain.com": "usernameB@domain.com"
  }
  "UserDisplayName": {
    <...>,
    "usernameA@domain.com": "usernameA"
  }

I think this reuse happens inside GaiaScreenHandler::HandleCompleteAuthentication(), as it goes down to user_manager::known_user::GetAccountId(), which returns an existing entry as long as there's a match in GAIA ID, regardless of the e-mails match. After that point, the obtained AccountId begins pointing to the "usernameA@domain.com" e-mail, which leads to all sorts of problems subsequently.

alemate@: Assigning back to you as this seems to be an issue with the sign-in logic.
I think there are two possible ways of fixing this - either by making the cleanup during the user removal more complete, or by overwriting the e-mail in the Local State during sign-in.

alemate@, do you have any updates regarding this issue?

alemate/rkc - can you please reassign this bug if the current owner cannot handle this. At the very least can you ACK the bug, let us know what else info you need or set ETA on when this could be resolved ?

A P0 priority without a milestone is, confusing.
Moving back down to P1 - we should be able to get this into M65.

Spoke with TPM on this bug; not blocking beta releases, but need to request merge to 64 after it is resolved.

Spoke with alemate@; reassigning as he is working on 789048

poromov@, any update? Can we expect it to be fixed in M64?

Most of our team is OOO this week, will be back mid-next week.

poromov@
sorry for keep asking,
Is there any update that can be shared with the customer?
Can we expect it to be fixed in M64?

Hey, sorry, I'm just back from a vacation and starting working on this.
No expectations about M64 yet.

We're like a week out from our stable cut - seems unlikely to get into 64.

The bug from Comment #15 is pretty simple:
After known_user::RemovePrefs(<accountId>) is called, a call to known_user::SetStringPref(<accountId>, "wallpaper-files-id ", ...) happens that re-creates entry for the user being deleted.

The code path that calls both functions is:

ChromeUserManagerImpl::RemoveNonCryptohomeData() -> ChromeUserManager::RemoveNonCryptohomeData() -> known_user::RemovePrefs()

ChromeUserManagerImpl::RemoveNonCryptohomeData() -> WallpaperControllerClient::Get()->RemoveUserWallpaper() -> WallpaperControllerClient::GetFilesId() -> user_manager::known_user::SetStringPref()

The problems is that WallpaperControllerClient::GetFilesId() stores (caches) the wallpaper file id in local state prefs regardless of whether the user is already deleted or not.

I'm not sure whether it's a regression or not as there was a large refactoring in wallpaper code in last months, but for sure it's a bug.

I'm reassigning this to Wenzhao, who made most of the changes in wallpaper code recently. Please, try to fix it while I'm OOO till Monday.

I think the safest fix at this point is to switch line 1027 and 1029 here and add a comment: https://cs.chromium.org/chromium/src/chrome/browser/chromeos/login/users/chrome_user_manager_impl.cc?type=cs&q=ChromeUserManager::RemoveNonCryptohomeData&sq=package:chromium&l=1027

(This is not a recent regression. The refactoring didn't touch this part.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dd0dde6efa37cdcba862a0a6587c6c744dba206d

commit dd0dde6efa37cdcba862a0a6587c6c744dba206d
Author: Wenzhao Zang <wzang@chromium.org>
Date: Sat Jan 13 08:57:30 2018

cros: Postpone removing user prefs in |RemoveNonCryptohomeData|

Bug:  778077 
Change-Id: Id68b2f08d151b01f9c1f090513b1bae7fa7ea741
Reviewed-on: https://chromium-review.googlesource.com/862710
Commit-Queue: Wenzhao (Colin) Zang <wzang@chromium.org>
Reviewed-by: Alexander Alekseev <alemate@chromium.org>
Cr-Commit-Position: refs/heads/master@{#529172}
[modify] https://crrev.com/dd0dde6efa37cdcba862a0a6587c6c744dba206d/chrome/browser/chromeos/login/users/chrome_user_manager_impl.cc
[modify] https://crrev.com/dd0dde6efa37cdcba862a0a6587c6c744dba206d/components/user_manager/user_manager_base.cc

This bug requires manual review: We are only 9 days from stable.
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), kbleicher@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Checked that the  issue #2  from Comment #12 is fixed on ToT build:

If user is removed from a device and renamed - it could be successfully added and logged in on the same device using the new user name.

Wenzhao, please proceed with merging the fix into M64.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d24171ccb98934353232ba5b894cf92902a696be

commit d24171ccb98934353232ba5b894cf92902a696be
Author: Wenzhao Zang <wzang@chromium.org>
Date: Wed Jan 17 19:43:16 2018

cros: Postpone removing user prefs in |RemoveNonCryptohomeData|

TBR=wzang@chromium.org

(cherry picked from commit dd0dde6efa37cdcba862a0a6587c6c744dba206d)

Bug:  778077 
Change-Id: Id68b2f08d151b01f9c1f090513b1bae7fa7ea741
Reviewed-on: https://chromium-review.googlesource.com/862710
Commit-Queue: Wenzhao (Colin) Zang <wzang@chromium.org>
Reviewed-by: Alexander Alekseev <alemate@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#529172}
Reviewed-on: https://chromium-review.googlesource.com/871180
Reviewed-by: Wenzhao (Colin) Zang <wzang@chromium.org>
Cr-Commit-Position: refs/branch-heads/3282@{#527}
Cr-Branched-From: 5fdc0fab22ce7efd32532ee989b223fa12f8171e-refs/heads/master@{#520840}
[modify] https://crrev.com/d24171ccb98934353232ba5b894cf92902a696be/chrome/browser/chromeos/login/users/chrome_user_manager_impl.cc
[modify] https://crrev.com/d24171ccb98934353232ba5b894cf92902a696be/components/user_manager/user_manager_base.cc

As verified in M64.0.3282.122 10176.61.0 beta candy/paine, this issue has been fixed and is no longer reproduced.

As observed in my test, after the domain was renamed, chrome://policy showed the user policy was invalid with user@old_domain. After signed out and signed in with user@new_domain, chrome://policy still showed the policy was invalid with user@old_domain (even though the current user was user@new_domain). The same result after rebooting the device. Removed the user pod in the sign-in screen. Successfully signed in with user@new_domain. No more error in chrome://policy was observed and user policy was updated with user@new_domain.