Monorail Project: gerrit Issues People Development process History Sign in
New issue
Advanced search Search tips
Issue 5090 Some users unable to login on 2.13.x (with LDAP?) after upgrading from 2.12.x, "name already in use"
Starred by 7 users Project Member Reported by choro...@wikimedia.org, Dec 8 Back to list
Status: Submitted
Owner: ----
Closed: Dec 9



Sign in to add a comment
Affected Version: 2.13.3

What steps will reproduce the problem?
1. Upgrade from 2.12.5
2. Witness that some users cannot login with an error

What is the expected output?
User is able to log in just fine

What do you see instead?
Cannot assign user name "awight" to account 4189; name already in use.

Please provide any additional information below.
We upgraded today from 2.12.5 to 2.13.3. We went offline for a full reindex, including accounts. After coming back up, some users reported the issue mentioned above. One user was able to work around it by changing the capitalization of his username to match what was in account_external_ids but others had no such luck with the workaround.

Deleting the offending account & account_external_id rows allowed them to log in. However, reassigning them their old account_id back again made the issue reappear. This means I could likely work around it by reassigning everything in changes, comments, etc etc tables to their new account_id, but this obviously wouldn't scale beyond a couple of affected accounts (currently unknown as to how many, approx 4000k total accounts).

Using standard LDAP, users identify and are displayed from their CN, and their shell/http username for git operations is from their SN.
 
Project Member Comment 1 by choro...@wikimedia.org, Dec 8
Labels: -Priority-3 Priority-1
Project Member Comment 2 by choro...@wikimedia.org, Dec 8
So let's compare these two account_external_ids queries:

gerrit> select * from account_external_ids where account_id = 2;
 account_id | email_address       | password     | external_id
 -----------+---------------------+--------------+---------------
 2          | chadh@wikimedia.org | NULL         | gerrit:Chad
 2          | NULL                | <redacted>   | username:demon
(2 rows; 7 ms)

gerrit> select * from account_external_ids where account_id = 143;
 account_id | email_address | password | external_id
 -----------+---------------+----------+----------------
 143        | NULL          | NULL     | username:awight
(1 row; 1 ms)

Interesting, the broken user only has 1 row! So out of curiosity, I created a second entry for the busted user 143 into account_external_ids to mirror my entry with 2 rows. However, upon trying to log in Gerrit happily deleted the new second entry I added!
More digging reveals this https://github.com/gerrit-review/gerrit/commit/94732bfa1945a02bd4441b70a501cfe5d21eb907 could be the patch that broke this.
Project Member Comment 4 by david.os...@gmail.com, Dec 9
 > Interesting, the broken user only has 1 row! So out of curiosity, I created a second entry for the busted user 143 into account_external_ids to mirror my entry with 2 rows. However, upon trying to log in Gerrit happily deleted the new second entry I added!

In 2.13 account index was implemented. If you mutate the database, it's not reflected in the account index state, obviously. After database mutation, you need to reindex account index. Unfortunately, you cannot reindex one specific account, it was added only to master and will be in upcoming 2.14 release: [1]. What you can try is adding missing rows to account_external_id for all affected users, stop gerrit, reindex only account index (there is an option to reindex command), start gerrit. Does it work now? 

Can you upgrade the development site with all user base data to master? Can you reproduce the problem there? Does it fix it by mutating the database and reindexing per user base?

* [1] https://gerrit-review.googlesource.com/91880
Project Member Comment 5 by david.os...@gmail.com, Dec 9
> More digging reveals this https://github.com/gerrit-review/gerrit/commit/94732bfa1945a02bd4441b70a501cfe5d21eb907 could be the patch that broke this.

No, it didn't broke it. This patch is different approach to implement account data reindexing on account mutation operation, compared to my original approach: [1].

When acount data is mutated the underlying index state muste be updated. But because in addition the data is cached by gerrit cache, two things must happen:

* evict account cache
* reindex account index

The diff you pointed to, used cache eviction operation to kick reindexing of account index.

Note, that 2.13.3 works as expected with OAuth authentication scheme. I assume that the issue you are observing is specific to LDAP auth scheme, somehow. 

To track down the issue, i would try to follow those steps:

* Ensure the database state is correct (2 rows for user "awight"
* Reindex account index
* Login with user "awight" and investifate why gerrit thinks, that this user doesn't exist.
* [1] https://gerrit-review.googlesource.com/76677
Project Member Comment 6 by ekempin@google.com, Dec 9
I think this is what is happening:

> gerrit> select * from account_external_ids where account_id = 143;
> account_id | email_address | password | external_id
> -----------+---------------+----------+----------------
> 143        | NULL          | NULL     | username:awight

For this account the 'gerrit:awight' external ID is missing in the database.
Hence this external ID is also missing in the account index.
When the user tries to login, Gerrit doesn't find the external ID 'gerrit:awight' (since it's missing in the index).
Hence Gerrit tries to create a new account, let's say with account ID X.
It then creates the exteral ID with the Gerrit scheme for it:

> account_id | email_address        | password | external_id
> -----------+----------------------+----------+----------------
> X          | awight@wikimedia.org | NULL     | gerrit:awight

Then it tries to insert the external ID for the username scheme 'username:awight' but since this external ID is already taken, the insertion fails and Gerrit deletes the newly created account with its external ID.

Now let's have a look why manually inserting the 'gerrit:awight' external ID for account 143 doesn't work:

'gerrit:awight' is manually inserted for account 143:

> gerrit> select * from account_external_ids where account_id = 143;
> account_id | email_address        | password | external_id
> -----------+----------------------+----------+----------------
> 143        | NULL                 | NULL     | username:awight
> 143        | awight@wikimedia.org | NULL     | gerrit:awight

The account index didn't see this update, hence 'gerrit:awight' is not found when the user tries to login.
Hence Gerrit tries to create a new account, let's say with account ID X.
It then upserts (!) the exteral ID with the Gerrit scheme for the new account [1].
Upserting means that 

> 143        | awight@wikimedia.org | NULL     | gerrit:awight

is updated to

> X          | awight@wikimedia.org | NULL     | gerrit:awight

Then it tries to insert the external ID for the username scheme 'username:awight' but since this is already taken, the insertion fails and Gerrit deletes the newly created account *with its external ID*.

As result the 'gerrit:awight' for account 143 is gone.

How to fix it?

After the missing 'gerrit:awight' was inserted for account 143 into the database, this account must be reindex.
Since the REST endpoint for this not available in stable-2.13, you can only stop Gerrit, run reindex for all account and then restart Gerrit.

Still the question remains why the external ID was missing in the first place.

Also the upsert that overwrite the external ID of another account is questionable.

[1] https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-2.13/gerrit-server/src/main/java/com/google/gerrit/server/account/AccountManager.java#260




Project Member Comment 7 by ekempin@google.com, Dec 9
> After the missing 'gerrit:awight' was inserted for account 143 into the
> database, this account must be reindex.
> Since the REST endpoint for this not available in stable-2.13, you can only 
> stop Gerrit, run reindex for all account and then restart Gerrit.

I've cherry-picked the change the implements the REST endpoint to reindex a single account to stable-2.13 [1]. This should make it easier to fix this:

* insert the missing external ID into the database
* reindex this one account

No shutdown of Gerrit needed with this.

[1] https://gerrit-review.googlesource.com/92833
Hi thanks, will that prevent any future accounts from having this problem with capitals.

Will it also work for existing users that haven't got this problem yet?
Project Member Comment 9 by ekempin@google.com, Dec 9
 > Hi thanks, will that prevent any future accounts from having this problem
 > with capitals.

No, this has nothing to do with making the login case-insensitive.
These instructions say how to fix accounts that cannot login at all because their external ID is missing.

> Will it also work for existing users that haven't got this problem yet?

If an external ID for any existing user gets lost, you can fix it the same way: insert the external ID into the database + reindex the account.

Ok thanks, but note that most of the users are having problems due to the case sensitive.

Users who have there username as Reedy have to use reedy.
Project Member Comment 11 by ekempin@google.com, Dec 9
 > It then upserts (!) the exteral ID with the Gerrit scheme for the new
 > account [1].
 > Upserting means that 
 > 
 > > 143        | awight@wikimedia.org | NULL     | gerrit:awight
 >
 > is updated to
 > 
 > > X          | awight@wikimedia.org | NULL     | gerrit:awight

I've pushed [1] to prevent overwriting of the external ID.

[1] https://gerrit-review.googlesource.com/92834
Labels: FixedIn-2.13.4
Status: Submitted
Hi thanks.

This issue is also about case sensitive logins, since gerrit 2.13 it seems to have got a lot stricter. We are using workarounds such as if your username is Reedy login doing reedy and the opposite too (examples).
Hi it seems that, that fixes it, it also seems to kind of fix the capital problem too.
Comment 15 Deleted
Comment 16 Deleted
Comment 17 Deleted
Sign in to add a comment