Assigning to myself for now to dig further.

Hmm, this is the first time I'm hearing of this, but I'll take a look at some other perf graphs to see if landing other out of line stuff moved the needle somewhere. 

I reverted https://chromium.googlesource.com/chromium/src/+/718d68bed07d2fcf9b6c7ab7e051860bdbf1547f and tried on a local Debug build, with and without performed similarly. Trying again with Release build now.

Tried this with Release, no significant difference that I can tell.

However, it's interesting that one of my test accounts takes 8-11 seconds to initialize for both Debug/Release, while my other test account takes <1 second. I _think_ the big delay is being caused by PasswordSyncableService::MergeDataAndStartSyncing, and having different passwords is causing the difference.

Looking at the tp50/75/95 for Sync.ConfigureTimeLong_OK we see WILDLY different results. This may all be a red herring, but if something changed that pushed more users from a fast code path to a slow code path, it might explain what's happening here.

https://uma.googleplex.com/timeline_v2?q=%7B%22day_count%22%3A%2235%22%2C%22end_date%22%3A%22latest%22%2C%22window_size%22%3A1%2C%22filters%22%3A%5B%7B%22fieldId%22%3A%22channel%22%2C%22operator%22%3A%22EQ%22%2C%22study%22%3A%22%22%2C%22selected%22%3A%5B%221%22%5D%7D%2C%7B%22fieldId%22%3A%22milestone%22%2C%22operator%22%3A%22COMPARE%22%2C%22study%22%3A%22%22%2C%22selected%22%3A%5B%2250%22%2C%2251%22%5D%7D%5D%2C%22histograms%22%3A%5B%22Sync.ConfigureTime_Long.OK%22%5D%2C%22default_entry_values%22%3A%7B%22measureModel%22%3A%7B%22measure%22%3A%22%22%2C%22buckets%22%3A%5B%5D%2C%22percentiles%22%3A%5B%2250%22%5D%2C%22formula%22%3A%22%22%7D%2C%22zeroBased%22%3Atrue%2C%22logScale%22%3Afalse%2C%22showLowVolumeData%22%3Afalse%2C%22showVersionAnnotations%22%3Atrue%7D%2C%22entries%22%3A%5B%7B%22measureModel%22%3A%7B%22measure%22%3A%22percentile%22%2C%22percentiles%22%3A%5B%2295%22%2C%2250%22%5D%7D%7D%5D%7D

Okay, so it looks like saving passwords is really slow, like ~800ms per password slow. On initial sync we always have to pay this cost. Isn't the end of the world, it's in a different thread. But it does block our association process.

However, the big change is that we've recently added a new password field, federation_origin/federation_url in https://chromium.googlesource.com/chromium/src/+/0604e9e30c88836aaab577189b7c1f19f2ab55d8 . Whats happening is that this causes the Directory's password specifics to differ from the local model's specifics, and it looks like we have remote changes we need to incorporate, and we save EVERY local password that hasn't been modified since this last patch landed. And what do you know, this patch landed on Feb 26th, which matches our timeline exactly.

However, it looks like the password folks are already aware there are problems! Yesterday they landed https://codereview.chromium.org/1780683002 , although their description of the problem looks different from ours. Syncing my local client to HEAD to see if we still see degradation after their last fix.

https://codereview.chromium.org/1780683002 does fix loading time, marking as duplicate.

Nice find!

Indeed, the logic which compares the Sync data with the existing passwords became broken in r377834. As the result the whole password database was updated. It resulted in the performance regression.