Thanks for the report and sorry for the inconvenience!

Redirecting to Signin folks, is this expected?

Tried to reproduce the issue on reported chrome version 71.0.3578.98 using Windows 10.
Steps: 
-----
1. Launched reported chrome 
2. Installed PIA extension and tried to signup the extension 
As we have observed that the extension is paid.Hence, routing this issue to MTV team for help in further triaging and adding "TE-NeedsTriageFromMTV" label to it. 
As per comment #0, suspecting: https://chromium-review.googlesource.com/c/chromium/src/+/1252868/ , 
from the changelog: https://chromium.googlesource.com/chromium/src/+log/8984cac941465ae9724821910f92b35e6d239234..895f6277b7d72c7a952731aee3581feb257e1db5. 

Hence CC'ing dev (asanka@chromium.org) for further inputs on it.

Thanks.!

https://chromium-review.googlesource.com/c/chromium/src/+/1252868/ is for HTTP auth, which is unrelated to sync.  Since this is related to how stuff is synced, seems unlike to be a network stack issue.

This may or may not be useful, but I know that when this particular extension starts up enabled, some sites would show the basic auth dropdown asking for a user/pass. I'm not sure exactly how this extension does its VPN-ing, however.

I've attached a video showing this, though if I recall correctly it used to stay around until I cleared it and refreshed rather than popping up then immediately closing. I've been avoiding shutting my browser down with the VPN enabled so I don't have to keep logging into my other accounts every morning.

Sorry about the extension being paid; I'd be happy to provide my credentials temporarily if that helps, though PIA has a 5-concurrent device limit I'd rather not hit. I also tried going through a bunch of "free" VPN extensions on the Web Store, but none of them seemed to offer the ability to remain enabled after a restart like PIA does.

Deleted: syncbug3.mp4 1.7 MB

	syncbug3.mp4 1.7 MB View Download

Could you attach a NetLog of the issue, perhaps?
https://www.chromium.org/for-testers/providing-network-details

I'll try grabbing a NetLog at launch tonight, but during the day while I was busy, I let my machine build chromium from source both with and without 5d2da02a711bc40ec41b8bd5b15a3bd20232aa5a.

Reverting that commit fixes the issue!

I've attached a video showing both builds, as in the repo on the left, and with the revert on the right.

Deleted: syncbug_fixed.mp4 13.5 MB

	syncbug_fixed.mp4 13.5 MB View Download

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Here are two NetLog json files captured using --log-net-log with the same exact testing as in my previous post; before reverting is "bad", after reverting is "good". I did not change the capture mode, so hopefully these don't include any of my credentials.

Deleted: bad.json 496 KB

bad.json 496 KB View Download

Deleted: good.json 1.1 MB

good.json 1.1 MB View Download

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Removing from the Sync triage queue again. This is related to Signin. Sync stopps working only because we loose the Google account cookies.

 Issue 915835  has been merged into this issue.

As was mentioned on https://bugs.chromium.org/p/chromium/issues/detail?id=915835 we are the developers of said extension and would be happy to provide a test account for PIA should this prove useful, as well as answer any questions that might help!

I also had some NetLogs from https://bugs.chromium.org/p/chromium/issues/detail?id=915835, one that worked with my account being paused for just a second, and a failed NetLog where the paused status was permanent.

From digging through the NetLogs and trying to find a difference between the two, it looks like the failed NetLog tried to attempt an OAuth token twice with both tries being blocked due to the proxy not being able to log in. I think this has to do with the fact that the extension was not bootstrapped fully by that point and the onAuthRequired event handler within the extension that provides the credentials to the proxy was not registered. Chrome then proceeded to call:

122: URL_REQUEST
https://accounts.google.com/Logout?source=ChromiumAccountReconcilorDice

which seems to log out the account permanently, making no further attempts at an OAuth token.


On the working NetLog, chrome tried to request an OAuth Token three times. The first two failed due to the proxy not having proper credentials as before. But on the third attempt the extension was bootstrapped fully and could provide credentials. I believe this is what caused the paused status for about a second before the final OAuth call goes through.


Strangely enough, there seems to be some kind of timing/proximity issue here. If the proxy connection is set to a location close to where I'm located, the account will always fail to sync (permanently paused). But if I set the proxy location to be somewhere across the world, the account will be paused for a second and then successfully sync. 

Hope that helps. 

Hopefully merging your issue into mine doesn't lose any traction. I did at least bisect last week and trace the bug back to 5d2da02a711bc40ec41b8bd5b15a3bd20232aa5a (see above), though I'm not sure if that commit caused the bug or just revealed some previously unknown issue.

I leave the PIA extension on almost all of the time, but I keep forgetting to disconnect, which means when I come back I get the nice surprise of having to log back into all of my accounts again...

[removing the duplicate copies of me on here.]

jacob.b.bailey@gmail.com: Sorry for getting late this this bug. Would it be possible to  open chrome://signin-internals when this happens, take a screenshot and attach it to the bug (this is a Chrome client internal debug page and has some private information like email that you may want to hash out before copy-pasting it to the bug).

Also assigning to droger@ for follow-up.

msarda@google.com: I've attached a screenshot of the chrome://signin-internals page after a failed auth attempt on browser restart.

Deleted: Screen Shot 2018-12-20 at 8.35.32 AM.png 332 KB

	Screen Shot 2018-12-20 at 8.35.32 AM.png 332 KB View Download

Here's a copy of what that same page looks like after restarting the browser with a proxy location that's farther away. This is the scenario where the account sync is briefly paused but eventually syncs.

Deleted: Screen Shot 2018-12-20 at 8.48.05 AM.png 327 KB

	Screen Shot 2018-12-20 at 8.48.05 AM.png 327 KB View Download

Here's my screenshot. The left is plain Chromium at the revision listed, the right is with that commit reverted. I started both, started the VPN, then restarted both.

Deleted: signin-internals.png 169 KB

	signin-internals.png 169 KB View Download

Is there some other piece of info needed, or has the bot forgotten to remove the label?

Possibly also the same issue:  Bug 915108 .

Hmm, they both seem to involve proxying and some issue at startup, but I wouldn't think it's the same, given that they're on 70 and the commit I bisected to was only introduced in 71 (and they don't mention accounts being signed out). 70 doesn't exhibit the behavior in this issue. (I'm still running 70 on my laptop without trouble.)

Hi,

I read through all the comments, I think the issue reported by me here( https://bugs.chromium.org/p/chromium/issues/detail?id=918321#c1 )is somewhat similar to this one.

 Bug 915108  is also reported to be similar(it was also raised by me), so possible all of these are related to the same thing. 

PS - I still haven't found a solution for that one as well.

So, this is affecting our customers greatly and we are losing customers because of the sync issues. Since we are the developers, we can provide you with any details you need.  

@Reporter - Could you please tell me how did you revert the OS to previous stable version because it worked for me before?

-Nidhi

Bug 918321 definitely looks the same, though perhaps even worse in your case because it can't be disabled (vs in PIA I can disable it before shutting down so that it isn't enabled when the browser starts again).

@Nidhi If by "revert the OS", you mean revert to an older version of Chrome/Chromium, I had found the commit that causes (or at least surfaces) this issue and built Chromium from source with and without it. It's not an official stable build, and required me to setup a bunch of API secrets to test syncing (since that's through Google APIs).

On Windows, I'm stuck at 71+ because Chrome will automatically upgrade itself and the profile, but on my Linux machine I simply haven't updated the package from 70 to 71.

@Jacob
I'm afraid I can't do that, since we have customers with Chromebooks using our product and because chrome OS updates itself in the background there wouldnt be any way for them to revert it to the previous build and so all of them would be facing this issue! That will create a lot of backlash for us. I hope someone from the chromium team replies back soon!

I understand, I was just answering your question as to how I was able to get the non-broken behavior on a new build.

@Jacob
I appreciate your help.

I have one more question. So Google probably has a list of API calls which they make for syncing, right? What our extension does is it blocks Internet content according to the needs of our users. SO, probably the issue might be that some of the URLs (related to the sync) are being affected because of it. I hope this is the issue and so then we can bypass these URLs in our PAC file.

All I need to find is the URLs for these API calls.

Comment 11 contains my netlogs, good and bad. You may find the info you need there.

Thanks!

@Jacob

Update - Adding the URLs to PAC file did help us solve the issue. So, thank you for your help. Really appreciate it. :-)

-Nidhi

By "URLS to PAC" I presume you mean just the hostnames of those URLs -- since PAC scripts are not given the full URL to https:// requests.

Yup, hostnames only.

Seems this was resolved by adding the URLs to the PAC file. Should this bug be closed? Is there anything left for us to investigate?

Note that I've looked at the signin-internal screenshots at comment #25 and I do not see any issues in there.

The mobicip.com user is a completely different user than the PIA folks. I have just tested the latest canary build (73.0.3668.0) with the latest PIA extension (1.8.1), and my original issue still reproduces. I don't think this issue should be closed.

I'm assuming the PAC file is something that determines which requests get proxied and which do not (reading https://developer.chrome.com/extensions/proxy, it seems that way). If so, I don't think it's good that a proxy cannot reasonably proxy requests to _all_ hosts, or whichever hosts the mobicip included, since they did not say what exactly they did. The proxies worked fine before; a change here broke signin when a proxy is active.

What other information can I provide? You'll have to forgive me, but it's a bit frustrating to have bisected this problem to the exact commit that started it and have no indication that it's being looked at.

As for my screenshot on #25, I'm curious why it doesn't look like there's an issue. The left shows credentials failing to be accepted with 5d2da02a711bc40ec41b8bd5b15a3bd20232aa5a applied, and the right without that commit signed in without issue.

The PIA extension exposes the bypass list in its settings, so I can at least add "www.googleapis.com" to the list for now (tested and working in canary and on stable, chosen based on that screenshot's message), but I know that domain has a lot more served from it than just the things required for Chrome signin.

@msarda

No! This bug should NOT be closed just because the VPNs have come up with a quick fix.

I feel Jacob's frustration. As he says, "The proxies worked fine before; a change here (in Chrome) broke signin when a proxy is active."

It seems to be network related, rather than a signin issue. Reassigning.

Asanka, please take a look at the logs in comment #14, which show the behavior with and without https://chromium.googlesource.com/chromium/src/+/5d2da02a711bc40ec41b8bd5b15a3bd20232aa5a applied.

This bug relates to an extension's ability to provide HTTP auth to an HTTPS proxy.

Hrm...Does it?  It sounds to me like some signin-magic is clearing Google cookies when it can't access certain URLs.

kudos to @mmenke

Here is an excerpt from a blog post by Zach Koch, Chrome Product Manager; in which he attempts to mitigate the sign-in/sync fiasco of v.69

"We’re also going to change the way we handle the clearing of auth cookies. In the current version of Chrome, we keep the Google auth cookies to allow you to stay signed in after cookies are cleared. We will change this behavior that so all cookies are deleted and you will be signed out."

https://www.blog.google/products/chrome/product-updates-based-your-feedback/

Let me summarize to make sure I understand:
* sign-in/sync does not work when "www.googleapis.com" is proxied
* sign-in/sync works fine when "www.googleapis.com" is *not* proxied

mmenke@: Sign-in/sync requires Chrome to be able to reach www.googleapis.com (we need that to get OAuth2 access tokens for use sync). We map the response to the internal errors in code https://cs.chromium.org/chromium/src/google_apis/gaia/oauth2_access_token_fetcher_impl.cc?rcl=1074c8ac99fa1acb0228f4e9ae540e9fae7081bf&l=226 . I expect that in this case, we get an 405 HTTP error from the proxy that we map internally to a INVALID_GAIA_CREDENTIALS that lead to all the sign-in and sync machinery treating this as a permanent sign-in/sync error.

Form the list of the bisect and from comment #45, I think something changed in the way Chrome interacts with the proxy in this case. So I do think this is an issue in the network code.

Note: We could change the code in https://cs.chromium.org/chromium/src/google_apis/gaia/oauth2_access_token_fetcher_impl.cc?rcl=1074c8ac99fa1acb0228f4e9ae540e9fae7081bf&l=267 to add a new error, but it is still not clear to me if that error is permanent or transient from our perspective. Transient means that we would not surface any error UI in Chrome which could also be misleading as sync will not work either.

Since we can access the Internet, and proxy code has no Google-specific magic, the underlying network proxy code seems to be working correctly here.

Issue 916075 has been merged into this issue.

This is specifically whatever sign-in (or check) when the browser is starting up.

If I leave PIA enabled and relaunch, it will continue to be active when the browser starts again, and I'll see the behavior in my video (reports to me that syncing is paused and also logs me out of every account, probably due to that consistency stuff).

However, this doesn't prevent me from remaining connected with the proxy and logging in again.

If I add "www.googleapis.com" to PIA's bypass list, then relaunching does not put me into a paused state.

I think the core issue for proxy extension developers is wondering why the initial calls out to authenticate the chrome account are sent before the proxy extension has a chance to at least setup the OnAuthRequired handler so that the extension can provide a password on authentication requests. 

Not the responsibility of the proxy extension developers. Chrome Update broke it.

Since crbug.com/916075 is merged here, adding chromeOS since it's happening to numerous enterprise customers, who are using iboos (which also behaves like proxy, vpn) extension on chromeos.

As customer reported, it's happening on Chrome OS 70 (70.0.3538.110), but not happening anymore in 72.0.3626.49, so maybe we will need to know which change helped to resolve the issue, to be sure change will remain in 72 stable.

Where in bugs. does it say that it is not happening in 72.0.3626.49? Or, have you tried it in the beta?

I'm not convinced at all that 916075 is the same bug if they're saying 72 fixed it. I can still reproduce my original issue following the same steps on the latest beta build, 72.0.3626.64.

Even though my original issue was reported on 71, I have been testing on the canary, which is 73.

I've attached a screenshot showing my syncing paused on 72.0.3626.64. Not as good as a screencast, but I can certainly show it if needed.

Deleted: chrome72.png 162 KB

	chrome72.png 162 KB View Download