link from customer
https://drive.google.com/drive/folders/0BwQr3Yf1hJdyTVBUcnJtbDhhcUk?usp=sharing

[RGV is not needed]

Assuming it's one of my CLs, Looking at https://chromium.googlesource.com/chromium/src/+log/59.0.3071.0..60.0.3112.34?pretty=fuller&n=10000, I think https://codereview.chromium.org/2860033003 is the most likely cause (With second most likely being https://codereview.chromium.org/2841163002.  None of the others looks at all likely to me).

How are they configuring their PAC script location?  Administrative policy?  DHCP?

Also, are they only seeing this on ChromeOS, or other platforms as well?

Switching to P0 as this breaks basic functionality. Also marking as release block beta.

I was unable to repro in my test env. which has a proxy.pac URL set via policy for both user and device WiFi profiles (same SSID).

Working with customer to get full details of their policy setup to see if there's something specific here?

mmenke@ beyond policy and a net-internals dump, anything else you'd like to see here?

network policy: https://docs.google.com/a/google.com/document/d/1q1h5z6vX4ffFk00iA-fSAKwOewHzpWiq4R09cNg2eBQ/edit?usp=sharing

Noticing some strange policy settings for customer:

For the device's OU, it looks like RI-WiFi SSID is listed twice, once inherited and once locally applied, once with proxy settings and once without. Expected behavior in this scenario is unpredictable. This isn't the SSID they showed in the video but it's a sign issue may be on customer config end.

will see if correcting this issue might solve for customer.

So it looks like the PAC is set on policy based on the network type and name?  I assume we only support that on ChromeOS?

Also, is there some easy way one can configure a Chromebook to mimic having a particular enterprise policy, without mucking with custom ChromeOS builds?

[RBS as well just to make sure we don't roll this out to stable without resolving it]

mmenke@ easiest path is probably to setup a test domain with Chrome device management licenses in prod though there are ways to put device in dev mode and pull fake policy.

Yes, the PAC is specific to customer's WiFi network(s), customers can also force a proxy at the user level via policy but they aren't doing so here afaik.

YAPS can help you emulate policy without needing the admin console. Reach out to managed-devices@ if the instructions are not clear enough to set up on your end. You also need to be able to produce the proper ONC blob to apply network policy. 


https://sites.google.com/a/google.com/chrome-enterprise-new/faq/using-yaps

Moving to P1 with RBS. 

P0 is more of a "don't leave the office" feel to it and this isn't at that level.

I do have a putative fix, though the complete lack of tests of the code in question makes me rather less than confident that it will resolve the issue.

Oops, forgot to update the description - the description indicates some uncertainty, but discovered the exact cause of the regression, and am fairly confident my CL fixes it.

The patch landed on trunk, but the bot has yet to update this bug.  Patch is https://codereview.chromium.org/2950573002/.  I'll give it a day or two on canary and then request a merge.

[cvintila]:  Perhaps you could ask them to try canary, once the patch makes it there?  It's position 480623.  I'm unsure if it will make it into 61.0.3136.0 or not.

Checking

seems to have landed in 61.0.3136.0:

https://storage.googleapis.com/chromium-find-releases-static/ed8.html#ed8193383116668bc6571cb1eed64f0514530724

it may be another day or so before that Chromium build is included with Chromium OS canary builds.

I don't think the issue is fixed - I think my patch had the right idea, but didn't quite fix the issue.

 Issue 728875  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/29e5a5de469977d5ce690492dfa62d3a69f77b1c

commit 29e5a5de469977d5ce690492dfa62d3a69f77b1c
Author: mmenke <mmenke@chromium.org>
Date: Wed Jun 21 21:35:37 2017

Fix chromeos::ProxyConfigService starting with the wrong ProxyConfig.

BUG= 734565 

Review-Url: https://codereview.chromium.org/2946153002
Cr-Commit-Position: refs/heads/master@{#481310}

[modify] https://crrev.com/29e5a5de469977d5ce690492dfa62d3a69f77b1c/chromeos/BUILD.gn
[modify] https://crrev.com/29e5a5de469977d5ce690492dfa62d3a69f77b1c/chromeos/network/proxy/proxy_config_service_impl.cc
[modify] https://crrev.com/29e5a5de469977d5ce690492dfa62d3a69f77b1c/chromeos/network/proxy/proxy_config_service_impl.h
[add] https://crrev.com/29e5a5de469977d5ce690492dfa62d3a69f77b1c/chromeos/network/proxy/proxy_config_service_impl_unittest.cc
[modify] https://crrev.com/29e5a5de469977d5ce690492dfa62d3a69f77b1c/components/proxy_config/pref_proxy_config_tracker_impl.cc
[modify] https://crrev.com/29e5a5de469977d5ce690492dfa62d3a69f77b1c/components/proxy_config/pref_proxy_config_tracker_impl.h

I'm going to request a merge on Monday, but would be great if someone who can repro the issue could test using trunk before then.

Looks like #25 landed in canary 9676.0.0 / 61.0.3138.0 which is still building. We'll try to get that out to customer once it's built.

Requesting merge for both CLs (https://codereview.chromium.org/2950573002/ and https://codereview.chromium.org/2946153002).  I believe they should fix the issue, though we don't have confirmation of that.

This bug requires manual review: M60 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), josafat@(ChromeOS), bustamante@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Can someone validate this on ToT build to make sure the fix works?
Once verified we can evaluate merge to M60

cvintila, can you share a Canary image with customer who reported this issue and confirm it works for them? As soon as it's confirmed we should be good to merge down to 60 so that this makes stable release.

Ack

I see the CLs are built into 61.0.3138.0 - which is not yet out
Could you please double-check which build I should be looking at? :D
Many thanks,

more exactly 61.0.3144.0
image-wise, for samus I found
61	9706.0.0	61.0.3146.0	4141207	release	canary

preliminary assessment from customer: "I've just tested this build and it does seem to resolve the issue."

Could you help me check a particular aspect the customer reported:

When rebooting the Chromebooks, it defaults back to Direct rather than Proxy. This is strange because when tested earlier it was working. 

Also on Beta the problem also manifested itself when launched an Incognito window, but in the build provided today (9706.0.0	61.0.3146.0) it seems to work in Incognito. 

On wireless it seems to work fine, the problem seems to only appear if connecting to Ethernet and restart Chromebook. 

Ryuta, any chance you have a setup and can repro these latest details?

cvintila:  Did the "When rebooting the Chromebooks, it defaults back to Direct rather than Proxy. This is strange because when tested earlier it was working." comment apply to beta or Canary?  The following comment implies that someone is trying beta and expecting it to work, but the issue isn't fixed there, since no merges have been approved.

@c#43: I gave this (see c#36) to the customer to try: 61	9706.0.0	61.0.3146.0	4141207	release	canary

hi Matt & all: what would be the next steps here?

On the canary build we provided to the customer, the issue is still happening.

Many thanks,

It's going to be slow going if we have to float every prospective fix to the customer to try out. Do we not have an internal repro? Because I thought for regressions like this we typically had QA do a repro + bisect?

+cyrus to make sure this has the appropriate visibility, because we're now two weeks away from stable, and typically proxy issues like this have compliance implications for EDU customers.

My two patches fixed pretty clear issues in how we handled setting the initial proxy configuration during ProxyConfigTrackerImpl / chromeos::ProxyConfigServiceImpl setup, which now looks correct to me (If overly complicated).  We are creating the IOThread's ProxyConfigService a bit earlier during startup, so could imagine some things not being hooked up at that point, though we're creating the ProfileIOData's at that same point as before, so I assume that one must be having the same issue as well.

I'm still not entirely convinced the bug still exists on Canary.

Per email thread, I'm now requesting permission to revert https://codereview.chromium.org/2872903006 and https://codereview.chromium.org/2860033003 on M60 to get things working there.

The CLs would need a bunch of other reverts if we wanted to do the same on trunk (Basically most of my work for the past 2 months), so I don't think we want to revert it there - for better or for worse, reworking this code is a pre-requisite for network servicification.

The combined revert of the two CLs is https://codereview.chromium.org/2972223002/, and does indeed apply cleanly to branch 3112.

Test Update(Preliminary tests) on the latest M61: 
This issue isn't occurring anymore on the latest M61 Dev; Dev(9723.0.0;61.0.3151.0).
After reboot the Proxy PAC file config is still being applied for both Ethernet and Wifi networks. 
(TODO: need to test if the proxy is applied when browsing through Incognito mode.) 
Will run more tests on M61. 

Tried to bisect this issue and was able to repro this on builds as  back as far as one of the earliest M60 dev builds(9592.0.0;60.0.3112.0). Haven't narrowed it down to the exact build tho. 

Awaiting merge to the latest M60 beta to test this on beta channel builds.(c#51)

Details of Test Environment/Test Accounts with setup:
https://docs.google.com/document/d/14ynoHcemXLk64GR3KS7NoHnRXXW1y8ZUDUlRmtwt5xM/edit

The regressions was almost certainly caused by https://codereview.chromium.org/2860033003 (Which would make it 02505dac8a35d644e38d27b01e48645ce39599bc / 471791)

@Josafat, per comment #54, fix has been verified on ToT. Can we approve the merge request now?

Sorry, it's still not very clear to me: have we tested build 61.0.3146.0 (the one we gave to the customer to test), and could not reproduce their observations?

krishnargv did test 61.0.3151.0 (Which contains no fix for that that 61.0.3156.0 does not), and could not reproduce.  I don't think anyone has tested 61.0.3146.0.

CL-wise 61.0.3146.0 should have contained the fix for this issue

if not: could you please let me know which CrOS build contains the fix for this issue, so that we could check w/ the customer if this issue is fixed also in their environment? (I assume we want this re-assurance before rolling out)

Merge approved for 60.

Erm...which was approved?  The revert or the fix?  I'm happy to land either.

I provided canary-channel%2Fsamus%2F9706.0.0%2Fchromeos_9706.0.0_samus_recovery_canary-channel_mp-v3

I would say whichever you feel is the safest route to get 60 into a good state, my impression was a revert per comment 51/52. 

Great!  Will do.  I agree that's the safest route.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/28255e31f57b4aa645e1a3d9acc02a133537d347

commit 28255e31f57b4aa645e1a3d9acc02a133537d347
Author: mmenke <mmenke@chromium.org>
Date: Mon Jul 10 18:38:21 2017

Revert https://codereview.chromium.org/2860033003

This refactored how the ProxyConfigTrackerImpl starts up, and broke
proxy configuration on ChromeOS in the process. The issue has been at
least partially fixed on trunk, but issue does not appear to be
completely fixed.

This also reverts https://codereview.chromium.org/2872903006, a small
CL that has no impact on behavior, which was landed on top of the
other CL.

Revert "Don't delay creation of system URLRequestContext until first use."

This reverts commit 02505dac8a35d644e38d27b01e48645ce39599bc.

Revert "Use URLRequestContextStorage for the SystemURLRequestContext."

This reverts commit f2e53b2bec633349e045fd4566d841289a295247.

TBR=mmenke@chromium.org
BUG= 734565 
NOTRY=true
NOPRESUBMIT=true

Review-Url: https://codereview.chromium.org/2972223002
Cr-Commit-Position: refs/branch-heads/3112@{#562}
Cr-Branched-From: b6460e24cf59f429d69de255538d0fc7a425ccf9-refs/heads/master@{#474897}

[modify] https://crrev.com/28255e31f57b4aa645e1a3d9acc02a133537d347/chrome/browser/io_thread.cc
[modify] https://crrev.com/28255e31f57b4aa645e1a3d9acc02a133537d347/chrome/browser/io_thread.h
[modify] https://crrev.com/28255e31f57b4aa645e1a3d9acc02a133537d347/chrome/test/base/testing_io_thread_state.cc
[modify] https://crrev.com/28255e31f57b4aa645e1a3d9acc02a133537d347/components/proxy_config/pref_proxy_config_tracker.h
[modify] https://crrev.com/28255e31f57b4aa645e1a3d9acc02a133537d347/components/proxy_config/pref_proxy_config_tracker_impl.cc
[modify] https://crrev.com/28255e31f57b4aa645e1a3d9acc02a133537d347/components/proxy_config/pref_proxy_config_tracker_impl.h
[modify] https://crrev.com/28255e31f57b4aa645e1a3d9acc02a133537d347/components/proxy_config/pref_proxy_config_tracker_impl_unittest.cc

The updates as per comment 65 should fit into build 60.0.3112.65 - not built yet

I plan to ask the customer to try build 60.0.3112.65 on platform "samus" in order to check healthiness of this issue. Please let me know if you have a different suggestion.

Many thanks all :D

Going to marked this as fixed, but not entirely sure if it's completely fixed in M61 or not.

Verified issue is fixed as tested in M60.0.3112.79:9592.70.0 dev kevin and M61.0.3163.13:9765.7.0 dev kevin.

Tested having a device policy with direct connection and a user policy with autoproxy, both autoconnect.  Confirmed proxy successfully connected automatically each time. Tested signing out, powering down and restarting, rebooting via VT2 repeatedly without see any issue reconnecting to the autoproxy automatically.

I confirm the video of the customer.
I'm able to repro with the version of the customer on a peppy:
version:
Google Chrome: 60.0.3112.34 (Official Build) beta (64-bit)
Revision: 0
Platform: 9592.22.0 (Official Build) beta-channel peppy

with  60.0.3112.80 (actual beta pushed for peppy) is fixed
policy:
https://drive.google.com/open?id=0B01ZVp8vDQocRThlUlBIMldBMWM

link to chrome://version
https://drive.google.com/open?id=0B01ZVp8vDQocNFlFNE41WUlPcHc

I see the same issue on R62-9869.0.0 (Samus) and R61-9765.36.0 (Eve) devices. Do I re-open this issue or file a separate issue?

(unless anybody gives different direction) Maybe log a new one and tag it as RB for the respective launches?

Can no longer reproduce this issue on R61-9765.52.0 (Eve). If I see this again, will a new issue with logs.