Are you sure that is the customer's actual PAC file?

I will need a netlog in order to triage this.

The file attached in comment #0 do not seem relevant. I am guessing that is not the actual user's PAC file, as it would serve no point (it returns DIRECT for every branch, and never an actual proxy).

The biggest change in M67 was  Issue 680837 , which changed Chrome's proxy fallback to align with other browsers.

It is possible that could cause a performance regression if the enterprise deployment was including broken proxies in the list which fail slowly with a protocol error. Since before we would mark them as bad and skip over, whereas now we would try it first each time.

sorry, my bad, correct pac file is in the folder now.

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

eroman: so based on correct proxy, customer seems to be splitting lots of the requests between "proxy" hostname and "proxyf" hostname.

If one of those is bad, 66 and lower would blacklist and not keep trying whereas 67 will just keep trying it? How is order determined?

added general chrome debug log to the folder above. I also asked customer to check if some proxies are slower to respond or behaving badly based on eroman's comment.

Should that prove fruitless I'll collect net logs reproducing the issue also.

The NetLog should show whether the slowness if happening due to (lack of) fallback.

It is possible based on the PAC script being used, as a choice of two proxies is given.

The proxies are initially tried in order.

Is this indeed M67 stable blocker? Is there a workaround for this?

Pls note M67 has been out since 05/29.
05/29: 10% Windows and Mac, 100% Linux
06/07: 50% Windows and Mac, 100% Linux 

We already did total 3 stable releases 67.0.3396.62, 67.0.3396.79, 67.0.3396.87 (This release went out this morning, plan is to ramp up to 100% tomorrow). At the moment there is no plan to M67 stable respin unless extremely critical issue arise.

I recommend removing stable blocker.

Until we get a NetLog I can't properly triage this bug. But my expectation is this is an issue from just the one customer, and is due to a slow proxy server on their end.

(I haven't heard of wider issues with proxy in this release.)

removing RBS until we can show it's not slow/bad proxy in customer env.

Thank you eroman@ & jayhlee@.
We won't consider this as M67 stable blocker for further roll out to 100% tomorrow based on #11 & #12.

eroman@ netlog is in the Drive folder above. User turned on logging and then tried to visit www.google.com and www.cnn.com from another tab. Pages took in the range of 15-30 seconds to properly load.

Few more possibly relevent details from customer:

 - Issue only showing on Windows 7, Windows 10 users not seeing slowness.
 - Windows 7 devices are 32-bit while Windows 10 is 64-bit so possible it's that instead of Windows version.

Thanks for the log.

* I don't see evidence that this is related to  Issue 680837 , as there are no tunnel connection failures or proxy fallbacks.

* I don't see evidence of proxy slowness in the NetLog. The proxy server uses NTLM authentication, but in general is fast at establishing connections and serving results. The majority of URLRequests are completing in under 1 second.

* Some evidence of queuing delays for request at higher layer.


I don't see proxy as being the problem here. Is the customer sure this is related to proxy, or was that just a guess?

What exactly is the nature of the slowness:
  * Is the browser interactive while slow?
  * Can new tabs be opened, text selected on the current page, etc.
  * What does the loading status for the page say?

It could be useful to see a NetLog for a working Chrome in order to compare for differences.

My recommendations for next steps would be to have the customer run a bisect.

Instructions for running a bisect:

(1) Open a cmd.exe window

Ensure that python is installed by running and on the PATH:
python --version
This should output something like:
Python 2.7.3
(The script doesn't work with version 3.x of python)


(2) Download bisect-builds.py [1] from our source tree.

python -c "import urllib2; import base64; print base64.b64decode(urllib2.urlopen(\"https://chromium.googlesource.com/chromium/src/+/master/tools/bisect-builds.py?format=TEXT\").read())" > bisect-builds.py


(3) Start the bisection:

python bisect-builds.py -a win -g 540276 -b 550428 --use-local-cache -- --no-first-run --user-data-dir=C:\temp-chrome-profile https://www.google.com


The -g and -b numbers I give in step 3 set the search range between Chrome 66 and Chrome 67 (-g means the last known good version, and -b a known bad version).

The issue seems to be related to Hotlist-Enterprise. Hence, forwarding the issue to inhouse team for further triaging of the issue.

Thanks...!!

Unable to triage this issue from TE end because non availability of proxy network setup, hence adding TE-NeedsTriageHelp label for further triage from devteam

customer found that disabling the "user mode" module within their security software CrowdStrike resolved the performance issues for them. It's unclear what this software is doing and why it affects Chrome 67 and not 66. In any case, closing this issue as it does not seem to be a Chrome bug.

Thanks for the follow-up!

(I don't know what CrowdStrike does, but seems likely it hooks system APIs and would slow down applications. So this correlation makes sense)