New issue
Advanced search Search tips

Issue 757181 link

Starred by 28 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

Headless/DevTools doesn't handle net::ERR_SSL_CLIENT_AUTH_CERT_NEEDED error

Reported by kbla...@gmail.com, Aug 19 2017

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.39 Safari/537.36

Steps to reproduce the problem:
1. /opt/google/chrome-beta/chrome --headless --dump-dom https://archive.sap.com/

What is the expected behavior?
Chrome fetches the page's content.

What went wrong?
Page content not downloaded/dumped with headless mode, while chrome works as expected without headless mode.

Did this work before? N/A 

Does this work in other browsers? N/A

Chrome version: 61.0.3163.39  Channel: beta
OS Version: Gentoo Linux
Flash Version: 

/opt/google/chrome-beta/chrome --headless --dump-dom  -enable-logging --v=1 https://archive.sap.com/

[0819/172057.398845:VERBOSE1:zygote_main_linux.cc(537)] ZygoteMain: initializing 0 fork delegates
[0819/172057.399098:INFO:cpu_info.cc(50)] Available number of cores: 8
[0819/172057.407765:VERBOSE1:proxy_config_service_linux.cc(854)] All gsettings tests OK. Will get proxy config from gsettings.
[0819/172057.407917:VERBOSE1:proxy_config_service_linux.cc(1608)] Obtained proxy settings from GSETTINGS
[0819/172057.407956:VERBOSE1:webrtc_internals.cc(108)] Could not get the download directory.
[0819/172057.410912:VERBOSE1:sandbox_linux.cc(70)] Activated seccomp-bpf sandbox for process type: renderer.
[0819/172057.437039:VERBOSE1:sandbox_linux.cc(70)] Activated seccomp-bpf sandbox for process type: gpu-process.
[0819/172057.477273:ERROR:nss_util.cc(747)] After loading Root Certs, loaded==false: NSS error code: -8018
[0819/172057.477514:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Google 'Aviator' log
[0819/172057.477523:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Google 'Icarus' log
[0819/172057.477528:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Google 'Pilot' log
[0819/172057.477531:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Google 'Rocketeer' log
[0819/172057.477534:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Google 'Skydiver' log
[0819/172057.477538:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: DigiCert Log Server
[0819/172057.477541:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: DigiCert Log Server 2
[0819/172057.477545:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Symantec log
[0819/172057.477552:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Symantec 'Vega' log
[0819/172057.477556:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Symantec 'Sirius' log
[0819/172057.477560:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: WoSign log
[0819/172057.477564:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Venafi Gen2 CT log
[0819/172057.477567:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: CNNIC CT log
[0819/172057.477571:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: StartCom log
[0819/172057.477575:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Comodo 'Sabre' CT log
[0819/172057.477579:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Comodo 'Mammoth' CT log
[0819/172057.477583:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Izenpe log
[0819/172057.477587:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Venafi log
[0819/172057.477590:VERBOSE1:multi_log_ct_verifier.cc(75)] Adding CT log: Certly.IO log
[0819/172057.477595:VERBOSE1:proxy_service.cc(959)] PAC support disabled because there is no system implementation
[0819/172057.480961:VERBOSE1:gles2_cmd_decoder.cc(3401)] GL_OES_packed_depth_stencil supported.
[0819/172057.484007:VERBOSE1:gles2_cmd_decoder.cc(3401)] GL_OES_packed_depth_stencil supported.
[0819/172057.486247:VERBOSE1:gles2_cmd_decoder.cc(3401)] GL_OES_packed_depth_stencil supported.
[0819/172057.488851:VERBOSE1:gles2_cmd_decoder.cc(3401)] GL_OES_packed_depth_stencil supported.
[0819/172057.490971:VERBOSE1:gles2_cmd_decoder.cc(3401)] GL_OES_packed_depth_stencil supported.
[0819/172057.530198:VERBOSE1:navigator_impl.cc(242)] Failed Provisional Load: https://archive.sap.com/, error_code: -110, error_description: , showing_repost_interstitial: 0, frame_id: 2
<body></body>
[0819/172057.582530:VERBOSE1:proxy_config_service_linux.cc(552)] ~SettingGetterImplGSettings: releasing gsettings client
[0819/172057.582626:VERBOSE1:sandbox_ipc_linux.cc(123)] SandboxIPCHandler stopping.

Does the error code -110 refer to https://cs.chromium.org/chromium/src/net/base/net_error_list.h?l=157 ? Why is it working without headless mode?
 

Comment 1 by kbla...@gmail.com, Aug 19 2017

Could it be that chrome has issues with servers sending a "certificate request" during the ssl handshake when running in headless mode?
Cc: dvallet@chromium.org
Components: Internals>Headless
Thanks for the report!

Could you verify if this work for Canary? We've recently fixed an issue with NSS initialization

Comment 3 by kbla...@gmail.com, Aug 21 2017

I have built chromium from source yesterday, same result. I guess that's recent enough?

Comment 4 by kbla...@gmail.com, Aug 21 2017

"good": no --headless
"bad" : --headless
sap-archive_bad.png
180 KB View Download
sap-archive_good.png
226 KB View Download
Thanks for checking. 

I cannot reproduce on my build, both canary and stable work as expected. 

I think your guess is right, maybe there's an issue with the SSL certificate and your network and this is handled differently on non-headless (by default non-headless will ignore some errors), but I don't know how to reproduce it. 

You could try and use devtools to handle the certificate error, and see if ignoring it works: https://chromedevtools.github.io/devtools-protocol/tot/Security/#method-handleCertificateError

Comment 6 by kbla...@gmail.com, Aug 21 2017

Thanks for the advice! Are you saying that all your chrome builds give you the html of the page if you run
chrome --headless --dump-dom https://archive.sap.com/
?

I launched my chromium build with
chrome --remote-debugging-port=9222 --headless
and ran the attached Node.js program. I ended up with the attached empty screenshot. For other pages the screenshot was not empty.

Comment 7 by kbla...@gmail.com, Aug 21 2017

testChromeRemoteInteface.js
1.5 KB View Download
headless-results.png
2.7 KB View Download
Thanks for the script!

Yes, the site works for me. 
I've noticed that you mention a proxy server in your script. Note that your non-headless version may be picking up the proxy configuration from your user-data-dir. This is *not* supported in headless mode (and it actually uses a different user profile when running). 

Maybe try and setup the proxy configuration using the --proxy-server flag: 
https://cs.chromium.org/chromium/src/headless/app/headless_shell_switches.cc?rcl=d2382cf1b66dc65c43044320149a48fc1456bf83&l=43

Comment 9 by kbla...@gmail.com, Aug 21 2017

No I don't use any proxy server, I just quickly picked this script up at https://intoli.com/blog/making-chrome-headless-undetectable/ because it seems to do the "handleCertificateError" you mentioned and I didn't remove the comments from the original script.

Comment 10 by kbla...@gmail.com, Aug 21 2017

I guess https://chromedevtools.github.io/devtools-protocol/tot/Security/#method-handleCertificateError just handles invalid server certificates. It doesn't seem to help change chrome's behaviour in case the server sends a client certificate request.
I'm not sure then why it could be happening. I don't get any certificate error and the page renders just fine. 

You could add a console.log step in your handle certificate function to see if you are actually getting a cert error, or if there's other issue.

Comment 12 by kbla...@gmail.com, Aug 22 2017

I have no idea why this only happens to me and only in headless mode. I added a console.log in the Security.certificateError callback, nothing was logged. This is all I get in wireshark when visiting the page:
Screenshot from 2017-08-22 02-45-49.png
120 KB View Download

Comment 13 by kbla...@gmail.com, Aug 22 2017

I've tried headless firefox in the meantime without issues. I am usually using http://webdriver.io and selenium to communicate with the browsers.
Labels: TE-NeedsTriageHelp

Comment 15 by wbs...@gmail.com, Aug 23 2017

I have the same issue as mentioned above, and I used the command like this "./headless_shell --no-sandbox --disable-gpu --enable-logging --v=1 https://account.chsi.com.cn".

[0823/181016.905454:VERBOSE1:MemoryCache.cpp(166)] Evicting resource 0xb37aeff2f10
for "https://account.chsi.com.cn/" from cache
[0823/181016.906531:VERBOSE1:navigator_impl.cc(241)] Failed Provisional Load: https
://account.chsi.com.cn/, error_code: -110, error_description: , showing_repost_inte
rstitial: 0, frame_id: 2
[0823/181016.933449:VERBOSE1:navigation_controller_impl.cc(912)] Navigation finishe
d at (smoothed) timestamp 13147956616933442

My headless_shell version is `60.0.3090.0`. When I compiled version 62, I had the same issue. And on my ubuntu, using the command "./chrome --headless --disable-gpu https://account.chsi.com.cn", the error occurred again.

 

Comment 16 by kbla...@gmail.com, Aug 26 2017

I am able to reproduce the same problem with https://account.chsi.com.cn/ as well.
Summary: Headless doesn't handle net::ERR_SSL_CLIENT_AUTH_CERT_NEEDED error (was: headless mode fails for https://archive.sap.com)
 Issue 758452  has been merged into this issue.

Comment 19 by wbs...@gmail.com, Aug 28 2017

Using a mitm proxy solved this problem.

Comment 20 by kbla...@gmail.com, Aug 28 2017

Sounds like you're hiding it by letting the proxy handle the problematic ssl handshake.
Status: Available (was: Unconfirmed)
I know this isn't a self-contained repro, but the following illustrates the issue (and that server is not scheduled to shut down anytime soon)

```
'use strict';

const puppeteer = require('puppeteer');

(async () => {
    // NB: ignoreHTTPSErrors gets us only so far. It still trips over the
    // client certificates in our certs (even when they're optional), or
    // perhaps something else:
    //
    // https://cockroach-adriatic-0001.crdb.io:8080
    const browser = await puppeteer.launch({ignoreHTTPSErrors: true});
    const page = await browser.newPage();
    await page.goto('https://cockroach-adriatic-0001.crdb.io:8080', {waitUntil: 'networkidle'});
    await page.pdf({path: 'out.pdf', format: 'A4'});

    browser.close();
})();
```

Perhaps it helps, though it doesn't print the error message.
This does though: https://github.com/GoogleChrome/rendertron/pull/80 and you get `{ message: 'net::ERR_SSL_CLIENT_AUTH_CERT_NEEDED' }` when trying the above URL.
Will be very useful if this is fixed. We use the headless mode extensively and are affected by this issues for some of our major use cases.
thanks!
Depending on this issue as well. Waiting for a fix here :)
Owner: irisu@chromium.org
Did anyone find a workaround for this? I don't actually need (or even want) the client-side SSL certificate provisioning to succeed, I just can't let the Page.open() fail on the ERR_SSL_CLIENT_AUTH_CERT_NEEDED error. I tried to generate a self-signed certificate with a CN and saN of the service (OpenShift in my case) to connect to, convert it to PKCS12 [1], and import it into the NSS DB so that chromium should be able to see it [2]:

❱❱❱ certutil -d sql:.pki/nssdb/ -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

127.0.0.2 - test                                             u,u,u

But this doesn't make any apparent difference, Chromium headless still refuses to connect (same error).


[1] https://stackoverflow.com/questions/21141215/creating-a-p12-file
[2] https://chromium.googlesource.com/chromium/src/+/master/docs/linux_cert_management.md
Status: Assigned (was: Available)
Cc: dgozman@chromium.org eseckler@chromium.org
 Issue 754210  has been merged into this issue.
Components: Platform>DevTools
Summary: Headless/DevTools doesn't handle net::ERR_SSL_CLIENT_AUTH_CERT_NEEDED error (was: Headless doesn't handle net::ERR_SSL_CLIENT_AUTH_CERT_NEEDED error)
The same problem is reported in puppeteer
https://github.com/GoogleChrome/puppeteer/issues/963
Owner: ----
Status: Available (was: Assigned)
Is there any update on this? It is a pretty significant issue I would say.
Researching this bug I found many discussions where people ended up switching to Firefox.
Owner: caseq@chromium.org
Status: Assigned (was: Available)
Andrey, mind taking a look?
Status: Started (was: Assigned)
Project Member

Comment 35 by bugdroid1@chromium.org, Apr 5 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8f0691b18dc5d941d5b6b3c67a483da02400670

commit c8f0691b18dc5d941d5b6b3c67a483da02400670
Author: Andrey Kosyakov <caseq@chromium.org>
Date: Thu Apr 05 15:05:12 2018

Headless: do not cancel requests if server wants a certificate

... just tell content to proceed with no certificate instead
of cancelling the request.

Bug:  757181 
Change-Id: Ib7db0d5bbbdf73dfd3bad4fb5827519eb8c0dbbd
Reviewed-on: https://chromium-review.googlesource.com/996954
Reviewed-by: Alex Clarke <alexclarke@chromium.org>
Commit-Queue: Andrey Kosyakov <caseq@chromium.org>
Cr-Commit-Position: refs/heads/master@{#548422}
[modify] https://crrev.com/c8f0691b18dc5d941d5b6b3c67a483da02400670/headless/lib/browser/headless_content_browser_client.cc
[modify] https://crrev.com/c8f0691b18dc5d941d5b6b3c67a483da02400670/headless/lib/browser/headless_content_browser_client.h
[modify] https://crrev.com/c8f0691b18dc5d941d5b6b3c67a483da02400670/headless/lib/headless_browser_browsertest.cc

Status: Fixed (was: Started)
Wow this was quick! Thanks!
When should we expect a fixed build that we could use?
This really depends on what channel you are. This site lets you track what releases have certain commit: http://omahaproxy.appspot.com/
Commit c8f0691b... initially landed in 67.0.3390.0

running with chrome 68.0.3440.106 problem persists.
any idea why?
this problem persists with macos version. 70.0.3538.77
I am still having issues on macos version 70.0.3538.77 as well. It fails to attach client certificate stored in Keychain on headless mode but the request works fine with headless-mode disabled. Should I submit a separate bug report for this? 

Sign in to add a comment