Could it be that chrome has issues with servers sending a "certificate request" during the ssl handshake when running in headless mode?

Thanks for the report!

Could you verify if this work for Canary? We've recently fixed an issue with NSS initialization

I have built chromium from source yesterday, same result. I guess that's recent enough?

"good": no --headless
"bad" : --headless

Deleted: sap-archive_bad.png 180 KB

	sap-archive_bad.png 180 KB View Download

Deleted: sap-archive_good.png 226 KB

	sap-archive_good.png 226 KB View Download

Thanks for checking. 

I cannot reproduce on my build, both canary and stable work as expected. 

I think your guess is right, maybe there's an issue with the SSL certificate and your network and this is handled differently on non-headless (by default non-headless will ignore some errors), but I don't know how to reproduce it. 

You could try and use devtools to handle the certificate error, and see if ignoring it works: https://chromedevtools.github.io/devtools-protocol/tot/Security/#method-handleCertificateError

Thanks for the advice! Are you saying that all your chrome builds give you the html of the page if you run
chrome --headless --dump-dom https://archive.sap.com/
?

I launched my chromium build with
chrome --remote-debugging-port=9222 --headless
and ran the attached Node.js program. I ended up with the attached empty screenshot. For other pages the screenshot was not empty.

Thanks for the script!

Yes, the site works for me. 
I've noticed that you mention a proxy server in your script. Note that your non-headless version may be picking up the proxy configuration from your user-data-dir. This is *not* supported in headless mode (and it actually uses a different user profile when running). 

Maybe try and setup the proxy configuration using the --proxy-server flag: 
https://cs.chromium.org/chromium/src/headless/app/headless_shell_switches.cc?rcl=d2382cf1b66dc65c43044320149a48fc1456bf83&l=43

No I don't use any proxy server, I just quickly picked this script up at https://intoli.com/blog/making-chrome-headless-undetectable/ because it seems to do the "handleCertificateError" you mentioned and I didn't remove the comments from the original script.

I guess https://chromedevtools.github.io/devtools-protocol/tot/Security/#method-handleCertificateError just handles invalid server certificates. It doesn't seem to help change chrome's behaviour in case the server sends a client certificate request.

I'm not sure then why it could be happening. I don't get any certificate error and the page renders just fine. 

You could add a console.log step in your handle certificate function to see if you are actually getting a cert error, or if there's other issue.

I have no idea why this only happens to me and only in headless mode. I added a console.log in the Security.certificateError callback, nothing was logged. This is all I get in wireshark when visiting the page:

Deleted: Screenshot from 2017-08-22 02-45-49.png 120 KB

	Screenshot from 2017-08-22 02-45-49.png 120 KB View Download

I've tried headless firefox in the meantime without issues. I am usually using http://webdriver.io and selenium to communicate with the browsers.

I have the same issue as mentioned above, and I used the command like this "./headless_shell --no-sandbox --disable-gpu --enable-logging --v=1 https://account.chsi.com.cn".

[0823/181016.905454:VERBOSE1:MemoryCache.cpp(166)] Evicting resource 0xb37aeff2f10
for "https://account.chsi.com.cn/" from cache
[0823/181016.906531:VERBOSE1:navigator_impl.cc(241)] Failed Provisional Load: https
://account.chsi.com.cn/, error_code: -110, error_description: , showing_repost_inte
rstitial: 0, frame_id: 2
[0823/181016.933449:VERBOSE1:navigation_controller_impl.cc(912)] Navigation finishe
d at (smoothed) timestamp 13147956616933442

My headless_shell version is `60.0.3090.0`. When I compiled version 62, I had the same issue. And on my ubuntu, using the command "./chrome --headless --disable-gpu https://account.chsi.com.cn", the error occurred again.

 

I am able to reproduce the same problem with https://account.chsi.com.cn/ as well.

 Issue 758452  has been merged into this issue.

Using a mitm proxy solved this problem.

Sounds like you're hiding it by letting the proxy handle the problematic ssl handshake.

I know this isn't a self-contained repro, but the following illustrates the issue (and that server is not scheduled to shut down anytime soon)

```
'use strict';

const puppeteer = require('puppeteer');

(async () => {
    // NB: ignoreHTTPSErrors gets us only so far. It still trips over the
    // client certificates in our certs (even when they're optional), or
    // perhaps something else:
    //
    // https://cockroach-adriatic-0001.crdb.io:8080
    const browser = await puppeteer.launch({ignoreHTTPSErrors: true});
    const page = await browser.newPage();
    await page.goto('https://cockroach-adriatic-0001.crdb.io:8080', {waitUntil: 'networkidle'});
    await page.pdf({path: 'out.pdf', format: 'A4'});

    browser.close();
})();
```

Perhaps it helps, though it doesn't print the error message.
This does though: https://github.com/GoogleChrome/rendertron/pull/80 and you get `{ message: 'net::ERR_SSL_CLIENT_AUTH_CERT_NEEDED' }` when trying the above URL.

Will be very useful if this is fixed. We use the headless mode extensively and are affected by this issues for some of our major use cases.
thanks!

Depending on this issue as well. Waiting for a fix here :)

Did anyone find a workaround for this? I don't actually need (or even want) the client-side SSL certificate provisioning to succeed, I just can't let the Page.open() fail on the ERR_SSL_CLIENT_AUTH_CERT_NEEDED error. I tried to generate a self-signed certificate with a CN and saN of the service (OpenShift in my case) to connect to, convert it to PKCS12 [1], and import it into the NSS DB so that chromium should be able to see it [2]:

❱❱❱ certutil -d sql:.pki/nssdb/ -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

127.0.0.2 - test                                             u,u,u

But this doesn't make any apparent difference, Chromium headless still refuses to connect (same error).


[1] https://stackoverflow.com/questions/21141215/creating-a-p12-file
[2] https://chromium.googlesource.com/chromium/src/+/master/docs/linux_cert_management.md

 Issue 754210  has been merged into this issue.

The same problem is reported in puppeteer
https://github.com/GoogleChrome/puppeteer/issues/963

Is there any update on this? It is a pretty significant issue I would say.
Researching this bug I found many discussions where people ended up switching to Firefox.

Andrey, mind taking a look?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8f0691b18dc5d941d5b6b3c67a483da02400670

commit c8f0691b18dc5d941d5b6b3c67a483da02400670
Author: Andrey Kosyakov <caseq@chromium.org>
Date: Thu Apr 05 15:05:12 2018

Headless: do not cancel requests if server wants a certificate

... just tell content to proceed with no certificate instead
of cancelling the request.

Bug:  757181 
Change-Id: Ib7db0d5bbbdf73dfd3bad4fb5827519eb8c0dbbd
Reviewed-on: https://chromium-review.googlesource.com/996954
Reviewed-by: Alex Clarke <alexclarke@chromium.org>
Commit-Queue: Andrey Kosyakov <caseq@chromium.org>
Cr-Commit-Position: refs/heads/master@{#548422}
[modify] https://crrev.com/c8f0691b18dc5d941d5b6b3c67a483da02400670/headless/lib/browser/headless_content_browser_client.cc
[modify] https://crrev.com/c8f0691b18dc5d941d5b6b3c67a483da02400670/headless/lib/browser/headless_content_browser_client.h
[modify] https://crrev.com/c8f0691b18dc5d941d5b6b3c67a483da02400670/headless/lib/headless_browser_browsertest.cc

Wow this was quick! Thanks!
When should we expect a fixed build that we could use?

This really depends on what channel you are. This site lets you track what releases have certain commit: http://omahaproxy.appspot.com/

Commit c8f0691b... initially landed in 67.0.3390.0

running with chrome 68.0.3440.106 problem persists.
any idea why?

this problem persists with macos version. 70.0.3538.77

I am still having issues on macos version 70.0.3538.77 as well. It fails to attach client certificate stored in Keychain on headless mode but the request works fine with headless-mode disabled. Should I submit a separate bug report for this?