Monorail Project: chromedriver Issues People Development process History Sign in
New issue
Advanced search Search tips
Issue 1772 Chromedriver running against canary chrome w/ headless flag requires XVFB for sendKey interactions
Starred by 50 users Reported by malone.s...@gmail.com, Apr 19 Back to list
Status: Fixed
Owner:
Closed: Jun 26
Cc:

Blocking:
issue chromium:604324


Show other hotlists

Hotlists containing this issue:
Hotlist-1
Hotlist-1
Watching-Issues


Sign in to add a comment
Issue Description:
Using a canary build of chrome w/ the headless flag, sendkeys doesn't work if driving with chromedriver. It'd be super nice to have a solution for this because otherwise the headless flag is wayyy less useful.

Steps to reproduce:
I'm using docker for this, so bear with me.

Throw all of this into a terminal:
1) docker run --entrypoint=bash -it justinribeiro/chrome-headless:latest
Inside the docker container from here on out
2) apt-get update && apt-get install wget unzip python python-pip
3) LATEST=$(wget -q -O - http://chromedriver.storage.googleapis.com/LATEST_RELEASE)
4) wget http://chromedriver.storage.googleapis.com/$LATEST/chromedriver_linux64.zip
5) unzip chromedriver_linux64.zip && ln -s $PWD/chromedriver /usr/local/bin/chromedriver
6) echo '#\!/usr/bin/env bash' > /usr/bin/google-chrome 
7) echo '/opt/stickmanventures/chrome-headless/headless_shell --headless --no-sandbox "$@"' >> /usr/bin/google-chrome
8) chmod 755 /usr/bin/google-chrome
9) pip install selenium
10) python
11) follow the python steps found here: https://sites.google.com/a/chromium.org/chromedriver/getting-started


Here's the output of the python steps:

root@9c6a5969d51b:/# python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> from selenium import webdriver
>>> 
>>> driver = webdriver.Chrome()
>>> driver.get('http://www.google.com/xhtml');
>>> search_box = driver.find_element_by_name('q')
>>> search_box.send_keys('ChromeDriver')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 349, in send_keys
    'value': keys_to_typing(value)})
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
    return self._parent.execute(command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 249, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 193, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: an X display is required for keycode conversions, consider using Xvfb
  (Session info: content shell=)
  (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.9.4-moby x86_64)
 
Project Member Comment 1 by gmanikpure@chromium.org, Apr 19
Labels: Needs-Feedback
Chrome Canary builds are not available for Linux platform. Did you test on Chrome Dev build? What is the version of chrome browser ?
Sorry, it was someone else's early build of the headless project that didn't have the version flag. Just realized that the google-chrome-unstable has the headless flag these days, so I used that.

root@aa791cabef8b:/# google-chrome --version
Google Chrome 59.0.3071.9 dev

To install in that container, you essentially need to take the steps I outlined above and skip steps 6-8, but instead do...

6) sh -c 'echo "deb [arch=amd64] https://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
7) apt-get install apt-transport-https && apt-get update && apt-get install google-chrome-unstable

I think this should work, but haven't tested it yet from the start of those instructions.

And then when you start the chromedriver in python, you'll need to add in a set of options for the headless chrome:

>>> from selenium.webdriver.chrome.options import Options
>>> 
>>> chrome_options = Options()
>>> chrome_options.add_argument("--disable-gpu")
>>> chrome_options.add_argument("--no-sandbox")
>>> chrome_options.add_argument("--headless")
>>> driver = webdriver.Chrome(chrome_options=chrome_options)


It still gives the same error though.
Project Member Comment 3 by gmanikpure@chromium.org, Apr 19
I tried the test without docker setup in following manner :-

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.add_argument('--headless')
opts.add_argument('--no-sandbox')
opts.add_argument('--disable-gpu')
opts.binary_location = "/usr/bin/google-chrome-unstable"

browser = webdriver.Chrome(chrome_options=opts, executable_path='/path/to/chromedriver')
browser.get('http://google.com')
elem = browser.find_element_by_name('q')
elem.send_keys('testing')
browser.quit()

and it works fine with Chromedriver v2.29 on Chrome v59.0.3071.9 dev.

Do you know if this happens only with docker setup? Can you also please provide chromedriver verbose logs?
Were you in a "headless environment"? If I add XVFB (or I imagine, if I run on mac/windows/another GUI environment), it does work. I guess I'm not sure where this should land, on the Chromedriver side, or the headless chrome side. I'll get you the verbose logs tomorrow. Our hope was that we wouldn't have to have the overhead of XVFB if chrome doesn't need to render anything. I know with just the chrome debugging protocol, we can do sendkey actions in a headless environment, so it just seemed like it should be possible for chromedriver.

(see: https://github.com/knq/chromedp/tree/master/examples/headless, which is running headless chrome in docker using the debugging protocol and is able to send keys)
Blocking: chromium:604324
#4 I am running into the same problems you are.

Booted up a minimal CentOS7 environment and just ran into this issue when trying to use sendKeys() (Java/Selenium). I'm using ChromeDriver v2.29.

I did see some material on https://github.com/SeleniumHQ/selenium/wiki/Advanced-User-Interactions#native-events-versus-synthetic-events re: Chrome's usage of WebKit Events vs "Raw Events". I assume this may help me towards using sendKeys(), but there is no information on how to toggle the usage of WebKit events.

Is the current workaround to just run an instance of Xvfb? Echoing #4, I had hoped that we wouldn't have to use Xvfb.
Ran into this with the following versions:

OS: Ubuntu 14.04
Chromium 60.0.3097.0
Chromedriver: 2.29.464589 (7cda776b17ae612ddc45b948e4a350a5295a1b5b)

Using ""selenium-webdriver" gem with Ruby. I have attached the script. The error comes only at "element.send_keys" call.

Full error message:

Selenium::WebDriver::Error::UnknownError: unknown error: an X display is required for keycode conversions, consider using Xvfb
  (Session info: headless chrome=60.0.3097.0)
  (Driver info: chromedriver=2.29.464589 (7cda776b17ae612ddc45b948e4a350a5295a1b5b),platform=Linux 4.4.0-64-generic x86_64)
	from /home/deploy/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/selenium-webdriver-3.4.0/lib/selenium/webdriver/remote/response.rb:69:in `assert_ok'
chromeheadless.rb
420 bytes View Download
Cc: samu...@chromium.org irisu@chromium.org jzfeng@chromium.org
Looks like we should maybe switch from keycode_text_conversion_x.cc to keycode_text_conversion_ozone.cc in headless mode?
Project Member Comment 9 by gmanikpure@chromium.org, May 11
Cc: johnchen@chromium.org
+johnchen@

John is the new lead for Chromedriver.
I have the same issue Ubuntu 16.04.2

Google Chrome: 59.0.3071.47 beta
Chrome Driver: 2.29.461571
Selenium: 3.4.1

>>> from selenium import webdriver
>>> from selenium.webdriver.chrome.options import Options
>>> opts = Options()
>>> opts.add_argument('--headless')
>>> opts.add_argument('--no-sandbox')
>>> opts.add_argument('--disable-gpu')
>>> browser = webdriver.Chrome(chrome_options=opts, executable_path='/usr/local/bin/chromedriver')
>>> browser.get('http://google.com')
>>> browser.title
'Google'
>>> browser.find_element_by_name('q').send_keys('test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 349, in send_keys
    'value': keys_to_typing(value)})
  File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
    return self._parent.execute(command, params)
  File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in execute
    self.error_handler.check_response(response)
  File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: an X display is required for keycode conversions, consider using Xvfb
  (Session info: headless chrome=59.0.3071.47)
  (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.4.0-75-generic x86_64)

Cc: -irisu@chromium.org
Owner: irisu@chromium.org
Status: Assigned
irisu@, this was also mentioned in the comments of https://developers.google.com/web/updates/2017/04/headless-chrome

Have you been able to repro this?
I have been able to reproduce the error.

I am able to avoid the error if I compile chrome and chromedriver with the use_ozone = true build flag. Unfortunately the choice between keycode_text_conversion_x.cc vs keycode_text_conversion_ozone.cc is made at compile time, whereas the choice to run chrome in headless mode is done at runtime.

I will have to look into this more to find a good solution.
Has there been any progress with this?
I can confirm this is happening too on Ubuntu Server headless with Chromedriver 2.29 and Chrome 59. This is a dealbreaker for using Chrome headless right now :(
> Is the current workaround to just run an instance of Xvfb? Echoing #4, I had hoped that we wouldn't have to use Xvfb.

Actually I have seen workarounds where people were using JavaScript to fill the form fields instead of using `sendKey`. If you're looking for an immediate fix to get rid of Xvfb that's probably your best shot.
I was able to re-compile chrome and chromedriver with use_ozone = true

This eliminated the send_keys errors, but at this point I'm still unclear if the timeout errors are my site or my build  :)
Same issue when using chromedriver 2.30.

Like previously stated, I was hoping I could use headless without Xvfb.
After all, its stated in the FAQ of this blogpost that Xvfb will not be required at all.

https://developers.google.com/web/updates/2017/04/headless-chrome

This blocking the use of headless chrome for me
Current implementation of chromedriver requires an X display to translate text into keystrokes on Linux, thus the requirement for Xvfb when a real X display isn't available.

irisu@chromium.org: you're currently the owner of this bug. Could you please give an update on the progress of finding a fix? Thanks.
Any updates on this topic?
I've confirmed that use_ozone compile-time option does work for ruby / capybara and the send_keys error.

It's a solid workaround for now.
can you please advice command line which will rebuild chrome for ubuntu 16
The upstream build instructions are documented [1]. The instructions were written for Ubuntu, so it should be relatively straightforward. 

[1] https://www.chromium.org/developers/how-tos/get-the-code
Hi.

Any update on that issue ? It makes all that effort put into headless version useless :/

Have a nice day.
I agree, it makes the headless version useless... It's a little alarming that the issue owner hasn't checked on this issue for 15 days or so

Owner:	█ irisu@chromium.org
Last visit 15 days ago
Hi,

For Heroku CI[1] we are hoping to only support Headless Chrome, rather than with Xvfb. Once this issue has been addressed I believe we can retire the Xvfb hacks that we have.

Thanks,
Guðmundur Bjarni

[1]: https://blog.heroku.com/heroku-ci-now-available
Hi,

When can we expect an update on this issue?  

Using Xvfb hacks makes headless chrome useless.  Using javascript hacks to "send keys" defeats the purpose of using chromedriver in the first place.  Disappointed that headless chrome is still not viable.
Labels: -Needs-Feedback ChromeDriver-2.31
Status: Fixed
Fixed with https://codereview.chromium.org/2952383002.
Hi.

When can we expect that released ?

Thank you and have a nice day.
The fix will be included in ChromeDriver 2.31. Its release date hasn't been determined yet.
🙏 Thank you so much! We'll keep an eye out for it then.
Agreed. Thank you very much for the effort put in to fix this. 
Excellent job, thank you very much! Really looking forward to the release!
Indeed. Thanks to jzfeng@chromium.org for Making the fix!
Hi.

I thought issue is with chrome itself and not with chromedriver ?

Thank you and have a nice day.
The issue is in ChromeDriver. As its name implies, the sendKey command sends simulated key strokes to Chrome, and ChromeDriver is responsible for converting the input text string into corresponding keystrokes to send to Chrome. On Linux, ChromeDriver uses X display to do the conversion, and fails when no X display is available. The change in comment #28 adds additional code in ChromeDriver to do the text to keystroke conversion when X display isn't available.
@johnchen, is there anywhere we can get a nightly build of chromedriver that includes the patch? I have tried building it myself, but I can't seem to get a build that isn't dynamically linked to other Chrome libraries that are not shipped with the official Chrome release debs. Alternatively, can you tell me what I need to do to build it myself? I have followed [1] but not got a build of chromedriver that I can use inside my container due to the aforementioned dynamic linking issue.

[1] https://chromium.googlesource.com/chromium/src/+/master/docs/linux_build_instructions.md
I have a dockerfile that can help with making a build that isn’t linked. https://gist.github.com/tabiodun/e4204ab83b3b7cfd2d28b478cf441162
@abiodun.tobi: Thanks for providing the dockerfile.

@etienne: There is no official nightly build of chromedriver, so unfortunately it's necessary to make your own build until the next release. If you make a release build of chromedriver (by adding --args="is_debug=false" option to gn command), it will have no other dependencies. Only debug build of chromedriver depends on other Chrome binaries.
Has anybody had success building chromedriver for use with CentOS 7? I have managed to build an executable using @abiodun.tobi's Dockerfile, but unfortunately it appears to require a newer glibc version than is available:

./chromedriver-2.31-dev: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./chromedriver-2.31-dev)

I ran into the same issue trying to compile directly on a CentOS 7 machine (albeit with the error coming during the compilation stage). 
Thank you for the fix. Is there is a release date for 2.31?
The release date for ChromeDriver 2.31 hasn't been determined yet.
I rented a huge server on AWS and cut a 2.31 release with this patch in it for `linux_64`. The binary is available on github here https://github.com/davidthornton/chromedriver-2.31
You the man, David. Just revisited my test from above and the send_keys passed with your compiled driver. Appreciate it! I will use this until the actual release. No more xvfb!!!
Comment 46 by hanswest...@gmail.com, Jul 20 (3 days ago)
Would this fix alone merit a quick release of 2.31? Seems that there's quite a few people awaiting it...
Comment 47 by jessedic...@gmail.com, Jul 20 (3 days ago)
I noticed their releases can be month's apart, which seem silly to me. I would think as soon as you have something that provides some kind of value, and if it does not break backwards compatibility, and it has been tested to satisfaction, then release it right away. Actually the code should always be kept in a releasable state so releases are not a big deal and happen frequently. Although it seems this is a beast to build, which may at least partially explain their slow releases. But really I am not in the know of how or why they really do things, just speculating and sharing ideas.
Comment 48 by hanswest...@gmail.com, Jul 20 (2 days ago)
I guess they have their reasons for moving at the pace they do. Fingers crossed...
Comment 49 by johnchen@chromium.org, Jul 20 (2 days ago)
The main reason we aren't able to make ChromeDriver releases as often as we (or the users) would like, is the process isn't fully automated and involves too many manual steps. We understand that this bug fix is important to many users, so we're kicking off the release process now, and we should have a release in a few days if we don't encounter any problems.
Sign in to add a comment