New issue
Advanced search Search tips
Starred by 2 users
Status: Fixed
Owner:
Closed: Jun 22
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment
Devserver should be skipped if check_health fails.
Project Member Reported by waihong@chromium.org, Feb 10 2017 Back to list
daisy_skate CQ failed:
https://uberchromegw.corp.google.com/i/chromeos/waterfall?builder=daisy_skate-paladin

Debug log:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/100519721-chromeos-test/chromeos4-row9-rack5-host9/debug

"""
02/10 09:44:02.135 ERROR|        dev_server:0427| Devserver call failed: "http://100.108.1.152:8082/check_health?", timeout: 2.0 seconds, Error: 
...

        500 Internal Server Error
        The server encountered an unexpected condition which prevented it from fulfilling the request.
        Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/local/google/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 1510, in check_health
    apache_client_count = self._get_process_count('apache')
  File "/usr/local/google/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 1489, in _get_process_count
    'pgrep -fc "%s"' % process_cmd_pattern, shell=True))
  File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1215, in _execute_child
    errpipe_read, errpipe_write = self.pipe_cloexec()
  File "/usr/lib/python2.7/subprocess.py", line 1167, in pipe_cloexec
    r, w = os.pipe()
OSError: [Errno 24] Too many open files
"""


 
Comment 1 by sbasi@chromium.org, Feb 10 2017
Issue 690840 has been merged into this issue.
Comment 2 by sbasi@chromium.org, Feb 10 2017
Labels: Hotlist-Fixit
Summary: Devserver should be skipped if check_health fails. (was: daisy_skate CQ: Devserver call failed: "http://100.108.1.152:8082/check_health?", timeout: 2.0 seconds)
Retooling this into a fixit bug.

Essentially if we hit an error when checking the health of the devserver, we should go on to the next one.
Owner: xixuan@chromium.org
Status: Started
hmm, actually we did do "when devserver health checking failed, go on to the next one", but the log doesn't clearly show that. Will fix this bug by adding more logs.

Another problem is this is actually a crash_devserver check fail, which won't be the real reason for a CQ failure.
Project Member Comment 5 by bugdroid1@chromium.org, Jun 22
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b273f6fab2b1fc34b60f091151dd1984acbfeb46

commit b273f6fab2b1fc34b60f091151dd1984acbfeb46
Author: xixuan <xixuan@chromium.org>
Date: Thu Jun 22 10:34:48 2017

autotest: Clearly log the process of devserver health check.

As  crbug.com/691009  shown, we don't have clear logs for how to iterate
devserver health check.

BUG= chromium:691009 
TEST=Ran unittest.

Change-Id: I155bdaaecb6b7f7e48abf028e7945cc950165f4e
Reviewed-on: https://chromium-review.googlesource.com/543755
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/b273f6fab2b1fc34b60f091151dd1984acbfeb46/client/common_lib/cros/dev_server.py

Status: Fixed
Sign in to add a comment