|Devserver should be skipped if check_health fails.|
|Project Member Reported by email@example.com, Feb 10 2017||Back to list|
daisy_skate CQ failed: https://uberchromegw.corp.google.com/i/chromeos/waterfall?builder=daisy_skate-paladin Debug log: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/100519721-chromeos-test/chromeos4-row9-rack5-host9/debug """ 02/10 09:44:02.135 ERROR| dev_server:0427| Devserver call failed: "http://100.108.1.152:8082/check_health?", timeout: 2.0 seconds, Error: ... 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond response.body = self.handler() File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/local/google/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 1510, in check_health apache_client_count = self._get_process_count('apache') File "/usr/local/google/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 1489, in _get_process_count 'pgrep -fc "%s"' % process_cmd_pattern, shell=True)) File "/usr/lib/python2.7/subprocess.py", line 566, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs) File "/usr/lib/python2.7/subprocess.py", line 710, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1215, in _execute_child errpipe_read, errpipe_write = self.pipe_cloexec() File "/usr/lib/python2.7/subprocess.py", line 1167, in pipe_cloexec r, w = os.pipe() OSError: [Errno 24] Too many open files """
Feb 10 2017,
Issue 690840 has been merged into this issue.
Feb 10 2017,
Retooling this into a fixit bug. Essentially if we hit an error when checking the health of the devserver, we should go on to the next one.
hmm, actually we did do "when devserver health checking failed, go on to the next one", but the log doesn't clearly show that. Will fix this bug by adding more logs. Another problem is this is actually a crash_devserver check fail, which won't be the real reason for a CQ failure.
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b273f6fab2b1fc34b60f091151dd1984acbfeb46 commit b273f6fab2b1fc34b60f091151dd1984acbfeb46 Author: xixuan <firstname.lastname@example.org> Date: Thu Jun 22 10:34:48 2017 autotest: Clearly log the process of devserver health check. As crbug.com/691009 shown, we don't have clear logs for how to iterate devserver health check. BUG= chromium:691009 TEST=Ran unittest. Change-Id: I155bdaaecb6b7f7e48abf028e7945cc950165f4e Reviewed-on: https://chromium-review.googlesource.com/543755 Commit-Ready: Xixuan Wu <email@example.com> Tested-by: Xixuan Wu <firstname.lastname@example.org> Reviewed-by: Xixuan Wu <email@example.com> [modify] https://crrev.com/b273f6fab2b1fc34b60f091151dd1984acbfeb46/client/common_lib/cros/dev_server.py
|► Sign in to add a comment|