Filing for tracking, assigning to infra for triage.

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/98949075-chromeos-test/chromeos4-row11-rack7-host2/debug/

01/30 20:57:28.297 ERROR|   logging_manager:0626| {'builds': "{'cros-version': 'reks-release/R58-9236.0.0'}", 'job_started': 1485838179, 'hostname': 'chromeos4-row11-rack7-host2', 'status_version': 1, 'label': 'reks-release/R58-9236.0.0/bvt-perbuild/cheets_GTS.4.1_r1.GtsPlacementTestCases', 'parent_job_id': 98936366, 'drone': 'chromeos-server69.hot.corp.google.com', 'user': 'chromeos-test', 'suite': 'bvt-perbuild', 'job_queued': 1485834545, 'retry_original_job_id': 98937175, 'experimental': 'False', 'build': 'reks-release/R58-9236.0.0'}

[...]

01/30 20:57:29.600 ERROR|        dev_server:0404| Devserver call failed: "http://100.108.1.152:8082/check_health?", timeout: 2.0 seconds, Error: 


    
    500 Internal Server Error
    
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    

    
        500 Internal Server Error
        The server encountered an unexpected condition which prevented it from fulfilling the request.
        Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/local/google/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 1510, in check_health
    apache_client_count = self._get_process_count('apache')
  File "/usr/local/google/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 1489, in _get_process_count
    'pgrep -fc "%s"' % process_cmd_pattern, shell=True))
  File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1215, in _execute_child
    errpipe_read, errpipe_write = self.pipe_cloexec()
  File "/usr/lib/python2.7/subprocess.py", line 1167, in pipe_cloexec
    r, w = os.pipe()
OSError: [Errno 24] Too many open files

    
    Powered by CherryPy 3.2.2
    
    


01/30 20:57:31.609 ERROR|        dev_server:0404| Devserver call failed: "http://100.108.1.59:8082/check_health?", timeout: 2.0 seconds, Error: retry exception (label="make_call"), timeout = 2s
01/30 20:57:31.639 ERROR|        dev_server:0404| Devserver call failed: "http://172.24.190.211:8082/check_health?", timeout: 2.0 seconds, Error: retry exception (label="make_call"), timeout = 2s
01/30 20:57:31.683 ERROR|        dev_server:0404| Devserver call failed: "http://172.25.61.81:8082/check_health?", timeout: 2.0 seconds, Error: retry exception (label="make_call"), timeout = 2s
01/30 20:57:31.722 ERROR|        dev_server:0404| Devserver call failed: "http://172.25.66.216:8082/check_health?", timeout: 2.0 seconds, Error: retry exception (label="make_call"), timeout = 2s
01/30 20:57:31.747 DEBUG|             retry:0155| Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0)