New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 647303 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Sep 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Several PFQ failures due to timeouts

Project Member Reported by afakhry@chromium.org, Sep 15 2016

Issue description

Currently, the following builders are failing due to timeouts in the HWTests phase:

cyan-chrome-pfq: https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq/builds/173
falco-chrome-pfq: https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-chrome-pfq/builds/9122
lumpy-chrome-pfq: https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-chrome-pfq/builds/9122
tricky-chrome-pfq: https://uberchromegw.corp.google.com/i/chromeos/builders/tricky-chrome-pfq/builds/2355

The HWTest phase seems to take almost 2 hours [See attached screenshot for cyan].
 
Screenshot from 2016-09-15 10:12:22.png
56.1 KB View Download

Comment 1 by x...@chromium.org, Sep 15 2016

Update: The lumpy-chrome-pfq[1] and cyan-chrome-pfq[2] are now green. The other two falco-chrome-pfq[3] and tricky-chrome-pfq[4] are still broken, but seems due to a different reason (HWTest abort error). Besides these two failed builders, veyron_minnie_chrome_pfq[5] is also failed (HWTest timeout error). They all failed HWTest, though the error messages are not exactly the same. It's hard to tell if they're caused by the same root cause. 

[1] https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-chrome-pfq/builds/9123
[2] https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq/builds/174
[3] https://uberchromegw.corp.google.com/i/chromeos/builders/falco-chrome-pfq/builds/4740
[4] https://uberchromegw.corp.google.com/i/chromeos/builders/tricky-chrome-pfq/builds/2356
[5] https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/177
Looking at a few of these from this master build:
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/3383

All 3 timed out even though looking at autoserv.INFO all tests appear to pass.

There is the following warning though:
parallel_simple was called with an empty arglist, did you forget to pass in a list of machines?

The failures in comment #1 all timed out in HWTest [sanity].

Three of them had autoserv logs that look like this:

09/14 23:51:07.186 INFO |        server_job:0129| INFO	----	Start sanity	timestamp=1473922267	localtime=Sep 14 23:51:07	
09/14 23:51:07.186 DEBUG|             suite:0825| Scheduling dummy_PassServer.sanity
09/14 23:51:07.633 DEBUG|             suite:1079| Adding job keyval for dummy_PassServer.sanity=77036127-chromeos-test
09/14 23:51:07.633 DEBUG|     dynamic_suite:0554| Waiting on suite.
09/15 02:54:42.390 INFO |        server_job:0129| START	----	cyan-chrome-pfq/R55-8803.0.0-rc1/sanity/dummy_PassServer.sanity	timestamp=1473933282	localtime=Sep 15 02:54:42	

There is a 3 hour delay after "Waiting on suite."

The 4th appeared to run successfully but the builder thought it failed and aborted. I have seen that elsewhere (e.g. in comment #2).

Owner: steve...@chromium.org
Status: Assigned (was: Untriaged)
The arm-generic timeouts are this:  issue 648436 

Status: WontFix (was: Assigned)
I don't see any consistent pattern to timeouts at this point, there are several open time out related issues.

Sign in to add a comment