cyan-chrome-pfq failed HWtest due to timeout |
|||||||||
Issue descriptionSee the latest build: https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq/builds/707 Selected Log info: Autotest instance: cautotest 02-01-2017 [03:22:22] Submitted create_suite_job rpc 02-01-2017 [03:22:33] Created suite job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=99175474 @@@STEP_LINK@Link to suite@http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=99175474@@@ --create_and_return was specified, terminating now. Will return from run_suite with status: OK 03:22:35: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmp2uXgUr/tmpNywxZW/temp_summary.json --raw-cmd --task-name cyan-chrome-pfq/R58-9240.0.0-rc1-arc-bvt-cq --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 18600 --io-timeout 18600 --hard-timeout 18600 --expiration 1200 '--tags=priority:PFQ' '--tags=suite:arc-bvt-cq' '--tags=build:cyan-chrome-pfq/R58-9240.0.0-rc1' '--tags=task_name:cyan-chrome-pfq/R58-9240.0.0-rc1-arc-bvt-cq' '--tags=board:cyan' -- /usr/local/autotest/site_utils/run_suite.py --build cyan-chrome-pfq/R58-9240.0.0-rc1 --board cyan --suite_name arc-bvt-cq --pool bvt --num 3 --file_bugs True --priority PFQ --timeout_mins 250 --retry True --minimum_duts 3 --suite_min_duts 3 --offload_failures_only False -m 99175474 @@@STEP_FAILURE@@@ 04:10:08: ERROR: Timeout occurred- waited 15614 seconds, failing. Timeout reason: This build has reached the timeout deadline set by the master. Either this stage or a previous one took too long (see stage timing historical summary in ReportStage) or the build failed to start on time. 04:10:09: INFO: Running cidb query on pid 32470, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x4f834d0>
,
Feb 1 2017
Anyone seen on this timeout error?
,
Feb 1 2017
So, this is a "master timeout" of 15,000 seconds or about 4h 20m. For such failures it's a good idea to look at the build times on viceroy: https://viceroy.corp.google.com/chromeos/build_details?build_config=cyan-chrome-pfq&build_number=707&experimental=False&duration=1d&refresh=-1 and compare to earlier builds, to see what stands out.
,
Feb 1 2017
semenzato@: Thanks for the suggestion! Compared with earlier builds, seems the "HWTest[sanity]" Stage takes more than half an hour than usual (though it still passed at last).
,
Feb 1 2017
The graph of earlier builds makes me worry a bit. Many of the green ones are not much shorter than the ones that timed out.
,
Feb 1 2017
Yes, especially cyan-chrome-pfq/692 which was passed but has duration of 15896 seconds, while the current one cyan-chrome-pfq/707 timed out at 15993 seconds. How do we determine the timeout number?
,
Feb 1 2017
I am not sure where those timeouts are set, or even if the time shown in that dashboard is the same as the time used to device when timing out. You would have to ask, or look at the code.
,
Feb 1 2017
Since it's infra related issue, assign to Infra deputy for further investigation or find an better owner.
,
Feb 1 2017
A new Chrome PFQ was kicked off. Monitoring...
,
Feb 1 2017
+hidehiko, nya fyi https://viceroy.corp.google.com/chromeos/build_details?build_config=cyan-chrome-pfq&build_number=702 build took 1h51m ? and arc-bvt-cq test only had 48 minutes remaining. (but that's a pretty normal condition, I wonder why this failed in particular)
,
Feb 1 2017
Uekawa, can you point to the source that sets the timeout for arc-bvt-cq? My impression is that these jobs are killed by the master timeout, not the step-specific timeout. Thanks!
,
Feb 2 2017
Even in passed cases, BuildPackage stage always took ~2h to finish, and 90% of time it is building chromeos-chrome. Building chrome should be done in <30m. I guess we do not use goma? Why?
,
Feb 2 2017
FYI, I found Issue 279618 about enabling goma in Chrome OS build.
,
Feb 3 2017
Update: Recent builds (~10 runs) passed without timeout error. However, looking at the build time dashboard on viceroy: https://viceroy.corp.google.com/chromeos/build_details?build_config=cyan-chrome-pfq&build_number=707&experimental=False&duration=1d&refresh=-1. It doesn't seem like the green ones are much shorter than the ones that timed out. And this timeout error also seems happen once in a while (see https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-chrome-pfq?numbuilds=100, e.g., build#707, build#702, build#682). Enabling goma as in Issue 279618 will definitely help decrease this type of error in the future.
,
Feb 3 2017
See also issue 611139 . I just did a random audit, and with ARC now part of the build and test stages, we are cutting things pretty close to the master timeout in general. We need to prioritize either making goma work (issue 279618) or increase the master timeout.
,
Feb 6 2017
We did a mass wipe of build slaves, which meant their next build needed to repo sync from scratch. Could have been a transient timeout.
,
Feb 9 2017
,
Apr 27 2017
,
Jun 29 2017
|
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by x...@chromium.org
, Feb 1 2017