New issue
Advanced search Search tips

Issue 750912 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Aug 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Increase chromiumos-sdk builder timeout

Project Member Reported by manojgupta@chromium.org, Jul 31 2017

Issue description

Some of the tryjobs fail as they hit the timeout limit of 16 hours on chromiumos-sdk. 
The main builder is *very close* to 16 hours limit e.g.
 This build took 15 hrs, 49 mins, 13 sec (https://uberchromegw.corp.google.com/i/chromiumos/builders/chromiumos-sdk/builds/8054)

The tryserver job list already has 2 jobs in the last week where the timeout was reached
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chromiumos-sdk

Please increase the timeout by a couple of hours to avoid future fails.
 
https://uberchromegw.corp.google.com/i/chromiumos/builders/chromiumos-sdk/builds/8051 took 15 hrs, 53 mins, 24 secs. 
Just 6 more minutes and the builder would have hit timeout.
Cc: dgarr...@chromium.org
dgarrett@ Is this the place where 16 hours limit is specified https://cs.corp.google.com/chromium_build/scripts/master/factory/chromeos_factory.py?dr=C&file:%5Escripts/+package:%5Echromium_build$&l=32
That seems likely. That number also appears to be global across all ChromeOS builds on all waterfalls.

Though, I'm not quite clear on the difference between 'timeout' and 'max_time' or why both exist.


Also related:

For the chromeos waterfall:

https://cs.corp.google.com/chromium_build_internal/masters/master.chromeos/master.cfg?rcl=26dd0a6a24e7d0271e0d39b636dbd2ef4cfa624a&l=136


And the chromiumos waterfall:

https://cs.corp.google.com/chromium_build/masters/master.chromiumos/master.cfg?rcl=4df95728c0f93f35524533569638745ababbfc20&l=91

But I don't see a similar function in the tryserver waterfall to hook into:

https://cs.corp.google.com/chromium_build/masters/master.chromiumos.tryserver/master.cfg

I'm game for bumping the global default.
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ee6b2cf3f51a2029b8a0316ab93f806ab3322b1d

commit ee6b2cf3f51a2029b8a0316ab93f806ab3322b1d
Author: Don Garrett <dgarrett@google.com>
Date: Wed Aug 02 23:43:16 2017

chromeos_factory: Increase default build max_time to 20 hours.

We have some builders bumping up against the 16 hour limit, so raise
it to 20 hours.

BUG= chromium:750912 

Change-Id: Iaaf5d49c50fd951a7eb08096758d96a45939c414
Reviewed-on: https://chromium-review.googlesource.com/598692
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/ee6b2cf3f51a2029b8a0316ab93f806ab3322b1d/scripts/master/factory/chromeos_factory.py

Owner: manojgupta@chromium.org
Status: Started (was: Untriaged)
So, in theory, it's fixed. Can you please confirm?
Does it need a waterfall restart?
Oh... good catch, yes.
Status: Verified (was: Started)
New tryjobs have a 20 hr timeout, closing.
Project Member

Comment 11 by bugdroid1@chromium.org, Aug 3 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/916358b9a8039f4c75ae17c1d5578be2407c3e5d

commit 916358b9a8039f4c75ae17c1d5578be2407c3e5d
Author: Manoj Gupta <manojgupta@google.com>
Date: Thu Aug 03 22:13:56 2017

chromiumos-sdk: Update timeout to 18 hrs from 16 hrs.

Current build time is too close to the current limit of 16 hours.
Increase timeout to 18 hours to get some more breathing room.

BUG= chromium:750912 
TEST=chromeos_config unittest pass.

Change-Id: Ide47ea91621b3c4f608071f1dc513ac35a4248ca
Reviewed-on: https://chromium-review.googlesource.com/595215
Commit-Ready: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/916358b9a8039f4c75ae17c1d5578be2407c3e5d/cbuildbot/config_dump.json
[modify] https://crrev.com/916358b9a8039f4c75ae17c1d5578be2407c3e5d/cbuildbot/chromeos_config_unittest.py
[modify] https://crrev.com/916358b9a8039f4c75ae17c1d5578be2407c3e5d/cbuildbot/chromeos_config.py

Components: -Infra>Client>ChromeOS

Sign in to add a comment