New issue
Advanced search Search tips

Issue 856245 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Rebalance DUTs for release build queue

Project Member Reported by jbrandmeyer@chromium.org, Jun 25 2018

Issue description

Release builders are failing with "not enough DUTs" on the following builders:

daisy_spring-release
veyron_rialto-release
parrot_ivb-release


See also https://luci-milo.appspot.com/buildbot/chromeos/master-release/4344
 

Comment 1 by fried...@google.com, Jun 25 2018

Components: -Infra>Labs
This isn't an Infra>Labs type request.

Comment 2 by xixuan@chromium.org, Jun 25 2018

Components: Infra>Client>ChromeOS>Test
Daisy_spring:

Increase bvt pool to 4 to merely cover the test, (The required minimum is 4).

xixuan@xixuan0:~/chromiumos/chromeos-admin$ dut-status -p bvt -b daisy_spring
hostname                       S   last checked         URL
chromeos6-row2-rack16-host8    OK  2018-06-25 02:03:22  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack16-host8/1504706-verify/
chromeos6-row2-rack16-host10   OK  2018-06-25 09:56:21  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack16-host10/1506363-reset/
chromeos6-row2-rack16-host18   OK  2018-06-25 09:57:34  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack16-host18/1506365-reset/
chromeos6-row2-rack16-host16   OK  2018-06-25 09:41:09  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack16-host16/1506345-reset/


It's still dangerous as we expect it to be 6. But there's no available DUTs in pool:suites. File b/110780104 to ask lab team to fix it.


Comment 3 by xixuan@chromium.org, Jun 25 2018

Veyron_realto:


Make bvt pool have 6 healthy DUTs:

xixuan@xixuan0:~/chromiumos/chromeos-admin$ dut-status -p bvt -b veyron_rialto
hostname                       S   last checked         URL
chromeos2-row2-rack10-host2    OK  2018-06-25 06:19:04  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row2-rack10-host2/634041-reset/
chromeos2-row2-rack10-host4    OK  2018-06-25 06:01:22  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row2-rack10-host4/633987-reset/
chromeos2-row2-rack10-host5    OK  2018-06-24 21:17:28  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row2-rack10-host5/633010-repair/
chromeos2-row2-rack10-host7    OK  2018-06-24 21:17:28  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row2-rack10-host7/633011-repair/
chromeos2-row2-rack10-host8    OK  2018-06-24 21:17:28  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row2-rack10-host8/633012-repair/
chromeos2-row2-rack10-host9    OK  2018-06-24 21:17:29  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row2-rack10-host9/633013-repair/

Send broken DUTs to lab for fixing. File b/110779921.

Comment 4 by xixuan@chromium.org, Jun 25 2018

Cc: reinauer@chromium.org namyoon@chromium.org yllin@chromium.org
parrot_ivb:

We don't have enough healthy DUTs for it. File b/110780982 for fixing. Add sheriff to know this.

Comment 5 by xixuan@chromium.org, Jun 25 2018

Status: Fixed (was: Untriaged)
parrot_ivb is back:

xixuan@xixuan0:~/chromiumos/chromeos-admin$ dut-status -p bvt -b parrot_ivb
hostname                       S   last checked         URL
chromeos6-row2-rack4-host7     OK  2018-06-25 15:01:28  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack4-host7/890822-reset/
chromeos6-row2-rack4-host8     OK  2018-06-25 14:59:09  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack4-host8/890819-reset/
chromeos6-row2-rack4-host14    OK  2018-06-25 15:12:11  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack4-host14/890831-verify/
chromeos6-row2-rack4-host15    OK  2018-06-25 15:13:52  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack4-host15/890840-reset/
chromeos6-row2-rack4-host11    OK  2018-06-25 15:14:34  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack4-host11/890842-reset/
chromeos6-row2-rack4-host13    OK  2018-06-25 15:12:11  http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos6-row2-rack4-host13/890836-verify/

Mark this bug as fixed.

Sign in to add a comment