New issue
Advanced search Search tips

Issue 836360 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

switch caroline-tot-chrome-pfq-informational builder to cave

Project Member Reported by ihf@chromium.org, Apr 24 2018

Issue description

caroline DUT availability in the lab is particularly scarce. The workaround in issue 835949 won't last long.

It is easier to get cave:
https://bugs.chromium.org/p/chromium/issues/detail?id=835949#c9

So consider
1) switch the caroline builder to use cave.
2) require less DUTs to work on the tests.
3) increase suite timeouts
4) run less tests
5) get away from pool based scheduling. It is not clear why informational builders need a pool, of this size and then not automatically maintained.
 
Cc: warx@chromium.org minch@chromium.org
Owner: warx@chromium.org
Passing onto the next gardener.

Comment 2 by warx@chromium.org, May 7 2018

Cc: -warx@chromium.org -minch@chromium.org malaykeshav@chromium.org
Owner: sammiequon@chromium.org
Didn't figure out where to go. Next gardener might contact ihf@ for code points.
Status: Assigned (was: Untriaged)

Comment 4 by ihf@chromium.org, May 8 2018

Cc: sammiequon@chromium.org
Owner: ihf@chromium.org
Addressing 2) less DUTs here
https://chromium-review.googlesource.com/#/c/chromiumos/chromite/+/1049487
Project Member

Comment 5 by bugdroid1@chromium.org, May 8 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8cf7ea75d5ffc48222bec84907256377b6f2e587

commit 8cf7ea75d5ffc48222bec84907256377b6f2e587
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Tue May 08 23:39:11 2018

chromeos_config: drop informational min_duts to 1.

Allow the continuous pools to do their job even with 1 DUT.

Currently the workload looks like a single DUT should be able
to cover all testing. So no need to provision 4 DUTs if one
can do.

But even if one DUT were not enough to cover informational
testing, it is better in times when working DUTs are scarce
in the lab for the suite to sometimes time out (infra flake)
than to refuse to start testing (permanent infra failure).

BUG=chromium:836360
TEST=look at caroline graphs
https://viceroy.corp.google.com/chromeos/dut_utilization?board=caroline&model=&pool=managed%3Acontinuous&status=Running&topstreams=40&duration=108858&mdb_role=chrome-infra&refresh=-1&utc_end=1525119484
Maybe, maybe minnie needs a minimum of 2, but I think the suite
grabs all idle DUTs in the pool anyways (which would be 6 here).
https://viceroy.corp.google.com/chromeos/dut_utilization?board=veyron_minnie&model=&pool=managed%3Acontinuous&status=Running&topstreams=40&duration=54429&mdb_role=chrome-infra&refresh=-1&utc_end=1525313968

Change-Id: I5864e1c41d356b4acf5d6722cf86c08fe258b18b
Reviewed-on: https://chromium-review.googlesource.com/1049487
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Pohsien Wang <pwang@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/8cf7ea75d5ffc48222bec84907256377b6f2e587/cbuildbot/config_dump.json
[modify] https://crrev.com/8cf7ea75d5ffc48222bec84907256377b6f2e587/cbuildbot/chromeos_config.py

Comment 6 by pwang@chromium.org, May 8 2018

Cc: pwang@chromium.org
5) should address the issue. What's the reason prevents us from merging the continuous pool with pool:suite?

Comment 7 by ihf@chromium.org, May 9 2018

continuous pool has a bit better service, as it takes devices from suite when they break.

Comment 8 by ihf@chromium.org, May 9 2018

At least that is the theory. One could merge continuous pool into another pool, like bvt or cq. But this is a more long term effort (ideally by writing a new scheduler).

I will look again if we can switch to cave. And if we can reduce some continuous pool sizes a little.
Cc: -sammiequon@chromium.org -malaykeshav@chromium.org derat@chromium.org
forward to next gardener

Comment 10 by la...@chromium.org, Jun 15 2018

Components: -Infra>Client>ChromeOS>Build Infra>Client>ChromeOS>Test

Comment 11 by derat@chromium.org, Jun 15 2018

Cc: -derat@chromium.org

Sign in to add a comment