New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 829898 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Add eve to Chrome ToT informational builders

Project Member Reported by jamescook@chromium.org, Apr 6 2018

Issue description

Eve is our lead device and has different graphics hardware than earlier versions. It also has more assistant features. We've been burned multiple times by graphics startup issues in chrome that only affect eve and related boards. Latest is issue 824809. It would be super-helpful to have narrower chrome revision ranges and be able to see if fixes worked without waiting for a Chrome PFQ uprev.

https://uberchromegw.corp.google.com/i/chromeos.chrome/waterfall?reload=300

If we need more capacity maybe this could replace caroline.

 
Cc: bpastene@chromium.org dpranke@chromium.org ihf@chromium.org achuith@chromium.org
Labels: -Pri-3 Pri-2
+1

Labels: Hotlist-CrOS-Gardener Build-PFQ-Failures

Comment 3 by ihf@chromium.org, Apr 6 2018

Sounds like the gardener should start preparing a change? Normally all that is needed to add a builder is
~/trunk/chromite/cbuildbot/chromeos_config.py
But IIRC informational builders were special and needed help from Chrome infra..

Once we have a builder we can allocate DUTs.

Luckily eve has a lot of DUTs in the lab, unluckily a lot of them look dead at the moment.
Cc: dgarr...@chromium.org
Adding new builders is not really a gardening task, we definitely need help from infra for this.

+dgarrett@ - Would it be easier to just add this to the new system? As long as the builder runs continuously and we can get the bisect info, it's not critical that the new builder is on the current informational waterfall.

I can add it to the new system easily enough, but until we move the others, gardeners will have multiple UIs to look at to see the current state of things.
Also, I'm not yet sure if it will generate SoM alerts from swarming.
Long term we can address all that, but the most important thing is to get the builder configured and running.

IIRC adding a builder to the informational waterfall isn't too difficult and something that a gardener can probably do.

I've prepared https://crrev.com/i/603610, which will start running this on swarming. That does not prevent running it on the chromeos.chrome waterfall.
Project Member

Comment 9 by bugdroid1@chromium.org, Apr 7 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e1982424f85ed7b793670de05c9ad66863aa282b

commit e1982424f85ed7b793670de05c9ad66863aa282b
Author: Don Garrett <dgarrett@google.com>
Date: Sat Apr 07 05:00:05 2018

chromeos_config: Move an eve to swarming.

eve-tot-chrome-pfq-informational. This is only bookkeeping... so far.

Also, update how we generate the chromium-pfq-informational swarming
configs.

BUG= chromium:829898 
TEST=chromeos_config_unittest

Change-Id: If81ac7b658ade36e87e0c0e8f5cc495147be40a3
Reviewed-on: https://chromium-review.googlesource.com/1000409
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/e1982424f85ed7b793670de05c9ad66863aa282b/cbuildbot/config_dump.json
[modify] https://crrev.com/e1982424f85ed7b793670de05c9ad66863aa282b/cbuildbot/waterfall_layout_dump.txt
[modify] https://crrev.com/e1982424f85ed7b793670de05c9ad66863aa282b/cbuildbot/chromeos_config.py

Project Member

Comment 10 by bugdroid1@chromium.org, Apr 7 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/65fdfe646c39ff6f2ec27c76e31abc923a8130a3

commit 65fdfe646c39ff6f2ec27c76e31abc923a8130a3
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Apr 07 07:59:29 2018

Roll src/third_party/chromite/ 188669bb7..f10b48279 (4 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/188669bb7526..f10b482799b3

$ git log 188669bb7..f10b48279 --date=short --no-merges --format='%ad %ae %s'
2018-04-04 djkurtz grunt: mark grunt-paladin important
2018-04-06 dgarrett chromeos_config: Mark terra-chrome-pfq important.
2018-04-06 dgarrett chromeos_config: Move an eve to swarming.
2018-04-06 dgarrett chromeos_config: Helper for custom board_configs.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:824482 , chromium:829898 ,chromium:None


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ia44e90f0126d2b0fbf2aeb7b5c7ea80d48502ae2
Reviewed-on: https://chromium-review.googlesource.com/1000955
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#549051}
[modify] https://crrev.com/65fdfe646c39ff6f2ec27c76e31abc923a8130a3/DEPS

Project Member

Comment 11 by bugdroid1@chromium.org, Apr 9 2018

Labels: merge-merged-config
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/c1ef69527f53742ff50d6acf8899c3902aa03f9b

commit c1ef69527f53742ff50d6acf8899c3902aa03f9b
Author: Don Garrett <dgarrett@google.com>
Date: Mon Apr 09 21:10:06 2018

Owner: dgarr...@chromium.org
Status: Assigned (was: Untriaged)
dgarret@ - can you describe what work is needed?  Should this be done
by a gardener or the Infra team?

Status: Started (was: Assigned)
I did this already, but misunderstood, and did it for chromium, not chrome. Remaining work is to fix that.

This will imply running hwtests, I don't know where they come from or how we ensure there are enough.

This build will be equivalent to most builds here:
  https://uberchromegw.corp.google.com/i/chromeos.chrome/waterfall
> This will imply running hwtests, I don't know where they come from or how we ensure there are enough.

The other informational builders use a pool named 'continuous'.  I expect
that for eve, we should do the same.

The procedure for allocating DUTs to a new pool is here:
    https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/creating-pools

Basically, the procedure requires allocating working DUTs from the suites
pool.

There are 28 eve DUTs in the suites pool:
    $ atest host list -b board:eve,pool:suites | grep '^chromeos' |  wc -l
    28

Alas, only one of them is currently working:
    $ dut-status -b eve -p suites -w | wc -l
    1

How many should be allocated?

Comment 16 by ihf@chromium.org, Apr 12 2018

Cc: rohi...@chromium.org
Just 2 or 3 for informational, but we need to get the broken DUTs repaired first. 28 dead ones cause big problems with arcnext testing.
*and 17 DUTs are locked.

Is there a bug for repairing broken Eve devices?
Project Member

Comment 18 by bugdroid1@chromium.org, Apr 12 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/563b7fd64864c057cceffdfa1fee980c47e8ad9b

commit 563b7fd64864c057cceffdfa1fee980c47e8ad9b
Author: Don Garrett <dgarrett@google.com>
Date: Thu Apr 12 22:14:40 2018

The eve-tot-chromium informational scheduler has been stopped.

The new builder exists, but has not DUTS, so hwtest are expected to fail.

https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=eve-tot-chrome-pfq-informational

Owner: jkop@chromium.org
Passing to the deputy to figure out what's going on with the DUT shortage, and get 2-3 assigned to eve continuous pool.
Thanks for being on top of this. Making this consistent with other PFQ informational builders will avoid a lot of confusion.

As soon as I see it working, I plan to move those builders to swarming, so there is a lot of self interest.

Comment 23 by jkop@google.com, Apr 12 2018

Englab has had persistent urgent issues that kept them from getting to Eve. I'll file a bug to increase that priority.

Comment 24 by jkop@chromium.org, Apr 13 2018

Owner: rsabanga@chromium.org

Comment 25 by jkop@chromium.org, Apr 13 2018

Owner: jkop@chromium.org
DUT shortage mitigated, will add to pool.
The builder in question is running, but not configured to run hwtests.  That means I made a mistake creating it, which I'll try to fix shortly.

https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=eve-tot-chrome-pfq-informational


It doesn't affect the need to allocate DUTs to the pool.
Project Member

Comment 27 by bugdroid1@chromium.org, Apr 14 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616

commit e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616
Author: Don Garrett <dgarrett@google.com>
Date: Sat Apr 14 00:52:21 2018

chromeos-config: Add hwtests to eve chrome tot informational.

When setting up this new builder, I forgot to add hwtests. Fix that.

BUG= chromium:829898 
TEST=cbuildbot/chromeos_config_unittest -u

Change-Id: I3ff653dfebfc684e03dbb381083bd50471ed76c1
Reviewed-on: https://chromium-review.googlesource.com/1012577
Tested-by: Don Garrett <dgarrett@chromium.org>
Trybot-Ready: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616/cbuildbot/config_dump.json
[modify] https://crrev.com/e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616/cbuildbot/chromeos_config.py

Project Member

Comment 28 by bugdroid1@chromium.org, Apr 14 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/534604989fb65f011221a41a5fd378485468b598

commit 534604989fb65f011221a41a5fd378485468b598
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Apr 14 05:15:09 2018

Roll src/third_party/chromite/ c23684514..2e290076c (7 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/c23684514c93..2e290076c8be

$ git log c23684514..2e290076c --date=short --no-merges --format='%ad %ae %s'
2018-04-12 pwang chromeos_config: Mark kevin-arcnext important
2018-04-11 dgarrett binhost_test: Update active config definition.
2018-04-13 dgarrett chromeos-config: Add hwtests to eve chrome tot informational.
2018-04-10 pprabhu cbuildbot: Add pre-cq builder to run infra specific unittests.
2018-04-12 manojgupta cros_setup_toolchains: Update list of post cross packages.
2018-04-12 dgarrett chromeos_config: Move chromiumos-sdk to 'utility' group.
2018-04-09 dgarrett remote_try: Remove buildbot specific behaviors.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:831929 , chromium:829898 ,chromium:None,chromium:820330,chromium:None,chromium:None


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I15b87a3f6a81bd9abdfcc59147a98f0d59932383
Reviewed-on: https://chromium-review.googlesource.com/1013383
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#550881}
[modify] https://crrev.com/534604989fb65f011221a41a5fd378485468b598/DEPS

Comment 29 by ihf@chromium.org, Apr 14 2018

Shoot, I know I approved the change in #27, but I think the bug should have been on "eve-arcnext" and not on "eve" if it wanted to achieve what it stated (cover the newest features like assistant and P).

James/Steven?
My immediate concerned was graphics/ozone-related startup issues, so either "eve" or "eve-arcnext" works for me.

(I presume "eve" still runs assistant, even if it's not on P?)

Comment 31 by ihf@chromium.org, Apr 16 2018

No assistant for eve, probably the next few months? Also unrelated problems in b/77861836.

Comment 32 by jkop@chromium.org, Apr 16 2018

Owner: ihf@chromium.org
Pass back to me if changes are needed.

Comment 33 by ihf@chromium.org, Apr 16 2018

Just to clarify, hopefully soon the Chrome build and assistant build should get more decoupled (I am not sure if fully), so this may not be a decisive feature for this issue.
I'm OK with no assistant on this builder. The graphics init stuff is my immediate concern.

Comment 35 by ihf@chromium.org, Apr 16 2018

Owner: jkop@chromium.org
Ok, then lets keep it this way and let Jacob resolve this issue.

Comment 36 by jkop@chromium.org, Apr 17 2018

So is arcnext needed?

Comment 37 by ihf@chromium.org, Apr 17 2018

For now: no. So if you think eve is otherwise fine please close the issue.

Comment 38 by jkop@chromium.org, Apr 17 2018

Status: Fixed (was: Started)
Alright, I wasn't clear on whether assistant was an addon or was related to the eve/eve-arcnext mapping.
Project Member

Comment 39 by bugdroid1@chromium.org, Apr 17 2018

Labels: merge-merged-testbranch
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/534604989fb65f011221a41a5fd378485468b598

commit 534604989fb65f011221a41a5fd378485468b598
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Apr 14 05:15:09 2018

Roll src/third_party/chromite/ c23684514..2e290076c (7 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/c23684514c93..2e290076c8be

$ git log c23684514..2e290076c --date=short --no-merges --format='%ad %ae %s'
2018-04-12 pwang chromeos_config: Mark kevin-arcnext important
2018-04-11 dgarrett binhost_test: Update active config definition.
2018-04-13 dgarrett chromeos-config: Add hwtests to eve chrome tot informational.
2018-04-10 pprabhu cbuildbot: Add pre-cq builder to run infra specific unittests.
2018-04-12 manojgupta cros_setup_toolchains: Update list of post cross packages.
2018-04-12 dgarrett chromeos_config: Move chromiumos-sdk to 'utility' group.
2018-04-09 dgarrett remote_try: Remove buildbot specific behaviors.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:831929 , chromium:829898 ,chromium:None,chromium:820330,chromium:None,chromium:None


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I15b87a3f6a81bd9abdfcc59147a98f0d59932383
Reviewed-on: https://chromium-review.googlesource.com/1013383
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#550881}
[modify] https://crrev.com/534604989fb65f011221a41a5fd378485468b598/DEPS

Sign in to add a comment