Add eve to Chrome ToT informational builders |
||||||||||||||
Issue descriptionEve is our lead device and has different graphics hardware than earlier versions. It also has more assistant features. We've been burned multiple times by graphics startup issues in chrome that only affect eve and related boards. Latest is issue 824809. It would be super-helpful to have narrower chrome revision ranges and be able to see if fixes worked without waiting for a Chrome PFQ uprev. https://uberchromegw.corp.google.com/i/chromeos.chrome/waterfall?reload=300 If we need more capacity maybe this could replace caroline.
,
Apr 6 2018
,
Apr 6 2018
Sounds like the gardener should start preparing a change? Normally all that is needed to add a builder is ~/trunk/chromite/cbuildbot/chromeos_config.py But IIRC informational builders were special and needed help from Chrome infra.. Once we have a builder we can allocate DUTs. Luckily eve has a lot of DUTs in the lab, unluckily a lot of them look dead at the moment.
,
Apr 6 2018
Adding new builders is not really a gardening task, we definitely need help from infra for this. +dgarrett@ - Would it be easier to just add this to the new system? As long as the builder runs continuously and we can get the bisect info, it's not critical that the new builder is on the current informational waterfall.
,
Apr 6 2018
I can add it to the new system easily enough, but until we move the others, gardeners will have multiple UIs to look at to see the current state of things.
,
Apr 6 2018
Also, I'm not yet sure if it will generate SoM alerts from swarming.
,
Apr 6 2018
Long term we can address all that, but the most important thing is to get the builder configured and running. IIRC adding a builder to the informational waterfall isn't too difficult and something that a gardener can probably do.
,
Apr 7 2018
I've prepared https://crrev.com/i/603610, which will start running this on swarming. That does not prevent running it on the chromeos.chrome waterfall.
,
Apr 7 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/e1982424f85ed7b793670de05c9ad66863aa282b commit e1982424f85ed7b793670de05c9ad66863aa282b Author: Don Garrett <dgarrett@google.com> Date: Sat Apr 07 05:00:05 2018 chromeos_config: Move an eve to swarming. eve-tot-chrome-pfq-informational. This is only bookkeeping... so far. Also, update how we generate the chromium-pfq-informational swarming configs. BUG= chromium:829898 TEST=chromeos_config_unittest Change-Id: If81ac7b658ade36e87e0c0e8f5cc495147be40a3 Reviewed-on: https://chromium-review.googlesource.com/1000409 Commit-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/e1982424f85ed7b793670de05c9ad66863aa282b/cbuildbot/config_dump.json [modify] https://crrev.com/e1982424f85ed7b793670de05c9ad66863aa282b/cbuildbot/waterfall_layout_dump.txt [modify] https://crrev.com/e1982424f85ed7b793670de05c9ad66863aa282b/cbuildbot/chromeos_config.py
,
Apr 7 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/65fdfe646c39ff6f2ec27c76e31abc923a8130a3 commit 65fdfe646c39ff6f2ec27c76e31abc923a8130a3 Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Sat Apr 07 07:59:29 2018 Roll src/third_party/chromite/ 188669bb7..f10b48279 (4 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/188669bb7526..f10b482799b3 $ git log 188669bb7..f10b48279 --date=short --no-merges --format='%ad %ae %s' 2018-04-04 djkurtz grunt: mark grunt-paladin important 2018-04-06 dgarrett chromeos_config: Mark terra-chrome-pfq important. 2018-04-06 dgarrett chromeos_config: Move an eve to swarming. 2018-04-06 dgarrett chromeos_config: Helper for custom board_configs. Created with: roll-dep src/third_party/chromite BUG= chromium:824482 , chromium:829898 ,chromium:None The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=chrome-os-gardeners@chromium.org Change-Id: Ia44e90f0126d2b0fbf2aeb7b5c7ea80d48502ae2 Reviewed-on: https://chromium-review.googlesource.com/1000955 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#549051} [modify] https://crrev.com/65fdfe646c39ff6f2ec27c76e31abc923a8130a3/DEPS
,
Apr 9 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/c1ef69527f53742ff50d6acf8899c3902aa03f9b commit c1ef69527f53742ff50d6acf8899c3902aa03f9b Author: Don Garrett <dgarrett@google.com> Date: Mon Apr 09 21:10:06 2018
,
Apr 11 2018
dgarret@ - can you describe what work is needed? Should this be done by a gardener or the Infra team?
,
Apr 12 2018
I did this already, but misunderstood, and did it for chromium, not chrome. Remaining work is to fix that. This will imply running hwtests, I don't know where they come from or how we ensure there are enough. This build will be equivalent to most builds here: https://uberchromegw.corp.google.com/i/chromeos.chrome/waterfall
,
Apr 12 2018
> This will imply running hwtests, I don't know where they come from or how we ensure there are enough.
The other informational builders use a pool named 'continuous'. I expect
that for eve, we should do the same.
The procedure for allocating DUTs to a new pool is here:
https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/chromeos-admin/creating-pools
Basically, the procedure requires allocating working DUTs from the suites
pool.
There are 28 eve DUTs in the suites pool:
$ atest host list -b board:eve,pool:suites | grep '^chromeos' | wc -l
28
Alas, only one of them is currently working:
$ dut-status -b eve -p suites -w | wc -l
1
,
Apr 12 2018
How many should be allocated?
,
Apr 12 2018
Just 2 or 3 for informational, but we need to get the broken DUTs repaired first. 28 dead ones cause big problems with arcnext testing.
,
Apr 12 2018
*and 17 DUTs are locked. Is there a bug for repairing broken Eve devices?
,
Apr 12 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/563b7fd64864c057cceffdfa1fee980c47e8ad9b commit 563b7fd64864c057cceffdfa1fee980c47e8ad9b Author: Don Garrett <dgarrett@google.com> Date: Thu Apr 12 22:14:40 2018
,
Apr 12 2018
The eve-tot-chromium informational scheduler has been stopped. The new builder exists, but has not DUTS, so hwtest are expected to fail. https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=eve-tot-chrome-pfq-informational
,
Apr 12 2018
Passing to the deputy to figure out what's going on with the DUT shortage, and get 2-3 assigned to eve continuous pool.
,
Apr 12 2018
Thanks for being on top of this. Making this consistent with other PFQ informational builders will avoid a lot of confusion.
,
Apr 12 2018
As soon as I see it working, I plan to move those builders to swarming, so there is a lot of self interest.
,
Apr 12 2018
Englab has had persistent urgent issues that kept them from getting to Eve. I'll file a bug to increase that priority.
,
Apr 13 2018
,
Apr 13 2018
DUT shortage mitigated, will add to pool.
,
Apr 13 2018
The builder in question is running, but not configured to run hwtests. That means I made a mistake creating it, which I'll try to fix shortly. https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=eve-tot-chrome-pfq-informational It doesn't affect the need to allocate DUTs to the pool.
,
Apr 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616 commit e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616 Author: Don Garrett <dgarrett@google.com> Date: Sat Apr 14 00:52:21 2018 chromeos-config: Add hwtests to eve chrome tot informational. When setting up this new builder, I forgot to add hwtests. Fix that. BUG= chromium:829898 TEST=cbuildbot/chromeos_config_unittest -u Change-Id: I3ff653dfebfc684e03dbb381083bd50471ed76c1 Reviewed-on: https://chromium-review.googlesource.com/1012577 Tested-by: Don Garrett <dgarrett@chromium.org> Trybot-Ready: Don Garrett <dgarrett@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616/cbuildbot/config_dump.json [modify] https://crrev.com/e3e9e04d8ef99ad5feb73f4dcaa8ece454d37616/cbuildbot/chromeos_config.py
,
Apr 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/534604989fb65f011221a41a5fd378485468b598 commit 534604989fb65f011221a41a5fd378485468b598 Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Sat Apr 14 05:15:09 2018 Roll src/third_party/chromite/ c23684514..2e290076c (7 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/c23684514c93..2e290076c8be $ git log c23684514..2e290076c --date=short --no-merges --format='%ad %ae %s' 2018-04-12 pwang chromeos_config: Mark kevin-arcnext important 2018-04-11 dgarrett binhost_test: Update active config definition. 2018-04-13 dgarrett chromeos-config: Add hwtests to eve chrome tot informational. 2018-04-10 pprabhu cbuildbot: Add pre-cq builder to run infra specific unittests. 2018-04-12 manojgupta cros_setup_toolchains: Update list of post cross packages. 2018-04-12 dgarrett chromeos_config: Move chromiumos-sdk to 'utility' group. 2018-04-09 dgarrett remote_try: Remove buildbot specific behaviors. Created with: roll-dep src/third_party/chromite BUG= chromium:831929 , chromium:829898 ,chromium:None,chromium:820330,chromium:None,chromium:None The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=chrome-os-gardeners@chromium.org Change-Id: I15b87a3f6a81bd9abdfcc59147a98f0d59932383 Reviewed-on: https://chromium-review.googlesource.com/1013383 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#550881} [modify] https://crrev.com/534604989fb65f011221a41a5fd378485468b598/DEPS
,
Apr 14 2018
Shoot, I know I approved the change in #27, but I think the bug should have been on "eve-arcnext" and not on "eve" if it wanted to achieve what it stated (cover the newest features like assistant and P). James/Steven?
,
Apr 16 2018
My immediate concerned was graphics/ozone-related startup issues, so either "eve" or "eve-arcnext" works for me. (I presume "eve" still runs assistant, even if it's not on P?)
,
Apr 16 2018
No assistant for eve, probably the next few months? Also unrelated problems in b/77861836.
,
Apr 16 2018
Pass back to me if changes are needed.
,
Apr 16 2018
Just to clarify, hopefully soon the Chrome build and assistant build should get more decoupled (I am not sure if fully), so this may not be a decisive feature for this issue.
,
Apr 16 2018
I'm OK with no assistant on this builder. The graphics init stuff is my immediate concern.
,
Apr 16 2018
Ok, then lets keep it this way and let Jacob resolve this issue.
,
Apr 17 2018
So is arcnext needed?
,
Apr 17 2018
For now: no. So if you think eve is otherwise fine please close the issue.
,
Apr 17 2018
Alright, I wasn't clear on whether assistant was an addon or was related to the eve/eve-arcnext mapping.
,
Apr 17 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/534604989fb65f011221a41a5fd378485468b598 commit 534604989fb65f011221a41a5fd378485468b598 Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Sat Apr 14 05:15:09 2018 Roll src/third_party/chromite/ c23684514..2e290076c (7 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/c23684514c93..2e290076c8be $ git log c23684514..2e290076c --date=short --no-merges --format='%ad %ae %s' 2018-04-12 pwang chromeos_config: Mark kevin-arcnext important 2018-04-11 dgarrett binhost_test: Update active config definition. 2018-04-13 dgarrett chromeos-config: Add hwtests to eve chrome tot informational. 2018-04-10 pprabhu cbuildbot: Add pre-cq builder to run infra specific unittests. 2018-04-12 manojgupta cros_setup_toolchains: Update list of post cross packages. 2018-04-12 dgarrett chromeos_config: Move chromiumos-sdk to 'utility' group. 2018-04-09 dgarrett remote_try: Remove buildbot specific behaviors. Created with: roll-dep src/third_party/chromite BUG= chromium:831929 , chromium:829898 ,chromium:None,chromium:820330,chromium:None,chromium:None The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=chrome-os-gardeners@chromium.org Change-Id: I15b87a3f6a81bd9abdfcc59147a98f0d59932383 Reviewed-on: https://chromium-review.googlesource.com/1013383 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#550881} [modify] https://crrev.com/534604989fb65f011221a41a5fd378485468b598/DEPS |
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by steve...@chromium.org
, Apr 6 2018Labels: -Pri-3 Pri-2