New issue
Advanced search Search tips

Issue 868525 link

Starred by 1 user
Status: Fixed
Owner:
jbudorick@chromium.org
Closed: Aug 28
Cc:
hinoka@chromium.org
efoo@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Fix broken luci migration on chromium.swarm bots

Project Member Reported by bpastene@chromium.org, Jul 27 
These are supposed to be running the swarming/staging.py recipe. They're running the chromium.py recipe.

This regressed after luci migrations. (Why did migrating them switch their recipe?!) Please fix this ASAP. It's blocking prod rolls of swarming bot config.

Before:
https://ci.chromium.org/buildbot/chromium.swarm/Android%20N5%20Swarm/9250
After:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20N5%20Swarm/9500

Comment 1 by hinoka@chromium.org, Jul 27

Owner: hinoka@chromium.org
Status: Assigned (was: Untriaged)
I'm surprised it didn't turn purple.  I'll fix this

Comment 2 by hinoka@chromium.org, Jul 27 

They're using the chromium recipe because that's the standard on chromium.  Using a different recipe makes them a special snowflake in the eyes of mass migration, and usually that's not a problem because they would just fail builder lookup in the recipe.

For some reason they didn't fail builder lookup, that's how they slipped through.
Project Member

Comment 3 by bugdroid1@chromium.org, Jul 27 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b5aa50a36185fae107bf31358d48ecb060a8882c

commit b5aa50a36185fae107bf31358d48ecb060a8882c
Author: Ryan Tseng <hinoka@google.com>
Date: Fri Jul 27 21:49:20 2018

[cr-buildbucket.cfg] Switch swarm-ci builders to use the swarming/staging recipe

These builders do not use the chromium recipe.

Bug:  868525 
Change-Id: Iac4a05d46397e70a25e47ca3a27a27300b32cff9
Reviewed-on: https://chromium-review.googlesource.com/1153297
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578817}
[modify] https://crrev.com/b5aa50a36185fae107bf31358d48ecb060a8882c/infra/config/global/cr-buildbucket.cfg

Comment 4 by bpastene@chromium.org, Jul 30 

They're running the right recipe (thnx) but now they're getting 403s from isolate:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20N5%20Swarm/9867

Does the service account it's using need some ACLs?

Comment 5 by hinoka@chromium.org, Jul 30 

Ugh these will all have to be moved into the -dev stack instances.
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 3 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/b9acde98b6406d46201391dc2783038c0c1f822a

commit b9acde98b6406d46201391dc2783038c0c1f822a
Author: Ryan Tseng <hinoka@google.com>
Date: Fri Aug 03 23:08:01 2018
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 6 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b4687041d81372cc7da89b21102b7c7c24d791d9

commit b4687041d81372cc7da89b21102b7c7c24d791d9
Author: Ryan Tseng <hinoka@google.com>
Date: Mon Aug 06 19:05:24 2018

Move chromium.swarm into -dev instances

Swarming does not support triggering tasks across instances (ie prod -> dev)

chromium.swarm exists only to drive traffic in the chromium-swarm-dev instance.
Move everything into cr-buildbucket-dev.

Bug:  868525 
Change-Id: Ice37edd7c8d668e1ec52962811b10fd8883ce659
Reviewed-on: https://chromium-review.googlesource.com/1155740
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Cr-Commit-Position: refs/heads/master@{#580941}
[modify] https://crrev.com/b4687041d81372cc7da89b21102b7c7c24d791d9/infra/config/global/cr-buildbucket-dev.cfg
[modify] https://crrev.com/b4687041d81372cc7da89b21102b7c7c24d791d9/infra/config/global/cr-buildbucket.cfg
[modify] https://crrev.com/b4687041d81372cc7da89b21102b7c7c24d791d9/infra/config/global/luci-milo-dev.cfg
[add] https://crrev.com/b4687041d81372cc7da89b21102b7c7c24d791d9/infra/config/global/luci-scheduler-dev.cfg
[modify] https://crrev.com/b4687041d81372cc7da89b21102b7c7c24d791d9/infra/config/global/luci-scheduler.cfg

Comment 8 by bpastene@chromium.org, Aug 7 

Any updates? I'm guessing the builder links moved. Everything https://ci.chromium.org/p/chromium/g/chromium.swarm/builders on 404s.
Project Member

Comment 9 by bugdroid1@chromium.org, Aug 8 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c9db81cce55b01e5901a0ab197dbf0fc599f2373

commit c9db81cce55b01e5901a0ab197dbf0fc599f2373
Author: Ryan Tseng <hinoka@google.com>
Date: Wed Aug 08 21:09:13 2018
Project Member

Comment 10 by bugdroid1@chromium.org, Aug 9 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/105131828333fd63e23427a1ec1429da909ec7a8

commit 105131828333fd63e23427a1ec1429da909ec7a8
Author: Ryan Tseng <hinoka@google.com>
Date: Thu Aug 09 09:02:20 2018

Comment 11 by bpastene@chromium.org, Aug 10 

Ping again. Where are mah bots!?

Comment 12 by jbudorick@chromium.org, Aug 13

Owner: jbudorick@chromium.org
taking this since hinoka is ooo.
Project Member

Comment 14 by bugdroid1@chromium.org, Aug 14 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c0cb5ca385631ac5b6e7bbc3d906e273acdfd296

commit c0cb5ca385631ac5b6e7bbc3d906e273acdfd296
Author: John Budorick <jbudorick@google.com>
Date: Tue Aug 14 01:34:26 2018

Comment 15 by jbudorick@chromium.org, Aug 14

Status: Started (was: Assigned)
Project Member

Comment 17 by bugdroid1@chromium.org, Aug 14 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/bb49b99b9ebc7418380fc123226e85bdc522ecb7

commit bb49b99b9ebc7418380fc123226e85bdc522ecb7
Author: John Budorick <jbudorick@google.com>
Date: Tue Aug 14 22:51:35 2018
Project Member

Comment 18 by bugdroid1@chromium.org, Aug 15 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/54909773806eec9fbfeda43f0a16b1f5fc44252d

commit 54909773806eec9fbfeda43f0a16b1f5fc44252d
Author: John Budorick <jbudorick@chromium.org>
Date: Wed Aug 15 17:28:05 2018

luci: restore luci-logdog-dev.cfg.

Currently, luci-logdog-dev isn't letting anything write to project
chromium.

Bug:  868525 
Change-Id: I970ccc0b6959f3d5830ca007df963a6edaf6693a
Reviewed-on: https://chromium-review.googlesource.com/1176079
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583301}
[add] https://crrev.com/54909773806eec9fbfeda43f0a16b1f5fc44252d/infra/config/global/luci-logdog-dev.cfg
Project Member

Comment 19 by bugdroid1@chromium.org, Aug 15 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/99255887a08ca83604a76a198b56377cd23846c0

commit 99255887a08ca83604a76a198b56377cd23846c0
Author: John Budorick <jbudorick@chromium.org>
Date: Wed Aug 15 23:58:22 2018

cr-buildbucket-dev: use auto_builder_dimension for luci.chromium.ci.

Bug:  868525 
Change-Id: I65aabb60d4823423d7f0bd4d5b51ea145c665658
Reviewed-on: https://chromium-review.googlesource.com/1176471
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583450}
[modify] https://crrev.com/99255887a08ca83604a76a198b56377cd23846c0/infra/config/global/cr-buildbucket-dev.cfg

Comment 20 by jbudorick@chromium.org, Aug 16 

cloud things:

 - granted Service Account Token Creator for chromium-ci-builder-dev@chops-service-accounts to luci-token-server-dev
 - granted Storage Object Viewer for gs://chrome-{,partner-}telemetry to chromium-ci-builder-dev@chops-service-accounts

Comment 21 by jbudorick@chromium.org, Aug 16 

https://luci-milo-dev.appspot.com/p/chromium/builders/luci.chromium.ci/Android%20N5%20Swarm is back to normal.
https://luci-milo-dev.appspot.com/p/chromium/builders/luci.chromium.ci/Android%20N5X%20Swarm is infra-failing because there's apparently only one live N5X on chromium-swarm-dev, but should otherwise be back to normal.

I'm adding 2 more MP indefinite gce machines to chromium-swarm-dev from ChromeOS and Linux to use in crrev.com/i/663328. Once that lands, I'll be removing auto_builder_dimension again s.t. we exercise the warm cache logic.

Still need to look into machines for the mac & win bots.
Project Member

Comment 22 by bugdroid1@chromium.org, Aug 16 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/17bdfb7f31d73ead33616eedcfb574615cb3a2bc

commit 17bdfb7f31d73ead33616eedcfb574615cb3a2bc
Author: John Budorick <jbudorick@google.com>
Date: Thu Aug 16 16:41:30 2018
Project Member

Comment 23 by bugdroid1@chromium.org, Aug 16 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/a514441598d7d0979f7a4a9eb03212a8b970e51e

commit a514441598d7d0979f7a4a9eb03212a8b970e51e
Author: John Budorick <jbudorick@google.com>
Date: Thu Aug 16 22:12:09 2018
Project Member

Comment 24 by bugdroid1@chromium.org, Aug 16 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e83956e35f6572c5d2e8cbe11f596f69c851fdbe

commit e83956e35f6572c5d2e8cbe11f596f69c851fdbe
Author: John Budorick <jbudorick@chromium.org>
Date: Thu Aug 16 22:26:07 2018

Revert "cr-buildbucket-dev: use auto_builder_dimension for luci.chromium.ci."

This reverts commit 99255887a08ca83604a76a198b56377cd23846c0.

Reason for revert: added more machines, going to try out warm caching

Original change's description:
> cr-buildbucket-dev: use auto_builder_dimension for luci.chromium.ci.
> 
> Bug:  868525 
> Change-Id: I65aabb60d4823423d7f0bd4d5b51ea145c665658
> Reviewed-on: https://chromium-review.googlesource.com/1176471
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>
> Commit-Queue: John Budorick <jbudorick@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#583450}

TBR=nodir@chromium.org,jbudorick@chromium.org

Change-Id: If807d179cf51089c8332a627ae4d2de78468e79a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  868525 
Reviewed-on: https://chromium-review.googlesource.com/1178993
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583856}
[modify] https://crrev.com/e83956e35f6572c5d2e8cbe11f596f69c851fdbe/infra/config/global/cr-buildbucket-dev.cfg

Comment 25 by bpastene@chromium.org, Aug 16 

> there's apparently only one live N5X on chromium-swarm-dev, but should otherwise be back to normal.

https://chrome-internal-review.googlesource.com/651355 was meant to help w/ that. Just realized I had drained that bot and forgot to undrain it... whoops. Brought it back up, so there should be a few more bullheads on dev shortly.

Comment 26 by bpastene@chromium.org, Aug 16 

And thnx for fixing these bots John!
Project Member

Comment 27 by bugdroid1@chromium.org, Aug 17 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/63ca2713e30d3dea9d8ec9394378fb5535479e0c

commit 63ca2713e30d3dea9d8ec9394378fb5535479e0c
Author: John Budorick <jbudorick@google.com>
Date: Fri Aug 17 21:05:53 2018
Project Member

Comment 29 by bugdroid1@chromium.org, Aug 28 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c5db543954cfea9ba6d8b7066a1bb9daaf9e393f

commit c5db543954cfea9ba6d8b7066a1bb9daaf9e393f
Author: John Budorick <jbudorick@google.com>
Date: Tue Aug 28 14:52:20 2018

Comment 30 by jbudorick@chromium.org, Aug 28 

Windows Swarm builder alive thanks to MP indefinite at https://chromium-swarm-dev.appspot.com/bot?id=win10-dev-2de21a4b-us-west1-c-zvs7

Comment 31 by jbudorick@chromium.org, Aug 28

Status: Fixed (was: Started)
►

Sign in to add a comment