New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Feb 1
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocked on:
issue 925172



Sign in to add a comment
link

Issue 913674: Automatically EnsurePoolHealthy for all models

Reported by pprabhu@chromium.org, Dec 10 Project Member

Issue description

This requires the crosskylabadmin app to be able to commit changes to chromeos/infra_internal/skylab_inventory project.

Manual EnsurePoolHealthy tool was added in  issue 900006 
 

Comment 1 by bugdroid1@chromium.org, Dec 11

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/4e797411d57db21060f9881a5febef494b21ea22

commit 4e797411d57db21060f9881a5febef494b21ea22
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Tue Dec 11 23:17:55 2018

skylab: Extract a printError function

BUG= chromium:913674 

Change-Id: I4293fa18c5e986d8bdc2c3a9f874d0f70fecda5f
Reviewed-on: https://chromium-review.googlesource.com/c/1372214
Reviewed-by: Allen Li <ayatane@chromium.org>
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19496}
[modify] https://crrev.com/4e797411d57db21060f9881a5febef494b21ea22/go/src/infra/cmd/skylab/internal/cmd/ensure-pool-healthy.go

Comment 2 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7717cb87160f62088c8b072e7660538bfc21a726

commit 7717cb87160f62088c8b072e7660538bfc21a726
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Wed Dec 12 00:17:10 2018

skylab: push parseArgs down the stack

BUG= chromium:913674 

Change-Id: Ic1325e19b958b49683a18699615229aa1be2494c
Reviewed-on: https://chromium-review.googlesource.com/c/1372215
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19499}
[modify] https://crrev.com/7717cb87160f62088c8b072e7660538bfc21a726/go/src/infra/cmd/skylab/internal/cmd/ensure-pool-healthy.go

Comment 3 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e7696ba2a36e736adb88b5c3d45011c1c26dc9de

commit e7696ba2a36e736adb88b5c3d45011c1c26dc9de
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Wed Dec 12 00:21:43 2018

skylab/ensure-pool-healthy: Fix panic() while printing

BUG= chromium:913674 

Change-Id: I4082472f73f23cdf426d88cff47e5c80f288bfc1
Reviewed-on: https://chromium-review.googlesource.com/c/1372217
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19500}
[modify] https://crrev.com/e7696ba2a36e736adb88b5c3d45011c1c26dc9de/go/src/infra/cmd/skylab/internal/cmd/ensure-pool-healthy.go

Comment 4 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/687d3cc3c7882aef224f3b5d43a8b805ee442937

commit 687d3cc3c7882aef224f3b5d43a8b805ee442937
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Wed Dec 12 00:37:16 2018

skylab: Split parseArgs

BUG= chromium:913674 

Change-Id: I117e8d3f33004860838f7a573548cff1b2a4523d
Reviewed-on: https://chromium-review.googlesource.com/c/1372216
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19501}
[modify] https://crrev.com/687d3cc3c7882aef224f3b5d43a8b805ee442937/go/src/infra/cmd/skylab/internal/cmd/ensure-pool-healthy.go

Comment 5 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ffcb03c4fc1fa0f9c8b39413589faff17112a578

commit ffcb03c4fc1fa0f9c8b39413589faff17112a578
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Wed Dec 12 00:55:01 2018

skylab: Add -all-models to ensure-pool-healthy subcommand

BUG= chromium:913674 

Change-Id: Iee7b7e5fdca0a1bda11993bad8127b72efd6cc55
Reviewed-on: https://chromium-review.googlesource.com/c/1372218
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19502}
[modify] https://crrev.com/ffcb03c4fc1fa0f9c8b39413589faff17112a578/go/src/infra/cmd/skylab/internal/cmd/ensure-pool-healthy.go

Comment 6 by pprabhu@chromium.org, Dec 20

Components: -Infra>Client>ChromeOS>Test Infra>Client>ChromeOS>Test>Platform

Comment 7 by bugdroid1@chromium.org, Jan 8

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d1f94abb4da00b19f465b411820fcbe2c94eeb93

commit d1f94abb4da00b19f465b411820fcbe2c94eeb93
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Tue Jan 08 20:10:44 2019

crosskylabadmin: Extract ensurePoolHealthyFor free function

BUG= chromium:913674 

Change-Id: I98f9993c5cce9da409463bed45cd4e782c438fc5
Reviewed-on: https://chromium-review.googlesource.com/c/1399337
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19832}
[modify] https://crrev.com/d1f94abb4da00b19f465b411820fcbe2c94eeb93/go/src/infra/appengine/crosskylabadmin/app/frontend/inventory/inventory.go

Comment 8 by bugdroid1@chromium.org, Jan 8

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/704ed33ea4d2dc7ef53ef3f1388c9fb2a02a969b

commit 704ed33ea4d2dc7ef53ef3f1388c9fb2a02a969b
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Tue Jan 08 21:35:00 2019

crosskylabadmin: Split off pool operations to own file

BUG= chromium:913674 

Change-Id: I3a3c9cdb2da58ddabd97e72c6765145448482278
Reviewed-on: https://chromium-review.googlesource.com/c/1399338
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19834}
[modify] https://crrev.com/704ed33ea4d2dc7ef53ef3f1388c9fb2a02a969b/go/src/infra/appengine/crosskylabadmin/app/frontend/inventory/inventory.go
[add] https://crrev.com/704ed33ea4d2dc7ef53ef3f1388c9fb2a02a969b/go/src/infra/appengine/crosskylabadmin/app/frontend/inventory/pool_operations.go

Comment 10 by bugdroid1@chromium.org, Jan 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f80b4a3720701f3381dc0d864fcf03630c4a9a28

commit f80b4a3720701f3381dc0d864fcf03630c4a9a28
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Fri Jan 18 01:05:37 2019

crosskylabadmin: Add endpoint to ensure pool health for all models

BUG= chromium:913674 

Change-Id: I2fa1048d91bca44c1ab9d01dd7b364bfded276ae
Reviewed-on: https://chromium-review.googlesource.com/c/1399339
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20065}
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/inventory.pb.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/app/frontend/inventory/pool_operations.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/tracker.pb.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/tasker.pb.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/tracker.mock.pb.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/validation.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/inventory.proto
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/inventoryserver_dec.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/inventory.mock.pb.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/app/clients/mock/swarming.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/common.pb.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/pb.discovery.go
[modify] https://crrev.com/f80b4a3720701f3381dc0d864fcf03630c4a9a28/go/src/infra/appengine/crosskylabadmin/api/fleet/v1/tasker.mock.pb.go

Comment 11 by bugdroid1@chromium.org, Jan 18

Project Member

Comment 12 by bugdroid1@chromium.org, Jan 18

Project Member
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/6863e9388877a7404f392e3589114491b3232fb8

commit 6863e9388877a7404f392e3589114491b3232fb8
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Fri Jan 18 18:15:41 2019

Comment 13 by pprabhu@chromium.org, Jan 24

Blockedon: 925172

Comment 14 by bugdroid, Jan 31

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6e6c3d7c4f74f92cf2944f87ff2fff85175d1202

commit 6e6c3d7c4f74f92cf2944f87ff2fff85175d1202
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Jan 31 23:26:42 2019

crosskylabadmin: Enable cron to ensure critical pool health

BUG= chromium:913674 

Change-Id: I24ae7b45d634c290b0d644a6fba68724bd7b33c7
Reviewed-on: https://chromium-review.googlesource.com/c/1403217
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20386}
[modify] https://crrev.com/6e6c3d7c4f74f92cf2944f87ff2fff85175d1202/go/src/infra/appengine/crosskylabadmin/app/appengine/cron.yaml

Comment 15 by pprabhu@chromium.org, Feb 1

Status: Fixed (was: Assigned)
-dev and -prod pushed and cron jobs are kicking.

But they're still failing due to issue 919668

Sign in to add a comment