bots need to: - Stash away provisionable labels as DUT-state - Report these as part of dimensions
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/infra_internal/skylab_inventory/+/d85141379405851c087e1be1d9ea3258595fe951 commit d85141379405851c087e1be1d9ea3258595fe951 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 16 21:25:05 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/6c8ead21d1abdc7acd761493d45002fafbe1337d commit 6c8ead21d1abdc7acd761493d45002fafbe1337d Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri May 18 23:42:31 2018 skylab_swarming_worker: Add flag for provisioning labels skylab_swarming_worker simply forwards the flag to lucifer_run_job, for now. BUG= chromium:843776 TEST=Manual, on skylab-drone Change-Id: I7bf9bb4af202b6dee2f4d510b6fe8daf4d332c7d Reviewed-on: https://chromium-review.googlesource.com/1064745 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/6c8ead21d1abdc7acd761493d45002fafbe1337d/src/lucifer/client/lucifer/lucifer.go [modify] https://crrev.com/6c8ead21d1abdc7acd761493d45002fafbe1337d/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/ca2a545db8ae16fc92c4d07ea2a0af60e2ee70c1 commit ca2a545db8ae16fc92c4d07ea2a0af60e2ee70c1 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue May 22 20:16:41 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/6c3dba252724955380460077d41ea05074d58bcb commit 6c3dba252724955380460077d41ea05074d58bcb Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue May 22 22:57:25 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/7f5a344b6c58cb2046d32534c1440b32cfbc9abb commit 7f5a344b6c58cb2046d32534c1440b32cfbc9abb Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:51:54 2018 hoststatus: Add provisionable labels Provisionable labels are another part of bot state cached on the swarming bot. BUG= chromium:843776 TEST=make check Change-Id: Ic6c3e149279aa0046816a17c51e3d85f22f8a740 Reviewed-on: https://chromium-review.googlesource.com/1066770 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/7f5a344b6c58cb2046d32534c1440b32cfbc9abb/src/lucifer/hoststatus/hoststatus_test.go [modify] https://crrev.com/7f5a344b6c58cb2046d32534c1440b32cfbc9abb/src/lucifer/hoststatus/hoststatus.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/b94559fe1f171cd19825622076a1b62ec401928c commit b94559fe1f171cd19825622076a1b62ec401928c Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:51:54 2018 lucifer_run_job: Report metrics for host event instead of host status. lucifer_run_job generates host status events. It does not know what host status the events get translated into. As such, it should generate metrics for the events instead of the final status. This CL is a preparation for moving hoststatus caching into the swarming package where it belongs by dropping this spurious dependency. BUG= chromium:843776 TEST=make check; manual on skylab-drone. Change-Id: If8afebcc24b187f7ddf7ec73181d7546d66c2ebc Reviewed-on: https://chromium-review.googlesource.com/1066771 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/b94559fe1f171cd19825622076a1b62ec401928c/src/lucifer/cmd/lucifer_run_job/metrics.go [modify] https://crrev.com/b94559fe1f171cd19825622076a1b62ec401928c/src/lucifer/cmd/lucifer_run_job/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/56546cc31a09bfcf9cb6e1c6d0451b32605271eb commit 56546cc31a09bfcf9cb6e1c6d0451b32605271eb Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:56:35 2018 Move hoststatus package to swarming/botcache hoststatus now houses all the cached dimensions on a Swarming bot, beyond just the host status. Move it to a swarming/botcache package, that may be expanded to cache other stuff on the bot beyond dimensions as well. BUG= chromium:843776 TEST=unittests, manual on skylab-drone. Change-Id: I24147729c55f4925492d5806eb1870e6884a65f1 Reviewed-on: https://chromium-review.googlesource.com/1066772 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/56546cc31a09bfcf9cb6e1c6d0451b32605271eb/src/lucifer/cmd/skylab_swarming_worker/lucifer.go [delete] https://crrev.com/b94559fe1f171cd19825622076a1b62ec401928c/src/lucifer/hoststatus/hoststatus.go [rename] https://crrev.com/56546cc31a09bfcf9cb6e1c6d0451b32605271eb/src/lucifer/swarming/botcache/dimensions_test.go [add] https://crrev.com/56546cc31a09bfcf9cb6e1c6d0451b32605271eb/src/lucifer/swarming/botcache/dimensions.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/25b02a6585bc93bf0b4c6317ade0927ced0563ad commit 25b02a6585bc93bf0b4c6317ade0927ced0563ad Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:56:35 2018 skylab_swarming_worker: Use parsed HostInfo - Add helper functions in lucifer/autotest package to marshal/unmarshal HostInfo data. - Use parsed HostInfo objects in skylab_swarming_worker. This is preparation to start including provisionable labels in the HostInfo data. BUG= chromium:843776 TEST=unittests, provision task on skylab-drone. Change-Id: I976819742de25d5f7d66bc9b5630c0ed716b897f Reviewed-on: https://chromium-review.googlesource.com/1068247 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [add] https://crrev.com/25b02a6585bc93bf0b4c6317ade0927ced0563ad/src/lucifer/autotest/hostinfo.go [add] https://crrev.com/25b02a6585bc93bf0b4c6317ade0927ced0563ad/src/lucifer/autotest/hostinfo_test.go [modify] https://crrev.com/25b02a6585bc93bf0b4c6317ade0927ced0563ad/src/lucifer/swarming/swarming.go [modify] https://crrev.com/25b02a6585bc93bf0b4c6317ade0927ced0563ad/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/250f48296adfcc9954bcb365457be3c677ad7e49 commit 250f48296adfcc9954bcb365457be3c677ad7e49 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:56:36 2018 swarming: Accept swarming.Bot as argument in botcache functions BUG= chromium:843776 TEST=unittests. Change-Id: I968a792787495f1b4f6ff907fbfa8b53a41e5050 Reviewed-on: https://chromium-review.googlesource.com/1068611 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/250f48296adfcc9954bcb365457be3c677ad7e49/src/lucifer/cmd/skylab_swarming_worker/lucifer.go [modify] https://crrev.com/250f48296adfcc9954bcb365457be3c677ad7e49/src/lucifer/swarming/botcache/dimensions_test.go [modify] https://crrev.com/250f48296adfcc9954bcb365457be3c677ad7e49/src/lucifer/swarming/botcache/dimensions.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/1d5adcc48be63b590c48ff6af9b03a79cf4026c1 commit 1d5adcc48be63b590c48ff6af9b03a79cf4026c1 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:56:36 2018 skylab_swarming_worker: Hoist botcache Load/Dump BUG= chromium:843776 TEST=unittests Change-Id: I816fb90ea6b45a669a8d82e96f63b4d4439614b5 Reviewed-on: https://chromium-review.googlesource.com/1068612 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/1d5adcc48be63b590c48ff6af9b03a79cf4026c1/src/lucifer/cmd/skylab_swarming_worker/lucifer.go [modify] https://crrev.com/1d5adcc48be63b590c48ff6af9b03a79cf4026c1/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/e4a48f93610a9fda2303771dd63af74002de3712 commit e4a48f93610a9fda2303771dd63af74002de3712 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:56:37 2018 skylab_swarming_worker: Rework error handling for dimension caching BUG= chromium:843776 TEST=unittests Change-Id: Iff26813c7f34fe1d5a16a878667dd10c29aa28b7 Reviewed-on: https://chromium-review.googlesource.com/1068613 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/e4a48f93610a9fda2303771dd63af74002de3712/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/c4aba35fe1eaf09fed7b3a21c857325498d3a2d0 commit c4aba35fe1eaf09fed7b3a21c857325498d3a2d0 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:56:37 2018 skylab_swarming_worker: Update cached dimensions with provisioned labels BUG= chromium:843776 TEST=unittests; provision task on skylab-drone Change-Id: I19e0209def7f9b32af30cc66ed1c9a919f75e31d Reviewed-on: https://chromium-review.googlesource.com/1068648 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/c4aba35fe1eaf09fed7b3a21c857325498d3a2d0/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/b5f924dab146abce65862598c3a03b5972a8ed01 commit b5f924dab146abce65862598c3a03b5972a8ed01 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 04:56:37 2018 swarming: Initialize Dimensions.ProvisionableLabels when missing BUG= chromium:843776 TEST=unittests, provision task on skylab-drone Change-Id: Iabd7cd53475d8c16cef88369a581d8dcec8b7e43 Reviewed-on: https://chromium-review.googlesource.com/1068651 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/b5f924dab146abce65862598c3a03b5972a8ed01/src/lucifer/swarming/botcache/dimensions.go [modify] https://crrev.com/b5f924dab146abce65862598c3a03b5972a8ed01/src/lucifer/swarming/botcache/dimensions_test.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/06c33794f8f12a95ae99551571e4aed3c535964c commit 06c33794f8f12a95ae99551571e4aed3c535964c Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 21:39:04 2018 lucifer_run_job: Duplicate host info arguments in provision step BUG= chromium:843776 TEST=unittests; manual provision on skylab-drone Change-Id: I7b62b0b1a7d5300db5fe59d795db7d059b9b52d2 Reviewed-on: https://chromium-review.googlesource.com/1069556 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/06c33794f8f12a95ae99551571e4aed3c535964c/src/lucifer/autotest/atutil/types.go [modify] https://crrev.com/06c33794f8f12a95ae99551571e4aed3c535964c/src/lucifer/cmd/lucifer_run_job/flags.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/8d8a18d2a8a8787e29aee1bd124ecc83b74d8312 commit 8d8a18d2a8a8787e29aee1bd124ecc83b74d8312 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 21:39:04 2018 skylab_swarming_worker: Update HostInfo with cached provisionable labels Provide cached provisionable labels to the autoserv invocations via host info. This allows the provision step to skip actual provision if all the provisionable labels are already present on the DUT. BUG= chromium:843776 TEST=make check; manual provision on skylab-drone. Second provision request skips actually provisioning the DUT. Change-Id: I42aad72dd1236b36482288796e1ce22ee36085d2 Reviewed-on: https://chromium-review.googlesource.com/1069589 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/8d8a18d2a8a8787e29aee1bd124ecc83b74d8312/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/0ec9de5637e85364f94e688f86d918a8b7b7b1d3 commit 0ec9de5637e85364f94e688f86d918a8b7b7b1d3 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 21:39:05 2018 swarming: Include ProvisionableAttributes in botcache This will be used to cache job_repo_url provisionable attribute. BUG= chromium:843776 TEST=unittests Change-Id: I00f67d303339a1a73d3b64e9bffe49d509207d2b Reviewed-on: https://chromium-review.googlesource.com/1069682 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/0ec9de5637e85364f94e688f86d918a8b7b7b1d3/src/lucifer/cmd/skylab_swarming_worker/lucifer.go [delete] https://crrev.com/8d8a18d2a8a8787e29aee1bd124ecc83b74d8312/src/lucifer/swarming/botcache/dimensions.go [add] https://crrev.com/0ec9de5637e85364f94e688f86d918a8b7b7b1d3/src/lucifer/swarming/botcache/botcache.go [rename] https://crrev.com/0ec9de5637e85364f94e688f86d918a8b7b7b1d3/src/lucifer/swarming/botcache/botcache_test.go [modify] https://crrev.com/0ec9de5637e85364f94e688f86d918a8b7b7b1d3/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/0fda5381bb858b2e308664cbda3da20eb6a98b0a commit 0fda5381bb858b2e308664cbda3da20eb6a98b0a Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 21:44:34 2018 skylab_swarming_worker: Cache provisionable attributes BUG= chromium:843776 TEST=unittests; provision task on skylab-drone Change-Id: I9b5f9645c7cd488ec7bb2e56e0ab68c0e8731832 Reviewed-on: https://chromium-review.googlesource.com/1069683 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/0fda5381bb858b2e308664cbda3da20eb6a98b0a/src/lucifer/cmd/skylab_swarming_worker/main.go
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/61245b437d470bd16e3cd31dfa5c87fdb10d8a63 commit 61245b437d470bd16e3cd31dfa5c87fdb10d8a63 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 21:44:35 2018 skylab_swarming_worker: Document some return values BUG= chromium:843776 TEST=None Change-Id: Ib7316c34b30b00d78edc7ecbdd987ce4a264723e Reviewed-on: https://chromium-review.googlesource.com/1069719 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/61245b437d470bd16e3cd31dfa5c87fdb10d8a63/src/lucifer/cmd/skylab_swarming_worker/main.go
Provisionable labels are now advertized as dimensions and provision is skipped (by autoserv) for already existing provisionable labels.
Comment 1 by bugdroid1@chromium.org
, May 16 2018