use buildbucket to launch release builder slaves |
|||||||||||||||||||
Issue descriptionForked from Issue 748259 Not yet, but there are no remaining blockers. I think the steps look like: A) It set up the buildbucket B) Tweak the buildbot instance to use it. C) Tweak the release masters to use the proper buildbucket instance.
,
Sep 13 2017
,
Oct 6 2017
Nxia agreed to take this as an OKR for this quater.
,
Dec 5 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/9736f5a2fea67a372c35e917d67fdb409d747b7b commit 9736f5a2fea67a372c35e917d67fdb409d747b7b Author: Ningning Xia <nxia@google.com> Date: Tue Dec 05 00:43:27 2017
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a5f9f90a9cb934e8d18e9dd3ec7a4d9c3cbeef37 commit a5f9f90a9cb934e8d18e9dd3ec7a4d9c3cbeef37 Author: Ningning Xia <nxia@chromium.org> Date: Tue Dec 05 00:53:14 2017 Enable buildbucket scheduler for master builds in chromeos_release BUG= chromium:755276 TEST=unit_tests Change-Id: If669ce7e3d97024135009b20467960c1d2294f4e Reviewed-on: https://chromium-review.googlesource.com/804596 Reviewed-by: Don Garrett <dgarrett@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/a5f9f90a9cb934e8d18e9dd3ec7a4d9c3cbeef37/lib/config_lib_unittest.py [modify] https://crrev.com/a5f9f90a9cb934e8d18e9dd3ec7a4d9c3cbeef37/lib/constants.py [modify] https://crrev.com/a5f9f90a9cb934e8d18e9dd3ec7a4d9c3cbeef37/lib/buildbucket_lib.py [modify] https://crrev.com/a5f9f90a9cb934e8d18e9dd3ec7a4d9c3cbeef37/lib/config_lib.py
,
Dec 5 2017
,
Dec 5 2017
ketakid@/josafa, can you please approve the merge requests?
,
Dec 5 2017
This bug requires manual review: Request affecting a post-stable build Please contact the milestone owner if you have questions. Owners: cmasso@(Android), cmasso@(iOS), gkihumba@(ChromeOS), govind@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Dec 5 2017
,
Dec 5 2017
,
Dec 5 2017
I plan to restart the chromeos_waterfall tomorrow morning, and need to submit the changes after the restart.
,
Dec 5 2017
Where does this need merged? I would think just 62/63/64 at this point? If this is a build infra change only and not impacting the system image it should be ok to merge (but if we break the release builders we will have to fix it).
,
Dec 5 2017
,
Dec 5 2017
yeah, 62, 63 and 64 are the right targets. This change will change the way the master build schedule the slaves (in cbuildbot), it won't impact the system images.
,
Dec 5 2017
Go for it.
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/f7d93b288cef1f4c11641a4e0fa9cd5d3d17d5c1 commit f7d93b288cef1f4c11641a4e0fa9cd5d3d17d5c1 Author: Ningning Xia <nxia@chromium.org> Date: Tue Dec 05 19:21:44 2017 Enable buildbucket scheduler for master builds in chromeos_release BUG= chromium:755276 TEST=unit_tests Change-Id: If669ce7e3d97024135009b20467960c1d2294f4e Reviewed-on: https://chromium-review.googlesource.com/806264 Reviewed-by: Don Garrett <dgarrett@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/f7d93b288cef1f4c11641a4e0fa9cd5d3d17d5c1/lib/config_lib_unittest.py [modify] https://crrev.com/f7d93b288cef1f4c11641a4e0fa9cd5d3d17d5c1/lib/constants.py [modify] https://crrev.com/f7d93b288cef1f4c11641a4e0fa9cd5d3d17d5c1/lib/buildbucket_lib.py [modify] https://crrev.com/f7d93b288cef1f4c11641a4e0fa9cd5d3d17d5c1/lib/config_lib.py
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/c408615c0c00c9b9f8e933f041a6a0345c42e3e2 commit c408615c0c00c9b9f8e933f041a6a0345c42e3e2 Author: Ningning Xia <nxia@chromium.org> Date: Tue Dec 05 19:22:00 2017 Enable buildbucket scheduler for master builds in chromeos_release BUG= chromium:755276 TEST=unit_tests Change-Id: If669ce7e3d97024135009b20467960c1d2294f4e Reviewed-on: https://chromium-review.googlesource.com/804678 Reviewed-by: Don Garrett <dgarrett@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/c408615c0c00c9b9f8e933f041a6a0345c42e3e2/lib/config_lib_unittest.py [modify] https://crrev.com/c408615c0c00c9b9f8e933f041a6a0345c42e3e2/lib/constants.py [modify] https://crrev.com/c408615c0c00c9b9f8e933f041a6a0345c42e3e2/lib/buildbucket_lib.py [modify] https://crrev.com/c408615c0c00c9b9f8e933f041a6a0345c42e3e2/lib/config_lib.py
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/d33014bda2608b9b8b7cc30d0c360e515c553cc0 commit d33014bda2608b9b8b7cc30d0c360e515c553cc0 Author: Ningning Xia <nxia@chromium.org> Date: Tue Dec 05 19:22:17 2017 Enable buildbucket scheduler for master builds in chromeos_release BUG= chromium:755276 TEST=unit_tests Change-Id: If669ce7e3d97024135009b20467960c1d2294f4e Reviewed-on: https://chromium-review.googlesource.com/804677 Reviewed-by: Don Garrett <dgarrett@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/d33014bda2608b9b8b7cc30d0c360e515c553cc0/lib/config_lib_unittest.py [modify] https://crrev.com/d33014bda2608b9b8b7cc30d0c360e515c553cc0/lib/constants.py [modify] https://crrev.com/d33014bda2608b9b8b7cc30d0c360e515c553cc0/lib/buildbucket_lib.py [modify] https://crrev.com/d33014bda2608b9b8b7cc30d0c360e515c553cc0/lib/config_lib.py
,
Dec 5 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/56c946e6d82c4ad4aab95b9e58f261cfb0546f7d commit 56c946e6d82c4ad4aab95b9e58f261cfb0546f7d Author: Ningning Xia <nxia@google.com> Date: Tue Dec 05 20:12:50 2017
,
Dec 5 2017
,
Dec 5 2017
M63 master-release can't schedule slaves with errors:
"result_details_json": "{\"error\": {\"message\": \"Builder mccloud-release not found\"}}",
looking into the buildslaves list, R62 and R63 are sharing the same builders.
+ nodir@/hinoka@, can we change the logic to fetch the builder as long as the builder list contains the requested builder?
,
Dec 5 2017
pasting the buildslaves https://uberchromegw.corp.google.com/i/chromeos_release/buildslaves
,
Dec 5 2017
turns out the builder name of a build in chromeos_release is configured differently (e.g. build12-m2 lakitu-release release-R63-10032.B lakitu-release release-R62-9901.B), I'll update the buildbucket tag accordingly.
,
Dec 5 2017
re c#21: if buildbot fetches only builders it knows about, then if someone makes a typo in the builder name of the build they schedule, the build will hang until 48h timeout is hit, which is not great. How about not scheduling builds to the new builder until the master is restarted?
,
Dec 5 2017
Re #24: actually all the builds scheduled with unknown builder_names completed with 'FAILURE' (e.g. 8961012059960837152), which means they're not hanging? and I'm going to port a fix to the release branches, so after that, the master build should clean (cancel) all the old slave builds in the cleanup stage in order to build new slaves.
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/9fb27e43e8e6d23358484081f2cbf1cf9fd75a19 commit 9fb27e43e8e6d23358484081f2cbf1cf9fd75a19 Author: Ningning Xia <nxia@chromium.org> Date: Tue Dec 05 23:51:18 2017 Change the builder_names of builds to schedule in chromeos_release The builder_name of release builds are configured differently in chromeos waterfall and chromeos_release waterfall. Change the builder_name tag in the buildbucket request so that buildbucket-scheduler can find the right builders to run the scheduled builds. BUG= chromium:755276 TEST=unit_tests Change-Id: I4511eb785aa5c60abf3daf592ebff78231c88c71 Reviewed-on: https://chromium-review.googlesource.com/809929 Reviewed-by: Don Garrett <dgarrett@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/9fb27e43e8e6d23358484081f2cbf1cf9fd75a19/cbuildbot/stages/scheduler_stages.py [modify] https://crrev.com/9fb27e43e8e6d23358484081f2cbf1cf9fd75a19/cbuildbot/stages/scheduler_stages_unittest.py
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/fbe009b920035161386957a8b6552a7f87f24e62 commit fbe009b920035161386957a8b6552a7f87f24e62 Author: Ningning Xia <nxia@chromium.org> Date: Tue Dec 05 23:53:58 2017 Change the builder_names of builds to schedule in chromeos_release The builder_name of release builds are configured differently in chromeos waterfall and chromeos_release waterfall. Change the builder_name tag in the buildbucket request so that buildbucket-scheduler can find the right builders to run the scheduled builds. BUG= chromium:755276 TEST=unit_tests Change-Id: I4511eb785aa5c60abf3daf592ebff78231c88c71 Reviewed-on: https://chromium-review.googlesource.com/809949 Reviewed-by: Ningning Xia <nxia@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/fbe009b920035161386957a8b6552a7f87f24e62/cbuildbot/stages/scheduler_stages.py [modify] https://crrev.com/fbe009b920035161386957a8b6552a7f87f24e62/cbuildbot/stages/scheduler_stages_unittest.py
,
Dec 5 2017
> actually all the builds scheduled with unknown builder_names completed with 'FAILURE' (e.g. 8961012059960837152), which means they're not hanging? yes, that's what happens now and I'd like to preserve that. I was replying to > can we change the logic to fetch the builder as long as the builder list contains the requested builder?
,
Dec 6 2017
Actually the buildbucket-integration is doing what I asked for. for example: for Name: "build12-m2" Builders: "lakitu-release release-R63-10032.B lakitu-release release-R62-9901.B" in https://uberchromegw.corp.google.com/i/chromeos_release/buildslaves, I expect when a PUT request is sent to Buildbucket with "builder_name: lakitu-release release-R63-10032.B", the build can be picked up by build12-m2. My old PUT request failed because I put "builder_name: lakitu-release" in the request tag list but the builder_name for release branches is configured differently: https://chrome-internal.googlesource.com/chrome/tools/build/+/master/masters/master.chromeos_release/branchcfg.py#53.
,
Dec 6 2017
FYI, I just kicked off a M63 master-release, it successfully scheduled the slaves via Buildbucket: https://uberchromegw.corp.google.com/i/chromeos_release/builders/parrot-release%20release-R63-10032.B/builds/55
,
Dec 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/befeff2a8fbfb85a083db953c61b42d6f01121fe commit befeff2a8fbfb85a083db953c61b42d6f01121fe Author: Ningning Xia <nxia@chromium.org> Date: Wed Dec 06 00:29:15 2017 Change the builder_names of builds to schedule in chromeos_release The builder_name of release builds are configured differently in chromeos waterfall and chromeos_release waterfall. Change the builder_name tag in the buildbucket request so that buildbucket-scheduler can find the right builders to run the scheduled builds. BUG= chromium:755276 TEST=unit_tests Change-Id: I4511eb785aa5c60abf3daf592ebff78231c88c71 Reviewed-on: https://chromium-review.googlesource.com/809956 Reviewed-by: Paul Hobbs <phobbs@google.com> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/befeff2a8fbfb85a083db953c61b42d6f01121fe/cbuildbot/stages/scheduler_stages.py [modify] https://crrev.com/befeff2a8fbfb85a083db953c61b42d6f01121fe/cbuildbot/stages/scheduler_stages_unittest.py
,
Dec 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/319328eb7386440ea0ffd79ed28e0ec40e5134b9 commit 319328eb7386440ea0ffd79ed28e0ec40e5134b9 Author: Ningning Xia <nxia@chromium.org> Date: Wed Dec 06 00:29:29 2017 Change the builder_names of builds to schedule in chromeos_release The builder_name of release builds are configured differently in chromeos waterfall and chromeos_release waterfall. Change the builder_name tag in the buildbucket request so that buildbucket-scheduler can find the right builders to run the scheduled builds. BUG= chromium:755276 TEST=unit_tests Change-Id: I4511eb785aa5c60abf3daf592ebff78231c88c71 Reviewed-on: https://chromium-review.googlesource.com/809813 Reviewed-by: Paul Hobbs <phobbs@google.com> Tested-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/319328eb7386440ea0ffd79ed28e0ec40e5134b9/cbuildbot/stages/scheduler_stages.py [modify] https://crrev.com/319328eb7386440ea0ffd79ed28e0ec40e5134b9/cbuildbot/stages/scheduler_stages_unittest.py
,
Dec 6 2017
your PUT request failed not because buildbot-integation did something. The build didn't even reach it. Buildbucket server requires builder_name value to match builder tag, that's why PUT failed. Assuming "what I asked for" referred to "+ nodir@/hinoka@, can we change the logic to fetch the builder as long as the builder list contains the requested builder?", I don't see how the behavior above is irrelevant to this request
,
Dec 6 2017
maybe I didn't explain clearly in #29, I meant the buildbucket-integration has been doing correctly, the problem was I sent the PUT requests with wrong builder_name tag (as builder_name in chromeos_release waterfall is configured differently compared to the other waterfall like chromeos). I've made a patch to send the PUT requests with the right builder_name tags for chromeos_release. Everything is good so far.
,
Dec 6 2017
I just verified both M64 and M63 scheduled slaves via buildbucket.
,
Dec 6 2017
,
Dec 11 2017
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Jan 9 2018
,
Feb 12 2018
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Feb 12 2018
|
|||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||
Comment 1 by akes...@chromium.org
, Aug 14 2017