Remove 24 core build hardware from chromeos/chromeos_release/chromiumos waterfalls. |
|||||||||||||
Issue descriptionWe've been seeing build timeouts associated with slower hardware. We've got enough unused hardware to stop using the slowing machines, so... let's just pull them out of use for ChromeOS builds.
,
Jan 19 2017
Can someone in the lab team help produce this list of builders?
,
Jan 19 2017
,
Jan 19 2017
I should point out that we might not have enough idle/fast machine right now. We expect to move a large number of builds onto GCE later today which should free up a lot of hardware.
,
Jan 19 2017
So this actually relates to bug 598162, where we (labs) were going to attempt to homogenize the CrOS bare metal fleet to mostly 40 (R620) core systems and leave only a few of the 32 (R720) core systems (main goal of removing the 24 core builders, as they're rather old). If it turns out we can't remove all of the 24 core builders, we can just replace the remainder with the 40 core builders seeing as we still have a supply of those. I'll start compiling a list.
,
Jan 19 2017
I'm hoping to free roughly 86 builders later today, and another 60 over the next few months.
,
Jan 19 2017
Looks like there's only 50 of the 24 core builders left: build16-m2.golo.chromium.org: poweredge r410 build17-m2.golo.chromium.org: poweredge r410 build20-m2.golo.chromium.org: poweredge r410 build25-m2.golo.chromium.org: poweredge r410 build35-m2.golo.chromium.org: poweredge r410 build41-m2.golo.chromium.org: poweredge r410 build44-m2.golo.chromium.org: poweredge r410 build49-m2.golo.chromium.org: poweredge r410 build59-m2.golo.chromium.org: poweredge r410 build60-m2.golo.chromium.org: poweredge r410 build61-m2.golo.chromium.org: poweredge r410 build62-m2.golo.chromium.org: poweredge r410 build63-m2.golo.chromium.org: poweredge r410 build64-m2.golo.chromium.org: poweredge r410 build65-m2.golo.chromium.org: poweredge r410 build66-m2.golo.chromium.org: poweredge r410 build67-m2.golo.chromium.org: poweredge r410 build69-m2.golo.chromium.org: poweredge r410 build70-m2.golo.chromium.org: poweredge r410 build71-m2.golo.chromium.org: poweredge r410 build73-m2.golo.chromium.org: poweredge r410 build74-m2.golo.chromium.org: poweredge r410 build75-m2.golo.chromium.org: poweredge r410 build76-m2.golo.chromium.org: poweredge r410 build77-m2.golo.chromium.org: poweredge r410 build78-m2.golo.chromium.org: poweredge r410 build79-m2.golo.chromium.org: poweredge r410 build80-m2.golo.chromium.org: poweredge r410 build81-m2.golo.chromium.org: poweredge r410 build82-m2.golo.chromium.org: poweredge r410 build83-m2.golo.chromium.org: poweredge r410 build84-m2.golo.chromium.org: poweredge r410 build85-m2.golo.chromium.org: poweredge r410 build86-m2.golo.chromium.org: poweredge r410 build87-m2.golo.chromium.org: poweredge r410 build88-m2.golo.chromium.org: poweredge r410 build89-m2.golo.chromium.org: poweredge r410 build90-m2.golo.chromium.org: poweredge r410 build92-m2.golo.chromium.org: poweredge r410 build93-m2.golo.chromium.org: poweredge r410 build94-m2.golo.chromium.org: poweredge r410 build95-m2.golo.chromium.org: poweredge r410 build96-m2.golo.chromium.org: poweredge r410 build97-m2.golo.chromium.org: poweredge r410 build151-m2.golo.chromium.org: poweredge r410 build152-m2.golo.chromium.org: poweredge r410 build154-m2.golo.chromium.org: poweredge r410 build155-m2.golo.chromium.org: poweredge r410 build156-m2.golo.chromium.org: poweredge r410 build157-m2.golo.chromium.org: poweredge r410
,
Jan 20 2017
I've produced this spreadsheet and broken them down by waterfall. I'm going to try and take them out of use, then let you know.
,
Jan 20 2017
SGTM.
,
Jan 21 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/d69623887c2061592f9b77706cdeffab26a3db33 commit d69623887c2061592f9b77706cdeffab26a3db33 Author: Don Garrett <dgarrett@google.com> Date: Fri Jan 20 23:59:44 2017
,
Jan 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/e1e2a0323c6dd2a732592975bb8db23644bfa8f8 commit e1e2a0323c6dd2a732592975bb8db23644bfa8f8 Author: Don Garrett <dgarrett@google.com> Date: Sat Jan 21 00:00:46 2017 chromiumos: Remove 24 core builders. This CL decommissions chromiumos 24 core builders However, this only leaves a single unallocated slave (which is currently offline). BUG= chromium:682789 Change-Id: I11c84d941e5d1e7d83f0ad6e15cdf7c26fb53b9b Reviewed-on: https://chromium-review.googlesource.com/431150 Reviewed-by: Daniel Jacques <dnj@chromium.org> Commit-Queue: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/e1e2a0323c6dd2a732592975bb8db23644bfa8f8/masters/master.chromiumos/slave_pool.json [modify] https://crrev.com/e1e2a0323c6dd2a732592975bb8db23644bfa8f8/masters/master.chromiumos/slaves.cfg
,
Jan 21 2017
After the associated waterfalls are restarted, the above machines will all be out of use, but that process might not finish until Monday, depending.
,
Jan 23 2017
,
Jan 24 2017
All of the build machines you list above have been removed from the assorted ChromeOS waterfalls that were using them.
,
Feb 2 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome-golo/chrome-golo/+/43d6a71c348aa01b9bd8c97413463533725663f5 commit 43d6a71c348aa01b9bd8c97413463533725663f5 Author: Bryce Albritton <dba@google.com> Date: Thu Feb 02 23:40:52 2017
,
Feb 2 2017
dgarrett - looks like build69-m2 and build82-m2 are still in configurations: $ botmap | grep build'(69|82)'-m2 build69-m2 /b master.chromeos_release unallocated-slave-pool linux build82-m2 /b master.chromeos_release unallocated-slave-pool linux I've already decom'd all the hosts, so those two will never return. If you need 2 replacements, please let me know.
,
Feb 3 2017
No need for new machines. We should have enough slack there to last a while.
,
Feb 3 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/23b5d965e6edbb1e3a8ab14363c809385edad3c3 commit 23b5d965e6edbb1e3a8ab14363c809385edad3c3 Author: Don Garrett <dgarrett@google.com> Date: Fri Feb 03 19:27:12 2017
,
Feb 3 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/23b5d965e6edbb1e3a8ab14363c809385edad3c3 commit 23b5d965e6edbb1e3a8ab14363c809385edad3c3 Author: Don Garrett <dgarrett@google.com> Date: Fri Feb 03 19:27:12 2017
,
Feb 3 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/23b5d965e6edbb1e3a8ab14363c809385edad3c3 commit 23b5d965e6edbb1e3a8ab14363c809385edad3c3 Author: Don Garrett <dgarrett@google.com> Date: Fri Feb 03 19:27:12 2017
,
Feb 3 2017
,
Mar 7 2017
Issue 598162 has been merged into this issue.
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
|||||||||||||
►
Sign in to add a comment |
|||||||||||||
Comment 1 by dgarr...@chromium.org
, Jan 19 2017