WebRTC's presubmit bot is broken |
|||
Issue descriptionhttps://build.chromium.org/p/tryserver.webrtc/builders/presubmit/ Oldest job has been waiting for more than 4 hours.
,
Oct 21 2016
Looking...
,
Oct 21 2016
Not broken. Just too many jobs. We should allocate a separate machine or two for it though, so it's not starved by other bots being queued up. There was also an outage: bug 658258
,
Oct 21 2016
tl;dr Yes. (Since I wrote this anyway, might just as well send). I think the slave pool shared by many builders on this master (slave*-c4 machines) is just overloaded. I checked all the slaves, and they all busy running some jobs. The similar queue has formed on all other builders too, e.g. https://build.chromium.org/p/tryserver.webrtc/builders/android_rel Here's total number of pending jobs on the master for the last 7 days: http://shortn/_oyP97GHCPy. Either is something really broken somewhere (I could not find where), or you just have an unusual high number of commits. --- Re bug 658258: it's about inability to start new VMs. All already running VMs aren't affected. It affects WebRTC master is a sense that we won't be able to add more slaves right away.
,
Oct 24 2016
Vadim: thanks for the thorough explanation and the stats link (we'll have to learn more how to monitor our masters, since it hasn't been an issue like this so far since we've had clear bottlenecks earlier). I've got 10 new VMs now that I will deploy to begin with + dedicate two only for the presubmit builder.
,
Oct 24 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/d77ad017a9799ee999cb8ac539c9ce35a8114839 commit d77ad017a9799ee999cb8ac539c9ce35a8114839 Author: kjellander <kjellander@chromium.org> Date: Mon Oct 24 04:37:25 2016 WebRTC: Add more VMs + dedicate two for presubmit bot. * Add 10 new VMs. * Dedicate two VMs for the presubmit builder, to avoid it being starved when there are too many jobs hogging the other machines. BUG=657563, 658261 TBR=ehmaldonado@chromium.org Review-Url: https://codereview.chromium.org/2447613002 [modify] https://crrev.com/d77ad017a9799ee999cb8ac539c9ce35a8114839/masters/master.tryserver.webrtc/slaves.cfg
,
Oct 24 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by d...@chromium.org
, Oct 21 2016Labels: Infra-Troopers