New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 658261 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Oct 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

WebRTC's presubmit bot is broken

Project Member Reported by ehmaldonado@chromium.org, Oct 21 2016

Issue description

https://build.chromium.org/p/tryserver.webrtc/builders/presubmit/

Oldest job has been waiting for more than 4 hours.
 

Comment 1 by d...@chromium.org, Oct 21 2016

Components: -Infra>Labs Infra
Labels: Infra-Troopers
Looking...
Not broken. Just too many jobs. We should allocate a separate machine or two for it though, so it's not starved by other bots being queued up.
There was also an outage: bug 658258
tl;dr Yes.

(Since I wrote this anyway, might just as well send).

I think the slave pool shared by many builders on this master (slave*-c4 machines) is just overloaded. I checked all the slaves, and they all busy running some jobs. 

The similar queue has formed on all other builders too, e.g. https://build.chromium.org/p/tryserver.webrtc/builders/android_rel

Here's total number of pending jobs on the master for the last 7 days: http://shortn/_oyP97GHCPy. Either is something really broken somewhere (I could not find where), or you just have an unusual high number of commits.

---

Re bug 658258: it's about inability to start new VMs. All already running VMs aren't affected. It affects WebRTC master is a sense that we won't be able to add more slaves right away.
Cc: vadimsh@chromium.org
Owner: kjellander@chromium.org
Status: Started (was: Untriaged)
Vadim: thanks for the thorough explanation and the stats link (we'll have to learn more how to monitor our masters, since it hasn't been an issue like this so far since we've had clear bottlenecks earlier).

I've got 10 new VMs now that I will deploy to begin with + dedicate two only for the presubmit builder.
Project Member

Comment 6 by bugdroid1@chromium.org, Oct 24 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/d77ad017a9799ee999cb8ac539c9ce35a8114839

commit d77ad017a9799ee999cb8ac539c9ce35a8114839
Author: kjellander <kjellander@chromium.org>
Date: Mon Oct 24 04:37:25 2016

WebRTC: Add more VMs + dedicate two for presubmit bot.

* Add 10 new VMs.
* Dedicate two VMs for the presubmit builder, to avoid it being
  starved when there are too many jobs hogging the other machines.

BUG=657563, 658261 
TBR=ehmaldonado@chromium.org

Review-Url: https://codereview.chromium.org/2447613002

[modify] https://crrev.com/d77ad017a9799ee999cb8ac539c9ce35a8114839/masters/master.tryserver.webrtc/slaves.cfg

Status: Fixed (was: Started)

Sign in to add a comment