Issue metadata
Sign in to add a comment
|
chromium_presubmit and other bots (luci.chromium.ci+luci.chromium.try) are failing to start |
||||||||||||||||||||||
Issue descriptione.g. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit/33444 shows "Internal Failure" and "LogDog stream not found / Job likely failed to start." This prevents CLs from landing as this bot is required by the commit queue.
,
Feb 13 2018
(Thanks to katydek@ for noticing this quickly.)
,
Feb 13 2018
Hi is there any update on this?
,
Feb 13 2018
AFAICT this affects all bots on luci.chromium.ci and luci.chromium.try, at least.
,
Feb 13 2018
I (sheriff) have closed the tree and have been trying to get in touch with iannucci@.
,
Feb 13 2018
,
Feb 13 2018
We are trying to mitigate with short term fix. If all goes well, things will get back to normal in 10 minutes.
,
Feb 13 2018
Making this bug public.
,
Feb 13 2018
The problem appears to be exhaustion of quota by pool service account under which recipes are bootstrapped. We are swapping the accounts now as short term fix. Long term fix will be increasing quota for pool account.
,
Feb 13 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/4654080ba7cf3ab07597ddacdf7d047c7baf98db commit 4654080ba7cf3ab07597ddacdf7d047c7baf98db Author: Vadim Shtayura <vadimsh@chromium.org> Date: Tue Feb 13 18:54:16 2018
,
Feb 13 2018
Note for posterity: we need to consider firing pre-emptive alerts when any of our accounts are approaching quota. Maybe doing so would be the wrong decision but it seems like it's time to revisit the question. After this outage is resolved.
,
Feb 13 2018
Outage has been mitigated. We'll now be requesting higher quota for this account after estimating how much it actually needs. Re #11: yes, iannucci@ and I have been brainstorming how to achieve that yesterday; we don't yet have an idea, but IMO it's worth looking.
,
Feb 13 2018
,
Feb 14 2018
Medium-term: we are getting higher quota in http://b/73310560 Long-term: stop burning git quota in recipe bootstrap: issue 811974 alert on soon-to-exhaust quota: issue 812042
,
Feb 14 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/c74d62d3ef9eea8faf0c66c6b6d06ab3d956fda7 commit c74d62d3ef9eea8faf0c66c6b6d06ab3d956fda7 Author: Andrii Shyshkalov <tandrii@chromium.org> Date: Wed Feb 14 22:58:44 2018 |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by jbroman@chromium.org
, Feb 13 2018Owner: iannucci@chromium.org