LUCI outage due to luci-config serving 404 on all requests |
|||
Issue description(Note: this bug is public) A change [1] has been deployed that caused luci-config to serve 404 on all requests. 404 is interpreted by clients as "no such config file". This caused many LUCI services to "forget" their configs: all Milo consoles gone, Logdog refusing log uploads due to "no such project" error, all luci-scheduler jobs gone, etc. We reverted the change, the services should be recovering now. Assigning this to myself as on-call, to monitor the recovery. [1] https://chromium.googlesource.com/infra/luci/luci-py/+/2bb891ae32138026f1762af2cd1fa8d000dbe648
,
Jun 8 2018
Issue 850791 has been merged into this issue.
,
Jun 8 2018
It appears everything has come back online successfully. Here is a graph (for those who have access) of infra build errors spiking and coming back down: http://shortn/_j1gfxRtJDy Looking at other graphs, the only anomaly I don't understand is a huge spike in Logdog HTTP 500 (1500 QPS!): http://shortn/_rkZKYvRUhc
,
Jun 8 2018
Assigning to Sana to write a postmortem when details of the root cause are clear.
,
Jun 12 2018
Sana wrote go/cit-pm-85
,
Jun 12 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by hinoka@chromium.org
, Jun 8 2018