Reduce buildbucket 403s by finding masters that peek nonexistent buckets |
||
Issue descriptionbuildbucket has a background level of about 1.5qps of 403s. This isn't harmful to the app, but makes diagnosis of new regressions a bit harder. We should hunt through master configs to find those that look for buckets which don't actually exist, and fix them. If this reduced the rate of 403s, we should correspondingly lower the alert threshold.
,
May 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/a53da2efbfa651d6437cbbedbe6222d512d3ef65 commit a53da2efbfa651d6437cbbedbe6222d512d3ef65 Author: Nodir Turakulov <nodir@google.com> Date: Thu May 04 16:04:45 2017 disconnect 3 masters from buildbucket masters chromium.android and client.webrtc.{branches, perf} poll non-existent build buckets and cause high HTTP 403 QPS on buildbucket. Stop the polling. R=agable@chromium.org BUG= 718224 Change-Id: Iab3c1ff9556f1daeccb82b78325d715c02ed5664 Reviewed-on: https://chromium-review.googlesource.com/495590 Reviewed-by: Aaron Gable <agable@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/a53da2efbfa651d6437cbbedbe6222d512d3ef65/masters/master.client.webrtc.branches/master_site_config.py [modify] https://crrev.com/a53da2efbfa651d6437cbbedbe6222d512d3ef65/masters/master.client.webrtc.perf/master_site_config.py [modify] https://crrev.com/a53da2efbfa651d6437cbbedbe6222d512d3ef65/masters/master.chromium.android/master_site_config.py
,
May 4 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/5175beeb2685903557ba09b30f5f0efcf4bfd6b2 commit 5175beeb2685903557ba09b30f5f0efcf4bfd6b2 Author: Nodir Turakulov <nodir@google.com> Date: Thu May 04 17:23:27 2017
,
May 4 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/2c3b237e92dde1c17775620ba53d258fa056623c commit 2c3b237e92dde1c17775620ba53d258fa056623c Author: Nodir Turakulov <nodir@google.com> Date: Thu May 04 23:02:19 2017
,
May 5 2017
http 403 rate is down to 0
,
May 5 2017
Awesome! Want to lower the alert threshold as well so it doesn't climb up again?
,
May 5 2017
I sent you an internal CL, but didn't update the bug because the latter is public
,
May 5 2017
the alert threshold was lowered |
||
►
Sign in to add a comment |
||
Comment 1 by no...@chromium.org
, May 4 2017