Issue 675483 link

Starred by 1 user

Issue metadata

Status:	Fixed
Owner:	vadimsh@chromium.org
Closed:	Dec 2016
Cc:	mar...@chromium.org
Components:	Infra>Platform>Swarming
EstimatedDays:	----
NextAction:	----
OS:	----
Pri:	2
Type:	Bug

Evil retry loop on 403 in Swarming bot code somewhere

Project Member Reported by vadimsh@chromium.org, Dec 18 2016

Issue description

We've been switching Fuchsia bots prefix from 'fuchsia-*' to 'fuchsia-vm-*', and forgot to shutdown 10 old 'fuchsia-bot-*' bots (than no longer match new prefix).

They managed to send 16153 requests (all failing with HTTP 403) in span of 20 min. At peak, these requests constituted 16% of ALL Swarming traffic: http://shortn/_6SOHnul1Vr

Looks like most of these calls were to /bot/event and /bot/poll endpoints.

Bots should do retry less vigorously.

Comment 1 by vadimsh@chromium.org, Dec 21 2016

Owner: vadimsh@chromium.org
Status: Started (was: Untriaged)

Found the bug. Fix: https://codereview.chromium.org/2593863002/

Project Member

Comment 2 by bugdroid1@chromium.org, Dec 21 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae

commit f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae
Author: vadimsh <vadimsh@chromium.org>
Date: Wed Dec 21 00:44:58 2016

Fix evil retry loop on poll errors.

There was no sleep when retrying poll errors (due to a regression).

BUG= 675483 
R=maruel@chromium.org

Review-Url: https://codereview.chromium.org/2593863002

[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/api/bot.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/bot_main.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/remote_client.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/remote_client_errors.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/remote_client_grpc.py

Project Member

Comment 3 by bugdroid1@chromium.org, Dec 21 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae

commit f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae
Author: vadimsh <vadimsh@chromium.org>
Date: Wed Dec 21 00:44:58 2016

Fix evil retry loop on poll errors.

There was no sleep when retrying poll errors (due to a regression).

BUG= 675483 
R=maruel@chromium.org

Review-Url: https://codereview.chromium.org/2593863002

[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/api/bot.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/bot_main.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/remote_client.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/remote_client_errors.py
[modify] https://crrev.com/f25ef8dfd624f0a4c42af66d9bab9d3c8a0523ae/appengine/swarming/swarming_bot/bot_code/remote_client_grpc.py

Comment 4 by vadimsh@chromium.org, Dec 21 2016

Status: Fixed (was: Started)

►

Issue 675483 link

Issue metadata

Evil retry loop on 403 in Swarming bot code somewhere

Issue description

Comment 1 by vadimsh@chromium.org, Dec 21 2016

Comment 1 Deleted

Comment 2 by bugdroid1@chromium.org, Dec 21 2016

Comment 2 Deleted

Comment 3 by bugdroid1@chromium.org, Dec 21 2016

Comment 3 Deleted

Comment 4 by vadimsh@chromium.org, Dec 21 2016

Comment 4 Deleted

Sign in to add a comment