New issue
Advanced search Search tips

Issue 910413 link

Starred by 5 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature



Sign in to add a comment

Need buttons on the bot page to invalidate named cache(s)

Project Member Reported by iannu...@google.com, Nov 30

Issue description

I.e. on https://chromium-swarm.appspot.com/bot?id=swarm2468-c4&sort_stats=total%3Adesc we could get a button to delete 1 or more of the caches the bot currently has.

Ideally this could work in conjunction with a bot query so you could do something like:

  Find all bots with {os:Linux} and delete cache {git}

Or something like that.
 
In terms of semantics, it would be fine for the bot to just do this the next time it picked up a task; when it asks for a task, it would get an instruction (maybe a maintenance task?) from swarming to kill off the various named caches.
Status: Available (was: Untriaged)
Yes, please. I vote for maintenance task.
We had decided against that because there's nothing blocking from triggering a old poisoning task after the cleanup. The way we had addressed that was to change the named cache name, which is an order of magnitude safer.
I'm not sure I follow what you mean "triggering a old poisoning task after the cleanup"? Changing the cache name is actually harder, in most cases, than clicking a button (in this case, a small collection of builders put corrupt data in their caches. Changing the cache name would require modifying the buildbucket server code). The point is to aid in debugging, to see if the cache is, in fact, corrupt and causing a persistent issue, or to clear it as a one-time action.

I see no reason why we shouldn't have both options (i.e. "delete what's there now" and "never mind, I'll use a different cache because this one is hopelessly broken everywhere"). They solve different problems.
Ok fair enough, but I'd probably make it indiscriminate; flush all caches, not a subset.
Flush all sgtm; would help discourage abuse

Comment 7 by bpastene@chromium.org, Jan 16 (6 days ago)

Cc: bpastene@chromium.org
We're regularly running into situations where we need to clobber the caches on a bot or group of bots. ( Bug 921967  for the latest occurrence.)

As is, we've rolled our own chained swarming.py invocations, but it would be very helpful if there was a button on the web page for that. One could even be added to the botlist page for mass deletion; something like "delete all named caches" that works across all selected bots.

Comment 8 by sergeybe...@chromium.org, Jan 16 (6 days ago)

+1 to an easy way to clean up lots of caches (I don't have a strong opinion on the specific technology).

FWIW, this likely will be implemented one way or another, if not on the swarming side, then as an infra script that fires up a bunch of tasks deleting the caches. Because troopers need it. Pick your poison :-)

Sign in to add a comment