as seen in issue 922840 (only the latest occurrence), our infra sometimes run into flakes in GS/gsutil, particularly related to locks/generation handling. b/112110347 has more details too.
we could mitigate this on our side by refreshing the locks less often. we have to refresh the locks so that we don't (incorrectly) break the lock because a particular job is taking longer than normal. but we should be able to add some logic to the core refresh func like:
- if last refresh was <1 min ago, return immediately
- otherwise, refresh the lock, and update the "last refresh" time
the signers currently set the lock timeout to 60 min. so using 1min (or maybe even 5min?) shouldn't be too bad as that's still <10% of the time for a timeout to be hit.