New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 728246 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: Bug



Sign in to add a comment

Daily bandwidth rate limit exceeded for <> fatal: protocol error: bad pack header error: Could not fetch origin

Project Member Reported by jmad...@chromium.org, May 31 2017

Issue description

POST git-upload-pack (420 bytes)
remote: Daily bandwidth rate limit exceeded for 182615506979@project.gserviceaccount.com
fatal: protocol error: bad pack header
error: Could not fetch origin

Seems bad.

https://build.chromium.org/p/tryserver.chromium.angle/builders/linux_angle_chromeos_rel_ng/builds/3106

Seems to be causing a lot of failures right now.


 
Cc: kbr@chromium.org ynovikov@chromium.org jmad...@chromium.org
Components: Internals>GPU>Testing Infra
Labels: Infra-Troopers Hotlist-Infra-Troopers Hotlist-PixelWrangler
Cc: estaab@chromium.org seanmccullough@chromium.org
This is causing almost all bots to fail on some CLs: https://chromium-review.googlesource.com/c/519643/
We're chatting with GoB to get quota reset for the day.
Owner: seanmccullough@chromium.org
Status: Assigned (was: Untriaged)
Assigning to Sean. ANGLE CQ is still down.
we're seeing fleet-wide GoB quota issues. We asked for a temporary increase earlier this AM but this error is saying "Short term bandwidth rate limit exceeded for 182615506979@project.gserviceaccount.com" - talking to GoB oncall now.

Comment 6 by no...@chromium.org, May 31 2017

fwiw there are 10 bots that run experimental {linux|mac|win}_chromium_rel_ng on LUCI. They fetch recipes (recipes-py.git, depot_tools.git, build.git) via git without caching.
I don't think it is the cause though, because it is just 10 bots and they don't fetch src.git without cache (they fetch src.git with git-cache).
example of a LUCI build failed due to rate limit error https://chromium-swarm.appspot.com/task?id=3678c62502f40610&refresh=10&show_raw=1&wide_logs=true
Agree those 10 bots are probably not the culprit.

I believe this is the same gob quota exceeded issue as the others today. 

Currently retrying https://chromium-review.googlesource.com/c/519643/ mentioned in #2 on CQ and it has so far succeeded in update_scripts steps where the earlier attempts today failed.
Seems to be fixed after the quota reset.
Status: Fixed (was: Assigned)
Thanks for the quota reset!

Sign in to add a comment