cipd was too slow?

[P14042 12:59:13.504 client.go:945 I] Packages to be installed:
[P14042 12:59:13.504 client.go:947 I]   infra_internal/goma/client/linux-amd64:b246cbd5271af55bfda4f8051f6b26762dc831ab
[P14042 12:59:13.505 client.go:824 I] cipd: resolving fetch URL for infra_internal/goma/client/linux-amd64:b246cbd5271af55bfda4f8051f6b26762dc831ab
[P14042 12:59:14.080 storage.go:249 I] cipd: initiating the fetch
[P14042 12:59:20.458 storage.go:214 I] cipd: about to fetch 19.9 Mb
[P14042 12:59:20.458 storage.go:205 I] cipd: fetching - 0%
...
[P14042 13:08:23.152 storage.go:205 I] cipd: fetching - 100%
[P14042 13:08:23.152 storage.go:221 I] cipd: fetch finished successfully
[P14042 13:08:23.152 client.go:779 I] cipd: successfully fetched infra_internal/goma/client/linux-amd64:b246cbd5271af55bfda4f8051f6b26762dc831ab

dba, are we still short on nat ports? I remember we had this issue a couple of weeks ago

It fetches directly from Google Storage (starting from line "[P14042 12:59:20.458 storage.go:214 I] cipd: about to fetch 19.9 Mb").

:( 

Not sure how to workaround that.

(Also, NAT port shortage would have resulted in a failure to establish a connection, not slow download).

Downloading clang packages from Google Storage using a https url ('https://commondatastorage.googleapis.com/chromium-browser-clang' -- https://cs.chromium.org/chromium/src/tools/clang/scripts/update.py?q=update.py+file:clang&sq=package:chromium&dr=C&l=407) using python's standard urllib2 is fast -- maybe your storage config has something that makes it slow?

It's probably compounded by the fact we've been maxing out or uplinks for the past 2-3 weeks. We have maintenance scheduled for tomorrow to work on upgrading the link.

cipd is fast in vast majority of cases too (it's the first time I see it reported as slow). Here how it works usually: https://build.chromium.org/p/chromium/builders/Linux%20x64/builds/20422/steps/ensure_goma.ensure_installed/logs/stdio

It seems to happening right now too. I'll SSH on the machine and investigate.

It's an issues with machine and/or Golo. 

I did "wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test10.zip" and it downloads the file at whooping 30 Kb/s. Modem speed basically :-/

(So dba@ conclusion seems to be correct).

Though I also filed https://bugs.chromium.org/p/chromium/issues/detail?id=617686, since it shouldn't be downloading anything at all.

PCR completed. 

ensure_goma step is fast again https://build.chromium.org/p/chromium/builders/Linux%20x64/builds/20471/steps/ensure_goma