New issue
Advanced search Search tips

Issue 913920 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Dec 11
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Task



Sign in to add a comment

storage.googleapis.com slow on Dec 11, 2018

Project Member Reported by fhorschig@chromium.org, Dec 11

Issue description

There was a tree closure because a call to https://storage.googleapis.com/nativeclient-once/object/newlib_arm_bc_a708c95953a96292307ec8cbaa0c9db31caf81bb.tgz timed out but it was reachable within the same second.

Another indication that storage.googleapis.com is slow today, is this failure:

The context_lost_tests failed on chromium.gpu/Android Release (Nexus 5X):
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20Release%20%28Nexus%205X%29/13715

Log snippet:
15604 2018-12-11 11:00:08.070 E: internal failure: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out.
Traceback (most recent call last):
  File "/b/swarming/swarming_bot.1.zip/client/run_isolated.py", line 734, in map_and_run
    use_symlinks=data.use_symlinks)
  File "/b/swarming/swarming_bot.1.zip/client/run_isolated.py", line 531, in fetch_and_map
    use_symlinks=use_symlinks)
  File "/b/swarming/swarming_bot.1.zip/client/isolateserver.py", line 1294, in fetch_isolated
    digest = fetch_queue.wait()
  File "/b/swarming/swarming_bot.1.zip/client/isolateserver.py", line 960, in wait
    digest = self._channel.next()
  File "/b/swarming/swarming_bot.1.zip/utils/threading_utils.py", line 377, in _task_executer
    result = func(*args, **kwargs)
  File "/b/swarming/swarming_bot.1.zip/client/isolateserver.py", line 865, in fetch
    sink(verifier.run())
  File "/b/swarming/swarming_bot.1.zip/client/local_caching.py", line 586, in write
    size = file_write(path, content)
  File "/b/swarming/swarming_bot.1.zip/client/local_caching.py", line 41, in file_write
    for d in content_generator:
  File "/b/swarming/swarming_bot.1.zip/client/isolateserver.py", line 1032, in run
    for chunk in self.stream:
  File "/b/swarming/swarming_bot.1.zip/client/isolateserver.py", line 262, in zip_decompress
    for chunk in content_generator:
  File "/b/swarming/swarming_bot.1.zip/client/isolate_storage.py", line 395, in fetch
    for data in connection.iter_content(NET_IO_FILE_CHUNK):
  File "/b/swarming/swarming_bot.1.zip/utils/net.py", line 691, in iter_content
    raise TimeoutError(e)
TimeoutError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out.
HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out.
15604 2018-12-11 11:00:09.091 E: Failed to fetch 2694b83f37f00bc170fbb7bd3189ba4e911846ab: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))
 
Labels: Infra-Troopers
+ Trooper FYI.

The report is for swarm236-c4, us-central1-f. I suspect internal transient failure.
Owner: sergeybe...@chromium.org
Status: WontFix (was: Available)
https://status.cloud.google.com/ doesn't show any outages, so it's probably a transient failure. Unfortunately, storage.googleapis.com is out of our control, and if it has a brief outage (a few minutes), it may cause a transient failure like that, but likely won't show up on Google-wide dashboard.

The builder is working fine now, closing as WontFix. If the issue persists, please reopen and provide specific details.

Sign in to add a comment