GetUploadTimeout() in upload_symbols.py calculates the timout with the following formula:
 timeout = max(filesize / 700MB * 1800 seconds, 120 seconds)
(https://chromium.googlesource.com/chromiumos/chromite/+/master/scripts/upload_symbols.py)

UPLOAD_MIN_TIMEOUT = 120 seconds is used as the minimum timeout to be robust against delays in connection set up.
However, from the log (pasted below), it seems connection set up may sometimes take more than 3 minutes.
We should change the value of UPLOAD_MIN_TIMEOUT to something larger than 3 minutes (e.g. 5 minutes).

> https://uberchromegw.corp.google.com/i/chromeos/builders/sumo-release/builds/861/steps/DebugSymbols/logs/stdio
14:06:56: INFO: Uploading symbol_file: git-prune/E94DF69D5A4FD3CA3A0FF10B61C349B10/git-prune.sym
14:06:57: INFO: upload of    2994813 bytes took 0:00:01.139412
14:06:57: INFO: Push state size: 96
14:06:57: INFO: sent git-prune.sym
14:06:57: INFO: Uploading symbol_file: git-merge/E94DF69D5A4FD3CA3A0FF10B61C349B10/git-merge.sym
14:10:08: INFO: upload of    2994813 bytes took 0:03:11.200617
14:10:08: INFO: Push state size: 96
14:10:09: INFO: sent git-merge.sym
14:10:09: INFO: Uploading symbol_file: git-stage/E94DF69D5A4FD3CA3A0FF10B61C349B10/git-stage.sym
14:10:09: INFO: upload of    2994813 bytes took 0:00:00.119050
14:10:09: INFO: Push state size: 96
14:10:09: INFO: sent git-stage.sym

In this example log, the sizes of git-prune.sym, git-merge.sym, and git-stage.sym are almost equal, but only git-merge.sym takes 3+ minutes more to finish uploading.

Uploaded a CL: https://chromium-review.googlesource.com/c/443946/

Please note that that timeout is per-symbol file, and total uploads can already take a long time. I'm saying that we shouldn't increase the timeout, just saying that we need to watch what it does to build time.

The below numbers show that symbol uploads normally take an hour, and have taken up to 3 hours. The numbers cover all successful canary builds during the last month.

DebugSymbols:
  duration:  median 1:02:50 mean 1:08:47 min 0:07:58 max 3:03:32

I think this shouldn't affect the total duration of the stage much, because this timeout is observed only once per build when it happens.
It can take about a few minutes longer than before, but I don't expect it take another couple of hours.

I agree, but want to confirm.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c00493767a5e9724d6cb34da9bb9fc4807db6e74

commit c00493767a5e9724d6cb34da9bb9fc4807db6e74
Author: Ryo Hashimoto <hashimoto@google.com>
Date: Sat Feb 25 18:32:06 2017

upload_symbols: Increase UPLOAD_MIN_TIMEOUT

Connection set up may sometimes take more than 3 minutes.
Increase UPLOAD_MIN_TIMEOUT to be robust against this.

BUG= chromium:692976 
TEST=./scripts/upload_symbols_unittest

Change-Id: Id1abc1f0770763c1595aa330406d42d915c51957
Reviewed-on: https://chromium-review.googlesource.com/443946
Commit-Ready: Ryo Hashimoto <hashimoto@chromium.org>
Tested-by: Ryo Hashimoto <hashimoto@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/c00493767a5e9724d6cb34da9bb9fc4807db6e74/scripts/upload_symbols.py

Not seeing upload timeout with recent builds.

Sweet!

Did anyone ever ping the crash server team? Maybe we should tell them if we're having performance problems instead of only working around them.

I guess it's WAI even when symbol upload takes a number of minutes for a relatively small symbol file.
sym_upload is the default symbol upload tool provided by breakpad, but it seems it's not caring about timeout.
https://cs.chromium.org/chromium/src/breakpad/src/common/linux/http_upload.cc?q=time.?out+file:%5Esrc/breakpad/src/common/+package:%5Echromium$&l=118&dr=C