corrupt or partial autotest_server_package.tar.bz2 download causes ssp failure |
||||||
Issue descriptionBuild in question: https://luci-milo.appspot.com/buildbot/chromeos/veyron_speedy-paladin/5060 04/21 04:27:30.397 DEBUG| dev_server:0908| Error occurred with exit_code 18 when executing the ssh call: Warning: Permanently added '172.24.184.160' (RSA) to the list of known hosts. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 14 14.4M 14 2128k 0 0 28349 0 0:08:55 0:01:16 0:07:39 28349 14 14.4M 14 2224k 0 0 11630 0 0:21:46 0:03:15 0:18:31 11630 15 14.4M 15 2336k 0 0 11182 0 0:22:38 0:03:33 0:19:05 11182 17 14.4M 17 2560k 0 0 12227 0 0:20:42 0:03:34 0:17:08 12227 21 14.4M 21 3120k 0 0 14829 0 0:17:04 0:03:35 0:13:29 14829 23 14.4M 23 3456k 0 0 16235 0 0:15:35 0:03:37 0:11:58 9636 24 14.4M 24 3568k 0 0 16729 0 0:15:08 0:03:38 0:11:30 60934 26 14.4M 26 3904k 0 0 17655 0 0:14:20 0:03:46 0:10:34 125k 27 14.4M 27 4016k 0 0 16003 0 0:15:49 0:04:16 0:11:33 35020 27 14.4M 27 4128k 0 0 16392 0 0:15:26 0:04:17 0:11:09 24325 28 14.4M 28 4224k 0 0 16718 0 0:15:08 0:04:18 0:10:50 19305 29 14.4M 29 4416k 0 0 17404 0 0:14:33 0:04:19 0:10:14 20964 30 14.4M 30 4512k 0 0 17745 0 0:14:16 0:04:20 0:09:56 18349 31 14.4M 31 4720k 0 0 18492 0 0:13:41 0:04:21 0:09:20 160k 33 14.4M 33 4944k 0 0 19283 0 0:13:07 0:04:22 0:08:45 174k 34 14.4M 34 5152k 0 0 20031 0 0:12:38 0:04:23 0:08:15 199k 36 14.4M 36 5472k 0 0 21174 0 0:11:57 0:04:24 0:07:33 219k 38 14.4M 38 5664k 0 0 21825 0 0:11:36 0:04:25 0:07:11 214k 38 14.4M 38 5776k 0 0 22195 0 0:11:24 0:04:26 0:06:58 206k 40 14.4M 40 5984k 0 0 22891 0 0:11:03 0:04:27 0:06:36 201k 41 14.4M 41 6192k 0 0 23585 0 0:10:44 0:04:28 0:06:16 190k 42 14.4M 42 6288k 0 0 23906 0 0:10:35 0:04:29 0:06:06 173k 43 14.4M 43 6496k 0 0 24539 0 0:10:19 0:04:31 0:05:48 156k 44 14.4M 44 6592k 0 0 24872 0 0:10:10 0:04:31 0:05:39 165k 47 14.4M 47 7008k 0 0 26346 0 0:09:36 0:04:32 0:05:04 218k 50 14.4M 50 7536k 0 0 28216 0 0:08:58 0:04:33 0:04:25 288k 53 14.4M 53 7936k 0 0 29609 0 0:08:33 0:04:34 0:03:59 322k 55 14.4M 55 8256k 0 0 30695 0 0:08:14 0:04:35 0:03:39 404k 57 14.4M 57 8560k 0 0 31716 0 0:07:59 0:04:36 0:03:23 395k 60 14.4M 60 8976k 0 0 33119 0 0:07:38 0:04:37 0:03:01 382k 62 14.4M 62 9296k 0 0 34169 0 0:07:24 0:04:38 0:02:46 344k 64 14.4M 64 9632k 0 0 35279 0 0:07:10 0:04:39 0:02:31 331k 67 14.4M 67 9952k 0 0 36343 0 0:06:58 0:04:40 0:02:18 340k 69 14.4M 69 10.0M 0 0 37376 0 0:06:46 0:04:41 0:02:05 338k 71 14.4M 71 10.3M 0 0 38412 0 0:06:35 0:04:42 0:01:53 334k 72 14.4M 72 10.5M 0 0 38940 0 0:06:30 0:04:42 0:01:48 350k curl: (18) transfer closed with 4183678 bytes remaining to read . 04/21 04:27:30.402 WARNI| retry:0238| <class 'autotest_lib.client.common_lib.error.CmdError'>(Command <ssh 172.24.184.160 'curl "http://172.24.184.160:8082/static/veyron_speedy-paladin/R60-9481.0.0-rc2/autotest_server_package.tar.bz2"'> failed, rc=18, Command returned non-zero exit status * Command: ssh 172.24.184.160 'curl "http://172.24.184.160:8082/static /veyron_speedy-paladin/R60-9481.0.0-rc2/autotest_server_package.tar.bz2"' Exit status: 18 Duration: 291.318702936
,
Apr 21 2017
> Possibly this is due to network overload on the shard. I haven't > tried to correlate the time. Not impossible for it to be overload on the devserver. I note that the devserver in this case is not in the lab; it's chromeos-server87.hot. See also bug 712274 for another example of a network problem (presumably load) between a shard and a non-lab devserver. Given the current state of things, I'm inclined to blame both this problem and that on the shard, not the devserver.
,
Apr 21 2017
It looks to me like we are separately downloading this ssp package to the shard for every test (or every server-side test that uses ssp?) Isn't that wasteful? Can we download once to shard and cache is somewhere?
,
Apr 21 2017
,
Apr 21 2017
related bug: crbug.com/712283 timeout for downloading autotest_server_package.tar.bz2 after 5 mins
,
Apr 22 2017
Re #3 The bz2 file is only around 5-8MB, I think the overhead is minimum.
,
Apr 25 2017
,
Mar 14 2018
This bug is Untriaged and very old. Because it has an owner, the status will be set to assigned to avoid closing a bug someone is using. If this bug still needs triage, change it back to Untriaged.
,
Jun 8 2018
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by akes...@chromium.org
, Apr 21 2017