That's using the new Go code, so maybe a regression with a lack (?) of proper retry there.

 Issue 917183  has been merged into this issue.

Likely caused by side effect of issue 894045.

Raising to P1 regression per  Issue 917183 . Please see more detailed investigation there. This is affecting many of the GPU bots and warrants immediate attention.

Humm it looks like Takuto is OOO for two weeks, and I'm going away too. We could switch the use_go_client flag, but I'm not confident enough to be sure it won't cause a long trail of failure in dependent project (v8, webrtc, dart).

I think retry is handled around
https://cs.chromium.org/chromium/infra/go/src/go.chromium.org/luci/client/cmd/swarming/collect.go?l=257&rcl=c0b14c8cd84155eef2c84619a837a817f0d27684

But retry did not work as intended?
I'll investigate, but I need more logs first maybe.

Still happening; see for example this run:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20FYI%20Release%20%28AMD%29/3909

both depth_capture_tests and gpu_process_launch_tests reported internal swarming failures for their shards, but both completed successfully, and quickly.

https://chromium-swarm.appspot.com/user/task/42296766c4b00a10
https://chromium-swarm.appspot.com/user/task/422967ca3e987f10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/144fe563d8b88a7eb8b765e19eb9ef8274d2f625

commit 144fe563d8b88a7eb8b765e19eb9ef8274d2f625
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Mon Jan 07 18:23:46 2019

[swarming] reduce the number of network workers

This is to reduce the too many open files error in 919079.
Also I want to see whether this helps to mitigate 915332.

Bug:  919079 , 915332
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I5d3c9d95d692aef5c64b984b1115808d2c901611
Reviewed-on: https://chromium-review.googlesource.com/c/1395713
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>

[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux_ubsan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_gtest_mac_gpu.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_win64_release.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux64_debug_failing_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_android64__m_nexus5x_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_mac_asan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux32_debug.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_mac_asan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/isolated_script_test_custom_merge_script_with_args.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_ubsan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_x86_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_mac_experimental.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_win32_release.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android64__l_nexus9_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_android32__m_nexus5x__failing_archive.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_perf_win7.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/isolated_script_test_custom_merge_script.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_harness_failure.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/dynamic_swarmed_isolated_script_test_failure_no_result_json.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_interrupted.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_branches_mac__beta_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_with_bad_retcode_isolated_script_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_linux_trusty.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android32__l_nexus5_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_x86_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_experimental.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux_msan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/swarming_test_failure.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_branches_win__beta_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x64_msvc_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_missing_all_shards.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_android_arm_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/more_configs.expected/luci_webrtc_ci_win__more_configs_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipe_modules/swarming/api.py
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux64_debug_forced.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x64_clang_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_msan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_win64_debug.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x86_clang_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android32__k_nexus5_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android64__n_pixel_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux64_release.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x64_msvc_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_android_experimental.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_failed_isolated_script_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x64_clang_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/more_configs.expected/luci_webrtc_try_win_x86_more_configs.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x86_msvc_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_with_args_on_trybot.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x86_clang_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_memcheck.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipe_modules/swarming/examples/full.expected/isolated_script_non_zero_exit_status.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_perf_linux_trusty.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_branches_mac__stable_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_win32_debug.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_mac_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_branches_linux__beta_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android32__l_nexus6_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_mac64_debug.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/more_configs.expected/luci_webrtc_ci_linux__more_configs_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android32__l_nexus4_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_on_trybot_passing.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_missing_shard.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_corrupt_json_isolated_script_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_android32__m_nexus5x__dbg_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_asan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_mac_gpu.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_asan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_perf_mac_10_11.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_win7.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/dynamic_isolated_script_test_on_trybot_failing.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x64_clang_dbg_win10.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux_asan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux64_debug.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/gtest_custom_merge_script.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_linux_gpu.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/swarmed_layout_tests_too_many_failures_for_retcode.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_win32_release__clang_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_mac_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x64_clang_dbg_win8.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_failed_isolated_script_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_android32__m_nexus5x_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_android_arm64_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux_ubsan_vptr.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_tsan2.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android32__n_nexus6_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/swarming_test_failure_no_patch_deapplication.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/more_configs.expected/luci_webrtc_try_linux_more_configs.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/swarming_test_with_priority_expiration_and_timeout.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_branches_linux__stable_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_android_arm64_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/client_webrtc_branches_win__stable_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_win32_asan.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_android64__m_nexus5x__dbg_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_isolated_script_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_android_arm_rel.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_passed_isolated_script_test_with_swarming_failure.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_win_non_gpu.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_failure_no_result_json.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_mac_10_11.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_win_x86_msvc_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_serialized_gtests.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_disabled.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_dbg.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_script_test_harness_failure.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_invalid_json_isolated_script_test.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_perf_perf_android32__l_nexus7_2_.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_try_linux_ubsan_vptr.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium_trybot.expected/swarmed_webkit_tests_unexpected_error.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_mac64_release.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_script_test_missing_shard.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/isolated_script_test_custom_results_handler.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_win_gpu.json
[modify] https://crrev.com/144fe563d8b88a7eb8b765e19eb9ef8274d2f625/scripts/slave/recipes/webrtc/standalone.expected/luci_webrtc_ci_linux_tsan_v2.json

If we don't see the exception for a week, I will mark this bug fixed.

The NextAction date has arrived: 2019-01-14

This is yet seems to happen.
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8924700732791617824/+/steps/webkit_layout_tests/0/stdout

Let me implement retry logic for connection reset error.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/223b733633918210bb349e973e2d1a305b3e3e5f

commit 223b733633918210bb349e973e2d1a305b3e3e5f
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Wed Jan 16 21:47:16 2019

[swarming] do not retry only when we see google api HTTP codes < 500

We see some network error when collecting tasks.
I'd like to retry in such cases too.

Bug: 920691, 915332
Change-Id: Ibf940e160aacf038e95a697795e88204b8b200f9
Reviewed-on: https://chromium-review.googlesource.com/c/1414553
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>

[modify] https://crrev.com/223b733633918210bb349e973e2d1a305b3e3e5f/client/cmd/swarming/common.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/recipes-py/+/3b4c9d9b26618f331b01287b466946bd768170cd

commit 3b4c9d9b26618f331b01287b466946bd768170cd
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Thu Jan 17 00:11:43 2019

[swarming] roll swarming client

This is to pick up
https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/1414553

Bug: 920691, 915332
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: build
Change-Id: I5cf75614e904321b860bc2a7d20240b72c02e2b9
Reviewed-on: https://chromium-review.googlesource.com/c/1416490
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Auto-Submit: Takuto Ikuta <tikuta@chromium.org>

[modify] https://crrev.com/3b4c9d9b26618f331b01287b466946bd768170cd/recipe_modules/swarming/api.py
[modify] https://crrev.com/3b4c9d9b26618f331b01287b466946bd768170cd/recipe_modules/swarming/examples/full.expected/override swarming.json
[modify] https://crrev.com/3b4c9d9b26618f331b01287b466946bd768170cd/recipe_modules/swarming/examples/full.expected/basic.json