In local repro, the full set of things we're pushing is 645 MB zipped and >950 MB (!) unzipped.

Here's everything over 10MB:

$ du -d 1 chrome/test/data third_party * | sort -n
10452   components/test
10456   components
10860   chrome/test/data/third_party
11992   chrome/test/data/image_decoding
20464   chrome/test/data/import
31568   net/data
32288   net
50764   chrome/test/data/pdf_private
65640   chrome/test/data/extensions
76788   chrome/test/data/perf
124716  chrome/test/data/gpu
192480  chrome/test/data/vr
303988  third_party/hunspell_dictionaries
310348  third_party
615132  chrome/test/data

This is a leading (sole?) cause of unit_test timeouts on android:
https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1512435960000&f=name%3Aunit_tests&f=os%3AAndroid&f=state%3ATIMED_OUT&l=1000&n=true&q=state&s=created_ts%3Adesc&st=1511744760000

On most of those, we're not actually running the tests until 10+ minutes into the task.

I think that I introduced a bug here in https://codereview.chromium.org/2502363005 :(

We were previously excluding things like the hunspell_dictionaries. I suspect we can exclude the vr test data as well.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/40af00b005ac282ad628e4ec149c836643d7bf1e

commit 40af00b005ac282ad628e4ec149c836643d7bf1e
Author: John Budorick <jbudorick@chromium.org>
Date: Thu Dec 07 03:46:51 2017

Remove hunspell_dictionaries from chrome/test:unit_tests data.

Not necessary since spellcheck was componentized.

Bug: 791632
Change-Id: I442a50cc15d1fc63714f65a2abbf62b8b254938c
Reviewed-on: https://chromium-review.googlesource.com/808984
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#522330}
[modify] https://crrev.com/40af00b005ac282ad628e4ec149c836643d7bf1e/chrome/test/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ce8da4e667fcf61dbcc9d45e82188249d42c3ba9

commit ce8da4e667fcf61dbcc9d45e82188249d42c3ba9
Author: John Budorick <jbudorick@chromium.org>
Date: Fri Dec 08 18:04:27 2017

[android] Fix minor bugs in --trace-all.

This CL:
 - disables tracing in subprocesses, which was causing occasional hangs
 - stops tracing at env teardown time

Bug: 791632
Change-Id: Iaa6cda1f1adcaa1f4cccd7ef147c944e63bdc689
Reviewed-on: https://chromium-review.googlesource.com/816061
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#522820}
[modify] https://crrev.com/ce8da4e667fcf61dbcc9d45e82188249d42c3ba9/build/android/pylib/local/device/local_device_environment.py
[modify] https://crrev.com/ce8da4e667fcf61dbcc9d45e82188249d42c3ba9/build/android/pylib/utils/instrumentation_tracing.py

Still happens occasionally.
https://ci.chromium.org/buildbot/tryserver.chromium.android/android_n5x_swarming_rel/422279
I   87.270s TimeoutThread-1-for-individual_device_set_up(0246e0d059a41794)  Pushing 13158 files via .zip of size 291708622
I   87.270s TimeoutThread-1-for-individual_device_set_up(0246e0d059a41794)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 0246e0d059a41794 push /b/swarming/w/it20PJD8/tmp1GF2IB/tmp.zip /data/local/tmp/temp_file-68cc65a5a7324.zip
I  129.903s individual_device_set_up(0246e0d059a41794)  Still working on PushChangedFiles(0246e0d059a41794, [('/b/swarming/w/ir/chrome/test/data', '/sdcard/chromium_tests_root/chrome/test/data'), ('/b/swarming/w/ir/base/test/data', '/sdcard/chromium_tests_root/base/test/data'), ('/b/swarming/w/ir/chrome/third_party/mock4js', '/sdcard/chromium_tests_root/chrome/third_party/mock4js'), ('/b/swarming/w/ir/components/test/data', '/sdcard/chromium_tests_root/components/test/data'), ('/b/swarming/w/ir/extensions/test/data', '/sdcard/chromium_tests_root/extensions/test/data'), ('/b/swarming/w/ir/google_apis/test/data', '/sdcard/chromium_tests_root/google_apis/test/data'), ('/b/swarming/w/ir/net/tools/testserver', '/sdcard/chromium_tests_root/net/tools/testserver'), ('/b/swarming/w/ir/third_party/accessibility-audit/axs_testing.js', '/sdcard/chromium_tests_root/third_party/accessibility-audit/axs_testing.js'), ('/b/swarming/w/ir/third_party/chaijs/chai.js', '/sdcard/chromium_tests_root/third_party/chaijs/chai.js'), ('/b/swarming/w/ir/third_party/pyftpdlib', '/sdcard/chromium_tests_root/third_party/pyftpdlib'), ('/b/swarming/w/ir/third_party/pywebsocket', '/sdcard/chromium_tests_root/third_party/pywebsocket'), ('/b/swarming/w/ir/third_party/tlslite', '/sdcard/chromium_tests_root/third_party/tlslite'), ('/b/swarming/w/ir/third_party/zlib/google/test/data', '/sdcard/chromium_tests_root/third_party/zlib/google/test/data'), ('/b/swarming/w/ir/tools/metrics/histograms/enums.xml', '/sdcard/chromium_tests_root/tools/metrics/histograms/enums.xml'), ('/b/swarming/w/ir/out/Release/pyproto/google', '/sdcard/chromium_tests_root/pyproto/google'), ('/b/swarming/w/ir/out/Release/chrome_100_percent.pak', '/sdcard/chromium_tests_root/paks/chrome_100_percent.pak'), ('/b/swarming/w/ir/out/Release/locales/en-US.pak', '/sdcard/chromium_tests_root/paks/en-US.pak'), ('/b/swarming/w/ir/out/Release/resources.pak', '/sdcard/chromium_tests_root/paks/resources.pak'), ('/b/swarming/w/ir/testing/buildbot/filters/unit_tests_cros_asan.filter', '/sdcard/chromium_tests_root/testing/buildbot/filters/unit_tests_cros_asan.filter'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/test.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/test.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/bad_format.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/bad_format.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/empty.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/empty.pb'), ('/b/swarming/w/ir/out/Release/icudtl.dat', '/sdcard/chromium_tests_root/icudtl.dat'), ('/b/swarming/w/ir/out/Release/content_shell.pak', '/sdcard/chromium_tests_root/paks/content_shell.pak'), ('/b/swarming/w/ir/out/Release/content_shell.pak.info', '/sdcard/chromium_tests_root/content_shell.pak.info'), ('/b/swarming/w/ir/out/Release/AHEM____.TTF', '/sdcard/chromium_tests_root/AHEM____.TTF'), ('/b/swarming/w/ir/out/Release/test_fonts/android_fallback_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_fallback_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/android_main_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_main_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/LICENSE', '/sdcard/chromium_tests_root/test_fonts/LICENSE'), ('/b/swarming/w/ir/out/Release/test_fonts/Ahem.ttf', '/sdcard/chromium_tests_root/test_fonts/Ahem.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Garuda.ttf', '/sdcard/chromium_tests_root/test_fonts/Garuda.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Devanagari.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Devanagari.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Gurmukhi.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Gurmukhi.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Tamil.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Tamil.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/MuktiNarrow.ttf', '/sdcard/chromium_tests_root/test_fonts/MuktiNarrow.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansCJKjp-Regular.otf', '/sdcard/chromium_tests_root/test_fonts/NotoSansCJKjp-Regular.otf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansKhmer-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/NotoSansKhmer-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Regular.ttf'), ('/b/swarming/w/ir/net/data', '/sdcard/chromium_tests_root/net/data')], retries=2, timeout=300, delete_device_stale=True)
I  189.944s individual_device_set_up(0246e0d059a41794)  Still working on PushChangedFiles(0246e0d059a41794, [('/b/swarming/w/ir/chrome/test/data', '/sdcard/chromium_tests_root/chrome/test/data'), ('/b/swarming/w/ir/base/test/data', '/sdcard/chromium_tests_root/base/test/data'), ('/b/swarming/w/ir/chrome/third_party/mock4js', '/sdcard/chromium_tests_root/chrome/third_party/mock4js'), ('/b/swarming/w/ir/components/test/data', '/sdcard/chromium_tests_root/components/test/data'), ('/b/swarming/w/ir/extensions/test/data', '/sdcard/chromium_tests_root/extensions/test/data'), ('/b/swarming/w/ir/google_apis/test/data', '/sdcard/chromium_tests_root/google_apis/test/data'), ('/b/swarming/w/ir/net/tools/testserver', '/sdcard/chromium_tests_root/net/tools/testserver'), ('/b/swarming/w/ir/third_party/accessibility-audit/axs_testing.js', '/sdcard/chromium_tests_root/third_party/accessibility-audit/axs_testing.js'), ('/b/swarming/w/ir/third_party/chaijs/chai.js', '/sdcard/chromium_tests_root/third_party/chaijs/chai.js'), ('/b/swarming/w/ir/third_party/pyftpdlib', '/sdcard/chromium_tests_root/third_party/pyftpdlib'), ('/b/swarming/w/ir/third_party/pywebsocket', '/sdcard/chromium_tests_root/third_party/pywebsocket'), ('/b/swarming/w/ir/third_party/tlslite', '/sdcard/chromium_tests_root/third_party/tlslite'), ('/b/swarming/w/ir/third_party/zlib/google/test/data', '/sdcard/chromium_tests_root/third_party/zlib/google/test/data'), ('/b/swarming/w/ir/tools/metrics/histograms/enums.xml', '/sdcard/chromium_tests_root/tools/metrics/histograms/enums.xml'), ('/b/swarming/w/ir/out/Release/pyproto/google', '/sdcard/chromium_tests_root/pyproto/google'), ('/b/swarming/w/ir/out/Release/chrome_100_percent.pak', '/sdcard/chromium_tests_root/paks/chrome_100_percent.pak'), ('/b/swarming/w/ir/out/Release/locales/en-US.pak', '/sdcard/chromium_tests_root/paks/en-US.pak'), ('/b/swarming/w/ir/out/Release/resources.pak', '/sdcard/chromium_tests_root/paks/resources.pak'), ('/b/swarming/w/ir/testing/buildbot/filters/unit_tests_cros_asan.filter', '/sdcard/chromium_tests_root/testing/buildbot/filters/unit_tests_cros_asan.filter'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/test.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/test.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/bad_format.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/bad_format.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/empty.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/empty.pb'), ('/b/swarming/w/ir/out/Release/icudtl.dat', '/sdcard/chromium_tests_root/icudtl.dat'), ('/b/swarming/w/ir/out/Release/content_shell.pak', '/sdcard/chromium_tests_root/paks/content_shell.pak'), ('/b/swarming/w/ir/out/Release/content_shell.pak.info', '/sdcard/chromium_tests_root/content_shell.pak.info'), ('/b/swarming/w/ir/out/Release/AHEM____.TTF', '/sdcard/chromium_tests_root/AHEM____.TTF'), ('/b/swarming/w/ir/out/Release/test_fonts/android_fallback_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_fallback_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/android_main_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_main_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/LICENSE', '/sdcard/chromium_tests_root/test_fonts/LICENSE'), ('/b/swarming/w/ir/out/Release/test_fonts/Ahem.ttf', '/sdcard/chromium_tests_root/test_fonts/Ahem.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Garuda.ttf', '/sdcard/chromium_tests_root/test_fonts/Garuda.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Devanagari.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Devanagari.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Gurmukhi.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Gurmukhi.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Tamil.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Tamil.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/MuktiNarrow.ttf', '/sdcard/chromium_tests_root/test_fonts/MuktiNarrow.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansCJKjp-Regular.otf', '/sdcard/chromium_tests_root/test_fonts/NotoSansCJKjp-Regular.otf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansKhmer-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/NotoSansKhmer-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Regular.ttf'), ('/b/swarming/w/ir/net/data', '/sdcard/chromium_tests_root/net/data')], retries=2, timeout=300, delete_device_stale=True)
I  228.202s TimeoutThread-1-for-individual_device_set_up(0246e0d059a41794)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 0246e0d059a41794 push /b/swarming/w/it20PJD8/tmpjmP7w8 /data/local/tmp/temp_file-8bbcdbea66209.sh
I  228.219s TimeoutThread-1-for-individual_device_set_up(0246e0d059a41794)  Large shell command will be run from file: PATH="/data/local/tmp/bin:$PATH" unzip /data/local/tmp/temp_file-68cc65a5a7324.zip&&chmod -R 777 /sdcard/chromium_tests_root/chrome/test/data /sdcard/chromium_tests_root/base/test/data /sdcard/chromium_tests_root/chrome/third_party/mock4js /sdcard/chromium_tests_root/components/test/data /sdcard/chromium_tests_root/extensions/test/data /sdcard/chromium_tests_root/google_apis/test/data /sdcard/chromium_tests_root/net/tools/testserver /sdcard/chromium_tests_root/third_party/accessibility-audit/axs_testing.js  ...
I  228.219s TimeoutThread-1-for-individual_device_set_up(0246e0d059a41794)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 0246e0d059a41794 shell '( sh /data/local/tmp/temp_file-8bbcdbea66209.sh );echo %$?'
I  249.988s individual_device_set_up(0246e0d059a41794)  Still working on PushChangedFiles(0246e0d059a41794, [('/b/swarming/w/ir/chrome/test/data', '/sdcard/chromium_tests_root/chrome/test/data'), ('/b/swarming/w/ir/base/test/data', '/sdcard/chromium_tests_root/base/test/data'), ('/b/swarming/w/ir/chrome/third_party/mock4js', '/sdcard/chromium_tests_root/chrome/third_party/mock4js'), ('/b/swarming/w/ir/components/test/data', '/sdcard/chromium_tests_root/components/test/data'), ('/b/swarming/w/ir/extensions/test/data', '/sdcard/chromium_tests_root/extensions/test/data'), ('/b/swarming/w/ir/google_apis/test/data', '/sdcard/chromium_tests_root/google_apis/test/data'), ('/b/swarming/w/ir/net/tools/testserver', '/sdcard/chromium_tests_root/net/tools/testserver'), ('/b/swarming/w/ir/third_party/accessibility-audit/axs_testing.js', '/sdcard/chromium_tests_root/third_party/accessibility-audit/axs_testing.js'), ('/b/swarming/w/ir/third_party/chaijs/chai.js', '/sdcard/chromium_tests_root/third_party/chaijs/chai.js'), ('/b/swarming/w/ir/third_party/pyftpdlib', '/sdcard/chromium_tests_root/third_party/pyftpdlib'), ('/b/swarming/w/ir/third_party/pywebsocket', '/sdcard/chromium_tests_root/third_party/pywebsocket'), ('/b/swarming/w/ir/third_party/tlslite', '/sdcard/chromium_tests_root/third_party/tlslite'), ('/b/swarming/w/ir/third_party/zlib/google/test/data', '/sdcard/chromium_tests_root/third_party/zlib/google/test/data'), ('/b/swarming/w/ir/tools/metrics/histograms/enums.xml', '/sdcard/chromium_tests_root/tools/metrics/histograms/enums.xml'), ('/b/swarming/w/ir/out/Release/pyproto/google', '/sdcard/chromium_tests_root/pyproto/google'), ('/b/swarming/w/ir/out/Release/chrome_100_percent.pak', '/sdcard/chromium_tests_root/paks/chrome_100_percent.pak'), ('/b/swarming/w/ir/out/Release/locales/en-US.pak', '/sdcard/chromium_tests_root/paks/en-US.pak'), ('/b/swarming/w/ir/out/Release/resources.pak', '/sdcard/chromium_tests_root/paks/resources.pak'), ('/b/swarming/w/ir/testing/buildbot/filters/unit_tests_cros_asan.filter', '/sdcard/chromium_tests_root/testing/buildbot/filters/unit_tests_cros_asan.filter'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/test.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/test.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/bad_format.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/bad_format.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/empty.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/empty.pb'), ('/b/swarming/w/ir/out/Release/icudtl.dat', '/sdcard/chromium_tests_root/icudtl.dat'), ('/b/swarming/w/ir/out/Release/content_shell.pak', '/sdcard/chromium_tests_root/paks/content_shell.pak'), ('/b/swarming/w/ir/out/Release/content_shell.pak.info', '/sdcard/chromium_tests_root/content_shell.pak.info'), ('/b/swarming/w/ir/out/Release/AHEM____.TTF', '/sdcard/chromium_tests_root/AHEM____.TTF'), ('/b/swarming/w/ir/out/Release/test_fonts/android_fallback_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_fallback_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/android_main_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_main_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/LICENSE', '/sdcard/chromium_tests_root/test_fonts/LICENSE'), ('/b/swarming/w/ir/out/Release/test_fonts/Ahem.ttf', '/sdcard/chromium_tests_root/test_fonts/Ahem.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Garuda.ttf', '/sdcard/chromium_tests_root/test_fonts/Garuda.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Devanagari.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Devanagari.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Gurmukhi.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Gurmukhi.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Tamil.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Tamil.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/MuktiNarrow.ttf', '/sdcard/chromium_tests_root/test_fonts/MuktiNarrow.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansCJKjp-Regular.otf', '/sdcard/chromium_tests_root/test_fonts/NotoSansCJKjp-Regular.otf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansKhmer-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/NotoSansKhmer-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Regular.ttf'), ('/b/swarming/w/ir/net/data', '/sdcard/chromium_tests_root/net/data')], retries=2, timeout=300, delete_device_stale=True)
I  310.030s individual_device_set_up(0246e0d059a41794)  Still working on PushChangedFiles(0246e0d059a41794, [('/b/swarming/w/ir/chrome/test/data', '/sdcard/chromium_tests_root/chrome/test/data'), ('/b/swarming/w/ir/base/test/data', '/sdcard/chromium_tests_root/base/test/data'), ('/b/swarming/w/ir/chrome/third_party/mock4js', '/sdcard/chromium_tests_root/chrome/third_party/mock4js'), ('/b/swarming/w/ir/components/test/data', '/sdcard/chromium_tests_root/components/test/data'), ('/b/swarming/w/ir/extensions/test/data', '/sdcard/chromium_tests_root/extensions/test/data'), ('/b/swarming/w/ir/google_apis/test/data', '/sdcard/chromium_tests_root/google_apis/test/data'), ('/b/swarming/w/ir/net/tools/testserver', '/sdcard/chromium_tests_root/net/tools/testserver'), ('/b/swarming/w/ir/third_party/accessibility-audit/axs_testing.js', '/sdcard/chromium_tests_root/third_party/accessibility-audit/axs_testing.js'), ('/b/swarming/w/ir/third_party/chaijs/chai.js', '/sdcard/chromium_tests_root/third_party/chaijs/chai.js'), ('/b/swarming/w/ir/third_party/pyftpdlib', '/sdcard/chromium_tests_root/third_party/pyftpdlib'), ('/b/swarming/w/ir/third_party/pywebsocket', '/sdcard/chromium_tests_root/third_party/pywebsocket'), ('/b/swarming/w/ir/third_party/tlslite', '/sdcard/chromium_tests_root/third_party/tlslite'), ('/b/swarming/w/ir/third_party/zlib/google/test/data', '/sdcard/chromium_tests_root/third_party/zlib/google/test/data'), ('/b/swarming/w/ir/tools/metrics/histograms/enums.xml', '/sdcard/chromium_tests_root/tools/metrics/histograms/enums.xml'), ('/b/swarming/w/ir/out/Release/pyproto/google', '/sdcard/chromium_tests_root/pyproto/google'), ('/b/swarming/w/ir/out/Release/chrome_100_percent.pak', '/sdcard/chromium_tests_root/paks/chrome_100_percent.pak'), ('/b/swarming/w/ir/out/Release/locales/en-US.pak', '/sdcard/chromium_tests_root/paks/en-US.pak'), ('/b/swarming/w/ir/out/Release/resources.pak', '/sdcard/chromium_tests_root/paks/resources.pak'), ('/b/swarming/w/ir/testing/buildbot/filters/unit_tests_cros_asan.filter', '/sdcard/chromium_tests_root/testing/buildbot/filters/unit_tests_cros_asan.filter'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/test.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/test.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/bad_format.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/bad_format.pb'), ('/b/swarming/w/ir/out/Release/gen/chrome/test/data/media/engagement/preload/empty.pb', '/sdcard/chromium_tests_root/gen/chrome/test/data/media/engagement/preload/empty.pb'), ('/b/swarming/w/ir/out/Release/icudtl.dat', '/sdcard/chromium_tests_root/icudtl.dat'), ('/b/swarming/w/ir/out/Release/content_shell.pak', '/sdcard/chromium_tests_root/paks/content_shell.pak'), ('/b/swarming/w/ir/out/Release/content_shell.pak.info', '/sdcard/chromium_tests_root/content_shell.pak.info'), ('/b/swarming/w/ir/out/Release/AHEM____.TTF', '/sdcard/chromium_tests_root/AHEM____.TTF'), ('/b/swarming/w/ir/out/Release/test_fonts/android_fallback_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_fallback_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/android_main_fonts.xml', '/sdcard/chromium_tests_root/test_fonts/android_main_fonts.xml'), ('/b/swarming/w/ir/out/Release/test_fonts/LICENSE', '/sdcard/chromium_tests_root/test_fonts/LICENSE'), ('/b/swarming/w/ir/out/Release/test_fonts/Ahem.ttf', '/sdcard/chromium_tests_root/test_fonts/Ahem.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Arimo-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Arimo-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Cousine-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Cousine-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/DejaVuSans.ttf', '/sdcard/chromium_tests_root/test_fonts/DejaVuSans.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Garuda.ttf', '/sdcard/chromium_tests_root/test_fonts/Garuda.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Gelasio-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Gelasio-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Devanagari.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Devanagari.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Gurmukhi.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Gurmukhi.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Lohit-Tamil.ttf', '/sdcard/chromium_tests_root/test_fonts/Lohit-Tamil.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/MuktiNarrow.ttf', '/sdcard/chromium_tests_root/test_fonts/MuktiNarrow.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansCJKjp-Regular.otf', '/sdcard/chromium_tests_root/test_fonts/NotoSansCJKjp-Regular.otf'), ('/b/swarming/w/ir/out/Release/test_fonts/NotoSansKhmer-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/NotoSansKhmer-Regular.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Bold.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Bold.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-BoldItalic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-BoldItalic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Italic.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Italic.ttf'), ('/b/swarming/w/ir/out/Release/test_fonts/Tinos-Regular.ttf', '/sdcard/chromium_tests_root/test_fonts/Tinos-Regular.ttf'), ('/b/swarming/w/ir/net/data', '/sdcard/chromium_tests_root/net/data')], retries=2, timeout=300, delete_device_stale=True)
C  369.875s individual_device_set_up(0246e0d059a41794)  Timed out. Dumping threads.
C  369.875s individual_device_set_up(0246e0d059a41794)  ********************************************************************************
C  369.875s individual_device_set_up(0246e0d059a41794)  Stack dump for thread 'TimeoutThread-1-for-individual_device_set_up(0246e0d059a41794)'
C  369.875s individual_device_set_up(0246e0d059a41794)  ********************************************************************************
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/.swarming_module/lib/python2.7/threading.py", line 774, in __bootstrap
C  369.876s individual_device_set_up(0246e0d059a41794)    self.__bootstrap_inner()
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/.swarming_module/lib/python2.7/threading.py", line 801, in __bootstrap_inner
C  369.876s individual_device_set_up(0246e0d059a41794)    self.run()
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
C  369.876s individual_device_set_up(0246e0d059a41794)    self._ret = self._func(*self._args, **self._kwargs)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
C  369.876s individual_device_set_up(0246e0d059a41794)    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
C  369.876s individual_device_set_up(0246e0d059a41794)    return f(*args, **kwargs)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1478, in PushChangedFiles
C  369.876s individual_device_set_up(0246e0d059a41794)    self._PushFilesImpl(host_device_tuples, all_changed_files)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1623, in _PushFilesImpl
C  369.876s individual_device_set_up(0246e0d059a41794)    files, [d for _, d in host_device_tuples]):
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1697, in _PushChangedFilesZipped
C  369.876s individual_device_set_up(0246e0d059a41794)    check_return=True)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
C  369.876s individual_device_set_up(0246e0d059a41794)    return impl()
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
C  369.876s individual_device_set_up(0246e0d059a41794)    return f(*args, **kwargs)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1112, in RunShellCommand
C  369.876s individual_device_set_up(0246e0d059a41794)    output = handle_large_output(cmd, large_output)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1081, in handle_large_output
C  369.876s individual_device_set_up(0246e0d059a41794)    return handle_large_command(cmd)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1069, in handle_large_command
C  369.876s individual_device_set_up(0246e0d059a41794)    return handle_check_return('sh %s' % script.name_quoted)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1054, in handle_check_return
C  369.876s individual_device_set_up(0246e0d059a41794)    return run(cmd)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1050, in run
C  369.876s individual_device_set_up(0246e0d059a41794)    return self.adb.Shell(cmd)
C  369.876s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 522, in Shell
C  369.876s individual_device_set_up(0246e0d059a41794)    output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False)
C  369.877s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 305, in _RunDeviceAdbCmd
C  369.877s individual_device_set_up(0246e0d059a41794)    check_error=check_error)
C  369.877s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
C  369.877s individual_device_set_up(0246e0d059a41794)    return impl()
C  369.877s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
C  369.877s individual_device_set_up(0246e0d059a41794)    return f(*args, **kwargs)
C  369.877s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 266, in _RunAdbCmd
C  369.877s individual_device_set_up(0246e0d059a41794)    env=cls._ADB_ENV)
C  369.877s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/cmd_helper.py", line 413, in GetCmdStatusAndOutputWithTimeout
C  369.877s individual_device_set_up(0246e0d059a41794)    for data in _IterProcessStdout(process, timeout=timeout):
C  369.877s individual_device_set_up(0246e0d059a41794)  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/cmd_helper.py", line 275, in _IterProcessStdoutFcntl
C  369.877s individual_device_set_up(0246e0d059a41794)    [child_fd], [], [], iter_aware_poll_interval)
C  369.877s individual_device_set_up(0246e0d059a41794)  ********************************************************************************

Also
https://ci.chromium.org/buildbot/tryserver.chromium.android/android_n5x_swarming_rel/422670
https://ci.chromium.org/buildbot/tryserver.chromium.android/android_n5x_swarming_rel/422642

2 out of 200 builds.

Issue 874840 has been merged into this issue.

 Issue 884626  has been merged into this issue.

Issue 893571 has been merged into this issue.

Issue 893477 has been merged into this issue.

10 out of 25 (57 out of 200) last builds on Marshmallow 64 bit Tester failed due to this error.

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Marshmallow%2064%20bit%20Tester

Issue 893244 has been merged into this issue.

Issue 895602 has been merged into this issue.

Still happening:
https://chromium-swarm.appspot.com/task?id=40c54a781059c910&refresh=10&show_raw=1

Looked some over the weekend. The unzip portion of the zip->push->unzip push path takes a while for unit_tests -- every run I checked, pass or fail, had a first attempt at pushing time out. I'm going to check what can be done to improve unzip performance in https://chromium.googlesource.com/chromium/src/+/HEAD/build/android/pylib/device/commands/java/src/org/chromium/android/commands/unzip/Unzip.java, but in the meantime, I'm raising the timeout for data dependency pushing for gtests.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6dcc239b7224b481cb12319fcfcbc46648c75212

commit 6dcc239b7224b481cb12319fcfcbc46648c75212
Author: John Budorick <jbudorick@chromium.org>
Date: Mon Oct 29 16:25:00 2018

android: raise gtest data dependency push timeout.

Bug: 791632
Change-Id: I2feff44cf4f875e064bc701fc7f74bf84811e3c8
Reviewed-on: https://chromium-review.googlesource.com/c/1304058
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#603517}
[modify] https://crrev.com/6dcc239b7224b481cb12319fcfcbc46648c75212/build/android/pylib/local/device/local_device_gtest_run.py

Unzipping is still slow but no longer seems to be flakily timing out. Updating title & priority accordingly.

It might make sense to instead do something like:

tar zc files | adb exec-in tar xzf -

So that we get streaming. I just tried it on an O device though and couldn't get "adb exec-in" to work at all :(

I would also be concerned about the consistency of tar across versions. Still probably worth investigating, though.

That said, I don't think the data transfer is the slow part -- IIRC, the core issue is the sheer number of files involved.