Also, a question - is it possible to run these tests without root privelege?

Can someone please answer the questions in #0 and #4?

The root failure looks similar to one we've seen elsewhere. Running `adb root` causes us to briefly lose contact with the device. Our best guess as to what's happening here is that the wait-for-device call completes *before* we lose the device, then when we do lose it, the commands we try to run fail.

Thanks for confirming this may be a telemetry problem.

Also, Looks like recently the flakiness shifted to "device_status" step, though what really failing is "provision_devices".

https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28NVIDIA%20Shield%20TV%29/builds/664
https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28NVIDIA%20Shield%20TV%29/builds/646
https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28NVIDIA%20Shield%20TV%29/builds/532

python -u /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/build/android/provision_devices.py --adb-path /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb --blacklist-file /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/out/bad_devices.json --output-device-blacklist /tmp/tmp0T1u2l.json -t Release
in dir /b/c/b/Android_Release__NVIDIA_Shield_TV_:
 allow_subannotations: False
 base_name: provision_devices
 cmd: ['python', '-u', '/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/build/android/provision_devices.py', '--adb-path', '/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb', '--blacklist-file', '/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/out/bad_devices.json', '--output-device-blacklist', '/tmp/tmp0T1u2l.json', '-t', 'Release']
 cwd: /b/c/b/Android_Release__NVIDIA_Shield_TV_
 env: {'GOMA_SERVICE_ACCOUNT_JSON_FILE': '/creds/service_accounts/service-account-goma-client.json', 'PATH': '/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/android_tools/sdk/platform-tools:/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/build/android:%(PATH)s'}
 infra_step: True
 name: provision_devices
 nest_level: 0
 ok_ret: frozenset([0])
 trigger_specs: []
full environment:
 AWS_CREDENTIAL_FILE: /b/build/site_config/.boto
 BOTO_CONFIG: /b/build/site_config/.boto
 BUILDBOT_BLAMELIST: [u'nigeltao@chromium.org']
 BUILDBOT_BRANCH: master
 BUILDBOT_BUILDBOTURL: https://build.chromium.org/p/chromium.gpu.fyi/
 BUILDBOT_BUILDERNAME: Android Release (NVIDIA Shield TV)
 BUILDBOT_BUILDNUMBER: 664
 BUILDBOT_CLOBBER: 
 BUILDBOT_GOT_REVISION: None
 BUILDBOT_MASTERNAME: chromium.gpu.fyi
 BUILDBOT_REVISION: beff0a104dbf869114f66db3efc2d52094050971
 BUILDBOT_SCHEDULER: chromium_commits
 BUILDBOT_SLAVENAME: build162-b1
 CHROME_HEADLESS: 1
 CIPD_CACHE_DIR: /b/c/cipd
 DISPLAY: :0.0
 GIT_USER_AGENT: git/2.11.0 linux2 build162-b1.labs.chromium.org
 GOMA_SERVICE_ACCOUNT_JSON_FILE: /creds/service_accounts/service-account-goma-client.json
 HOME: /home/chrome-bot
 INFRA_BUILDBOT_MASTER_CLASS_NAME: ChromiumGPUFYI
 INFRA_BUILDBOT_SLAVE_ACTIVE_SUBDIR: 
 INFRA_BUILDBOT_SLAVE_NAME: build162-b1
 LANG: en_US.UTF-8
 LOGDOG_COORDINATOR_HOST: luci-logdog.appspot.com
 LOGDOG_STREAM_PREFIX: bb/chromium.gpu.fyi/Android_Release__NVIDIA_Shield_TV_/664
 LOGDOG_STREAM_PROJECT: chromium
 LOGDOG_STREAM_SERVER_PATH: unix:/b/build/rr/tmp_8dBkF/butler.sock
 LOGNAME: chrome-bot
 PAGER: cat
 PATH: /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/android_tools/sdk/platform-tools:/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/build/android:/b/cipd_path_tools:/b/cipd_client:/home/chrome-bot/slavebin:/b/depot_tools:/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
 PWD: /b/build/slave/Android_Release__NVIDIA_Shield_TV_/build
 PYTHONIOENCODING: UTF-8
 PYTHONPATH: /b/build/site_config:/b/build/scripts:/b/build/scripts/release:/b/build/third_party:/b/build/third_party/google_api_python_client:/b/build/third_party/httplib2/python2:/b/build/third_party/infra_libs:/b/build/third_party/oauth2client:/b/build/third_party/pyasn1:/b/build/third_party/pyasn1-modules:/b/build/third_party/python-rsa:/b/build/third_party/requests_2_10_0:/b/build/third_party/setuptools-0.6c11:/b/build/third_party/site-packages:/b/build/third_party/uritemplate:/b/build_internal/site_config:/b/build_internal/symsrc:/b/build/slave:/b/build/third_party/buildbot_slave_8_4:/b/build/third_party/twisted_10_2:
 PYTHONUNBUFFERED: 1
 SHELL: /bin/bash
 USER: chrome-bot

I    0.002s TimeoutThread-1-for-MainThread  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb devices
I    0.007s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( ( c=/data/local/tmp/cache_token;echo $EXTERNAL_STORAGE;cat $c 2>/dev/null||echo;echo "58a8ff92-2cc5-11e7-a8cf-3417ebf03c0b">$c &&getprop )>/data/local/tmp/temp_file-cff09998d4884 );echo %$?'
I    0.066s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 pull /data/local/tmp/temp_file-cff09998d4884 /tmp/tmpXlmxYt/tmp_ReadFileWithPull
I    0.225s TimeoutThread-1-for-delete_temporary_file(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell 'rm -f /data/local/tmp/temp_file-cff09998d4884'
I    0.237s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 wait-for-device
I    0.240s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( test -d /sdcard );echo %$?'
I    0.289s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'sd_card_ready' met (0.1s)
I    0.289s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( pm path android );echo %$?'
I    0.725s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'pm_ready' met (0.5s)
I    0.725s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( getprop sys.boot_completed );echo %$?'
I    0.779s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'boot_completed' met (0.5s)
I    0.802s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( pm list packages );echo %$?'
I    1.234s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( pm list packages -s );echo %$?'
I    1.666s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( pm path org.chromium.chrome );echo %$?'
I    2.159s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 uninstall org.chromium.chrome
I    2.799s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( pm path org.chromium.native_test );echo %$?'
I    3.270s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 uninstall org.chromium.native_test
W    3.832s TimeoutThread-1-for-ProvisionDevice(0324216001899)  The command to run should preferably be passed as a sequence of args. If shell features are needed (pipes, wildcards, variables) clients should explicitly set shell=True.
I    3.832s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( rm -rf /sdcard/* );echo %$?'
W    3.932s TimeoutThread-1-for-ProvisionDevice(0324216001899)  The command to run should preferably be passed as a sequence of args. If shell features are needed (pipes, wildcards, variables) clients should explicitly set shell=True.
I    3.932s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( rm -rf /data/local/tmp/* );echo %$?'
I    3.996s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 reboot
I    4.300s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb devices
I    4.301s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'device_offline' met (0.3s)
I    4.304s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 wait-for-device
I   26.317s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 wait-for-device
I   26.320s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( ( c=/data/local/tmp/cache_token;echo $EXTERNAL_STORAGE;cat $c 2>/dev/null||echo;echo "68582f44-2cc5-11e7-89d2-3417ebf03c0b">$c &&getprop )>/data/local/tmp/temp_file-27336f10b2e9 );echo %$?'
I   26.438s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 pull /data/local/tmp/temp_file-27336f10b2e9 /tmp/tmp2DOW3c/tmp_ReadFileWithPull
I   26.601s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( test -d /sdcard );echo %$?'
I   26.601s TimeoutThread-1-for-delete_temporary_file(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell 'rm -f /data/local/tmp/temp_file-27336f10b2e9'
I   26.652s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'sd_card_ready' not met (0.3s)
I   31.657s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( test -d /sdcard );echo %$?'
I   31.709s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'sd_card_ready' not met (5.4s)
I   36.714s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( test -d /sdcard );echo %$?'
I   36.765s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'sd_card_ready' not met (10.4s)
I   41.771s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( test -d /sdcard );echo %$?'
I   41.830s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'sd_card_ready' not met (15.5s)
I   46.835s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( test -d /sdcard );echo %$?'
I   46.895s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'sd_card_ready' met (20.6s)
I   46.895s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( pm path android );echo %$?'
I   47.703s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'pm_ready' met (21.4s)
I   47.703s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 shell '( getprop sys.boot_completed );echo %$?'
I   47.763s TimeoutThread-1-for-ProvisionDevice(0324216001899)  condition 'boot_completed' met (21.4s)
I   47.794s TimeoutThread-1-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 root
C   47.858s ProvisionDevice(0324216001899)  (TimeoutThread-1-for-ProvisionDevice(0324216001899)) Exception on EnableRoot(0324216001899, retries=3, timeout=30), attempt 1 of 4: CommandFailedError('(device: 0324216001899) Unable to root device with user build.',)
I   47.858s TimeoutThread-2-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 root
C   47.861s ProvisionDevice(0324216001899)  (TimeoutThread-2-for-ProvisionDevice(0324216001899)) Exception on EnableRoot(0324216001899, retries=3, timeout=30), attempt 2 of 4: DeviceUnreachableError('0324216001899',)
I   47.861s TimeoutThread-3-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 root
C   47.865s ProvisionDevice(0324216001899)  (TimeoutThread-3-for-ProvisionDevice(0324216001899)) Exception on EnableRoot(0324216001899, retries=3, timeout=30), attempt 3 of 4: DeviceUnreachableError('0324216001899',)
I   47.865s TimeoutThread-4-for-ProvisionDevice(0324216001899)  [host]> /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0324216001899 root
E   47.869s ProvisionDevice(0324216001899)  Failed to provision device 0324216001899. Adding to blacklist.
Traceback (most recent call last):
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/build/android/provision_devices.py", line 122, in ProvisionDevice
    run_phase(SetProperties, reboot_timeout)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/build/android/provision_devices.py", line 101, in run_phase
    phase_func(device, options)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/build/android/provision_devices.py", line 264, in SetProperties
    device.EnableRoot()
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 159, in Run
    error_log_func=error_log_func)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 152, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/device_utils.py", line 451, in EnableRoot
    self.adb.Root()
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 875, in Root
    output = self._RunDeviceAdbCmd(['root'], timeout, retries)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 286, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/c/b/Android_Release__NVIDIA_Shield_TV_/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 263, in _RunAdbCmd
    raise device_errors.DeviceUnreachableError(device_serial)
DeviceUnreachableError: 0324216001899
I   47.869s ProvisionDevice(0324216001899)  Adding 0324216001899 to blacklist /b/c/b/Android_Release__NVIDIA_Shield_TV_/src/out/bad_devices.json for reason: provision_failure

Fix in 713883 should have fixed the rooting issues we were seeing on this bot. Please reopen on recurrence.