Cannot find the owner from go/perf-owners. cc'ing Ned and Ken.

reveman@ owns this

Oh, right. Just saw it. Not sure why previous search failed.

The error seems due to black listed device. But this failure has been there for a long time, not sure why not picked by infra team automatically. cc'ing martiniss and adding infra labs.


Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 70, in wrapper
    return f(dev, *args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 331, in individual_device_set_up
    step()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 322, in <lambda>
    return lambda: crash_handler.RetryOnSystemCrash(step, dev)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/crash_handler.py", line 32, in RetryOnSystemCrash
    return f(device)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 302, in push_test_data
    delete_device_stale=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 157, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 150, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1362, in PushChangedFiles
    check_return=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1041, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1010, in handle_large_output
    return handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 998, in handle_large_command
    return handle_check_return('sh %s' % script.name_quoted)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 983, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 979, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 521, in Shell
    command, output, status=status, device_serial=self._device_serial)
AdbShellCommandFailedError: (device: AG86044202E84GC) shell command run via adb failed on the device:
  command: sh /data/local/tmp/temp_file-6a39670db7a1b.sh
  exit status: 255
  output:
  - mkdir failed for /storage/sdcard0/chromium_tests_root/, Read-only file system

E   14.890s individual_device_set_up(AG86044202E84GC)  Device AG86044202E84GC blacklisted: individual_device_set_up
E   14.906s Main  Error occurred.
Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/test_runner.py", line 985, in main
    return RunTestsCommand(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 726, in RunTestsCommand
    return RunTestsInPlatformMode(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 838, in RunTestsInPlatformMode
    with json_writer, logcats_uploader, env, test_instance, test_run:
  File "/b/swarming/w/ir/build/android/pylib/base/test_run.py", line 39, in __enter__
    self.SetUp()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 335, in SetUp
    self._test_instance.GetDataDependencies())
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 263, in pMap
    r.pFinish(None)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 135, in pFinish
    self._objs.JoinAll()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 81, in wrapper
    on_failure(dev, f.__name__)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 300, in BlacklistDevice
    'All devices were blacklisted due to errors')
NoDevicesError: All devices were blacklisted due to errors
Command ['./gpu_perftests', '--adb-path', 'src/third_party/android_tools/sdk/platform-tools/adb', '--verbose', '--test-launcher-print-test-stdio=always'] returned exit code 87

I'm cc-ing jbudorick. I think something is strange about this test; other tests on the bot are running fine. The error is about something wrong with the file system. 

The device itself isn't caught by the infra ticket filer thing because it only fails this test. It's running other tests just fine.

 Issue 767610  has been merged into this issue.

I'm going to go ahead and disable gpu_perftests on A1 for the time being: it's been failing on the waterfall since at least September 20

The A1s, unlike most of our phones, have read-only emulated storage. For a while, we addressed this by remounting / RW. My memory on this is a bit fuzzy, but I believe we switched to putting SD cards in the phones, either when we upgraded to M or because they were more reliable than the remount mechanism.

In any even, this looks like it either doesn't have an SD card or isn't correctly using it.

When we added BattOrs to the A1 devices in the labs they didn't add SD cards to the new A1 devices if my memory is correct. I _think_ that got resolved, but its possible a device was missed.