jbudorick@ I think this is something going wrong on the infra end and twellington@ may have come across it before.

I'm seeing this on the trybots and locally when running render tests that are parameterized (e.g.  @ParameterAnnotations.UseMethodParameter(ModernParams.class))

Interesting... Only with method parameters, or also with class parameters?

From the logs, it appears as though the test runner isn't seeing a /failures directory at all:

I  802.429s run_tests_on_device(05e3eb3f3444fe4f)  preparing to run org.chromium.chrome.browser.ntp.snippets.ArticleSnippetsTest#testSnippetAppearance__EnableNTPModernLayout: {'annotations': {u'ParameterAnnotations$UseRunnerDelegate': {u'value': u'class org.chromium.chrome.test.ChromeJUnit4RunnerDelegate'}, u'MediumTest': {}, u'RunWith': {u'value': u'class org.chromium.base.test.params.ParameterizedRunner'}, u'Test': {u'expected': u'class org.junit.Test$None', u'timeout': u'0'}, u'ChromeModernDesign$Disable': {}, u'Feature': {u'value': [u'ArticleSnippets', u'RenderTest']}, u'CommandLineFlags$Add': {u'value': [u'disable-fre']}}, 'class': u'org.chromium.chrome.browser.ntp.snippets.ArticleSnippetsTest', 'is_junit4': True, 'method': u'testSnippetAppearance__EnableNTPModernLayout'}
I  802.429s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( echo -n '"'"'_ --strict-mode=testing --render-test-output-dir="/storage/emulated/legacy/render_test_output_dir" --enable-test-intents'"'"' > /data/local/tmp/test-cmdline-file );echo %$?'
I  802.445s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( test -e /data/local/tmp/test-cmdline-file );echo %$?'
I  802.476s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( TZ=utc ls -a -l /data/local/tmp/ );echo %$?'
I  802.509s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( cat /data/local/tmp/test-cmdline-file );echo %$?'
I  802.540s run_tests_on_device(05e3eb3f3444fe4f)  Flags now set on the device: ['--strict-mode=testing', '--render-test-output-dir=/storage/emulated/legacy/render_test_output_dir', '--enable-test-intents']
I  802.540s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( mkdir -p /storage/emulated/legacy/tmp-c7507ce993bf9 );echo %$?'
I  802.556s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f logcat -c
I  802.572s record_to_file  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f logcat -v threadtime chromium:v 'cr_*:v' DEBUG:I StrictMode:D
I  802.572s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( log -p i -t test_runner_py '"'"'START org.chromium.chrome.browser.ntp.snippets.ArticleSnippetsTest#testSnippetAppearance__EnableNTPModernLayout'"'"' );echo %$?'
I  802.604s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( ( p=org.chromium.chrome.tests;am instrument -w -r -e org.chromium.base.test.util.Screenshooter.ScreenshotDir /storage/emulated/legacy/tmp-c7507ce993bf9 -e class '"'"'org.chromium.chrome.browser.ntp.snippets.ArticleSnippetsTest#testSnippetAppearance__EnableNTPModernLayout'"'"' -e org.chromium.base.test.ScreenshotOnFailureStatement.ScreenshotFile /storage/emulated/legacy/temp_file-5ccf08b02a4f3.png "$p"/org.chromium.base.test.BaseChromiumAndroidJUnitRunner )>/data/local/tmp/temp_file-4bf65eef1bb65 2>&1 );echo %$?'
I  814.328s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f pull /data/local/tmp/temp_file-4bf65eef1bb65 /b/swarming/w/itqigLgM/tmpBXW0E6/tmp_ReadFileWithPull
I  814.334s TimeoutThread-1-for-delete_temporary_file(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell 'rm -f /data/local/tmp/temp_file-4bf65eef1bb65'
I  814.344s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( log -p i -t test_runner_py '"'"'END org.chromium.chrome.browser.ntp.snippets.ArticleSnippetsTest#testSnippetAppearance__EnableNTPModernLayout'"'"' );echo %$?'
I  814.423s run_tests_on_device(05e3eb3f3444fe4f)  Logcat saved to https://luci-logdog.appspot.com/v/?s=chromium%2Fandroid%2Fswarming%2Flogcats%2F3cf536c4946e7511%2F%2B%2Flogcat_logcat_org.chromium.chrome.browser.ntp.snippets.ArticleSnippetsTest.testSnippetAppearance__EnableNTPModernLayout_20180419T054739-UTC_05e3eb3f3444fe4f
I  814.423s archive  Writing text to logdog stream, logcat_logcat_org.chromium.chrome.browser.ntp.snippets.ArticleSnippetsTest.testSnippetAppearance__EnableNTPModernLayout_20180419T054739-UTC_05e3eb3f3444fe4f
I  814.431s run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/build/android/pylib/symbols/../../../../third_party/android_platform/development/scripts/stack --arch arm --output-directory /b/swarming/w/ir/out/Release --more-info /b/swarming/w/itqigLgM/tmpdVrz16
I  814.596s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( echo -n '"'"'_ --strict-mode=testing --enable-test-intents'"'"' > /data/local/tmp/test-cmdline-file );echo %$?'
I  814.627s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( test -e /data/local/tmp/test-cmdline-file );echo %$?'
I  814.660s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( TZ=utc ls -a -l /data/local/tmp/ );echo %$?'
I  814.691s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( cat /data/local/tmp/test-cmdline-file );echo %$?'
I  814.723s run_tests_on_device(05e3eb3f3444fe4f)  Flags now set on the device: ['--strict-mode=testing', '--enable-test-intents']
I  814.723s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( rm -f /data/local/tmp/chrome_timeout_scale );echo %$?'
I  814.755s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( test -e /storage/emulated/legacy/render_test_output_dir/failures );echo %$?'
I  814.771s TimeoutThread-1-for-run_tests_on_device(05e3eb3f3444fe4f)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05e3eb3f3444fe4f shell '( rm -f -r /storage/emulated/legacy/render_test_output_dir );echo %$?'

Will try to repro locally for more investigation. I think the interesting part to start with is what's happening w/ the bitmap saving logic in https://codesearch.chromium.org/chromium/src/chrome/test/android/javatests/src/org/chromium/chrome/test/util/RenderTestRule.java?rcl=f83d193ebe48911e5c178d3e0c55e619c68fac27&l=173

Re #3 -- with both method and class parameterization. ArticleSnippetsTest uses class parameterization and NewTabPageTest uses method parameterization and both of their render tests no longer output files on the bots.

Ah, the command line isn't initialized yet when the RenderTestRule is constructed, because the ParameterizedRunner constructs the test class earlier.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec953303fb739c2c8ce4953ea48d79e7ca95ae02

commit ec953303fb739c2c8ce4953ea48d79e7ca95ae02
Author: Bernhard Bauer <bauerb@chromium.org>
Date: Tue Apr 24 09:51:11 2018

Instantiate test class for parameterized tests at the same time as for non-parameterized tests.

ParameterizedRunnerDelegateFactory used to create the test class when creating
the runner, passing the instance in so that createTest() would just return it.
This meant that all test rules would be created at that time as well, which in
the case of RenderTestRule is before the command line is instantiated, so it
would not have the right value of the render test output directory.
(In addition, it defaults to using the test data directory, so writing failure
files would suceed, but the test runner wouldn't find them when pulling from
the device.)

With this change, the test class is created in the createTest() method to avoid
this and other subtle differences between parameterized and non-parameterized
tests.

Bug:  834712 
Change-Id: I49bec0e6dbbf5aebe92f29a0de65c9b60729646a
Reviewed-on: https://chromium-review.googlesource.com/1023854
Reviewed-by: Yaron Friedman <yfriedman@chromium.org>
Commit-Queue: Bernhard Bauer <bauerb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553043}
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/BUILD.gn
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/javatests/src/org/chromium/base/test/params/BaseJUnit4RunnerDelegate.java
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/javatests/src/org/chromium/base/test/params/BlockJUnit4RunnerDelegate.java
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/javatests/src/org/chromium/base/test/params/ParameterizedRunner.java
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/javatests/src/org/chromium/base/test/params/ParameterizedRunnerDelegate.java
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/javatests/src/org/chromium/base/test/params/ParameterizedRunnerDelegateCommon.java
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/javatests/src/org/chromium/base/test/params/ParameterizedRunnerDelegateFactory.java
[add] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/junit/src/org/chromium/base/test/params/ParameterizedRunnerDelegateCommonTest.java
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/base/test/android/junit/src/org/chromium/base/test/params/ParameterizedRunnerDelegateFactoryTest.java
[modify] https://crrev.com/ec953303fb739c2c8ce4953ea48d79e7ca95ae02/chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeJUnit4RunnerDelegate.java

It looks like this is working now. Thank you!

(Sample run that produced output: https://storage.cloud.google.com/chromium-result-details/html/chrome_public_test_apk_linux_android_rel_ng_535373_2018_04_24_T14_54_35-UTC)

Yay!