I think there are several problems:

1. Failures on android_blink_rel to run the tests. In cases where tests don't finish running and the results aren't uploaded, then there are no test results. I assume that this includes most of the purple builds listed for this builder where "webkit_tests (with patch)" failed. https://build.chromium.org/p/tryserver.chromium.android/builders/android_blink_rel?numbuilds=100

There is currently  bug 681208  about how frequent failures on android_blink_rel cause trouble for rebaseline-cl, which requires fetching results on all platforms before proceeding.

There are a couple types of improvements that could be made here:
 - Fixing issues that cause failures in android try jobs.
 - For rebaseline-cl in particular: Dealing with failures by filling in results from other platforms ( bug 673966 ). This is something I want to do anyway.

2. In the case of build 2062, as you pointed out, there *were* results
(https://storage.googleapis.com/chromium-layout-test-archives/android_blink_rel/2062/layout-test-results/results.html) but rebaseline-cl says it failed to get them. This seems like a bug in rebaseline-cl.

3. Not retrying? As far as I can tell, android_blink_rel is retrying steps that failed, e.g.
https://build.chromium.org/p/tryserver.chromium.android/builders/android_blink_rel/builds/2028

The problem seemed to magically go away overnight, so I have no idea what went wrong. But I agree with the analysis and mitigation steps.

The android bots don't run a huge number of tests, it seems, so it would be better not to ever need that for rebaselining. I understand that getting the information about which tests are affected from one place to another might be hard.

I've knocked back the priority since I am unblocked.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5c28e1999704819f1c19950bca846c982d7d8c76

commit 5c28e1999704819f1c19950bca846c982d7d8c76
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Mar 16 22:45:41 2017

Add more logging when fetching layout test results.

This changes logging in rebaseline_cl.py_and adds logging in buildbot.py.
Reason: Recently rebaseline-cl said it couldn't fetch results when it
appeared that results were actually archived.

If this happens again, a better log output would be helpful.

BUG= 701963 

Review-Url: https://codereview.chromium.org/2760463002
Cr-Commit-Position: refs/heads/master@{#457589}

[modify] https://crrev.com/5c28e1999704819f1c19950bca846c982d7d8c76/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot.py
[modify] https://crrev.com/5c28e1999704819f1c19950bca846c982d7d8c76/third_party/WebKit/Tools/Scripts/webkitpy/common/net/buildbot_unittest.py
[modify] https://crrev.com/5c28e1999704819f1c19950bca846c982d7d8c76/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py
[modify] https://crrev.com/5c28e1999704819f1c19950bca846c982d7d8c76/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py

I think that the main thing we want to do is allow rebaseline-cl to keep going when android_blink_rel fails ( bug 681208 ); meanwhile, after adding logging it should be easier to debug next time the script says that it couldn't fetch results when results were present.