Another example of failing builds on an unrelated CL:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/ios-simulator/167163
https://chromium-review.googlesource.com/c/chromium/src/+/1396234

In the last 200 builds on the CQ, there were 8 failures in ios_chrome_ui_egtests that look entirely unrelated to the CLs:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/ios-simulator?limit=200

This suggests that all CQ attempts have a roughly ~4% false reject rate caused by ios-simulator [this doesn't quite translate to the go/cq-slo-dash since that requires there to be a repeated build that succeeds -- in these cases, sometimes the build isn't repeated, and sometimes it still failures on repeat].

Here's another specific example:

build:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/ios-simulator/167053

Task:
https://chromium-swarm.appspot.com/task?id=423203f6ce8d8910&refresh=10&show_raw=1

According to the task, the test passes in ~10 minutes. But swarming reports it as a timeout after 30 minutes.

"""
Test Suite 'RequestDesktopMobileSiteTestCase' passed at 2019-01-04 06:18:39.734.
	 Executed 9 tests, with 0 failures (0 unexpected) in 121.535 (121.544) seconds
Test Suite 'ios_chrome_ui_egtests_module.xctest' passed at 2019-01-04 06:18:39.735.
	 Executed 43 tests, with 0 failures (0 unexpected) in 498.063 (501.871) seconds
Test Suite 'Selected tests' passed at 2019-01-04 06:18:39.736.
	 Executed 43 tests, with 0 failures (0 unexpected) in 498.063 (501.873) seconds


Test session results and logs:
	/Users/chrome-bot/Library/Developer/Xcode/DerivedData/temporary-dzflokpcsqwreedhtgvqjcfwiydv/Logs/Test/Test-Transient Testing-2019.01.04_06-09-54--0800.xcresult

2019-01-04 06:18:40.179 xcodebuild[31116:458405] [MT] IDETestOperationsObserverDebug: 525.904 elapsed -- Testing started completed.
2019-01-04 06:18:40.179 xcodebuild[31116:458405] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-01-04 06:18:40.179 xcodebuild[31116:458405] [MT] IDETestOperationsObserverDebug: 525.904 sec, +525.904 sec -- end
** TEST EXECUTE SUCCEEDED **
"""

I've split this into two tasks:

Investigating why the timeouts are occuring: Issue 919164
Adding a cheaper retry mechanism [that doesn't rely on CQ-layer build retries]: Issue 919165.