Since we have seen occassional timeouts on our Linux Viz FYI, I'm going to test there if an increased shard time helps. Then will look at getting more consistent Android coverage up, to gather if this is a flake or a new failure.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/74fa86982732f9a0c6b30e448098a87a40c8ef4f

commit 74fa86982732f9a0c6b30e448098a87a40c8ef4f
Author: jonross <jonross@chromium.org>
Date: Thu Aug 09 14:24:34 2018

Update Linux Viz telemetry timeouts

We occassionally see telemetry_unittests_viz and telemetry_perf_unittests_viz
timeout on the FYI Linux Viz bot.

The logs point to no specific test taking longer than on a normal run. Nor of
any test hanging. The bot also reports 22/22 done on the shard, and browser
teardown, but never compiles the final success tally.

We would like to speculatively increase the timeouts here to see if that
stabilizes the bot.

TEST=telemetry_unittests_viz and telemetry_perf_unittests_viz

Bug:871751
Change-Id: Ie0b8ef744180afe543d5ccf32a578b91a963a0bf
Reviewed-on: https://chromium-review.googlesource.com/1168546
Reviewed-by: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/74fa86982732f9a0c6b30e448098a87a40c8ef4f/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/74fa86982732f9a0c6b30e448098a87a40c8ef4f/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e06c6a7ba84f11a2c60e3b5b74c8507568639444

commit e06c6a7ba84f11a2c60e3b5b74c8507568639444
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 19:25:35 2018

Revert "Update Linux Viz telemetry timeouts"

This reverts commit 74fa86982732f9a0c6b30e448098a87a40c8ef4f.

Reason for revert: Due to Gerrit outage  http://crbug.com/872722 , we are reverting this CL. Please, re-land it after all clear is given. If you have questions, please ask on the bug. Sorry for the inconvenience.

Original change's description:
> Update Linux Viz telemetry timeouts
> 
> We occassionally see telemetry_unittests_viz and telemetry_perf_unittests_viz
> timeout on the FYI Linux Viz bot.
> 
> The logs point to no specific test taking longer than on a normal run. Nor of
> any test hanging. The bot also reports 22/22 done on the shard, and browser
> teardown, but never compiles the final success tally.
> 
> We would like to speculatively increase the timeouts here to see if that
> stabilizes the bot.
> 
> TEST=telemetry_unittests_viz and telemetry_perf_unittests_viz
> 
> Bug:871751
> Change-Id: Ie0b8ef744180afe543d5ccf32a578b91a963a0bf
> Reviewed-on: https://chromium-review.googlesource.com/1168546
> Reviewed-by: Nico Weber <thakis@chromium.org>

TBR=thakis@chromium.org,jonross@chromium.org

Change-Id: I408c7c39b9e9daf96aed0f4392a56e0b7e757e49
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  871751 
Reviewed-on: https://chromium-review.googlesource.com/1169793
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/e06c6a7ba84f11a2c60e3b5b74c8507568639444/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/e06c6a7ba84f11a2c60e3b5b74c8507568639444/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a86a8a6bcfa438fa3ac2eba6f02b3ad1f8e0756f

commit a86a8a6bcfa438fa3ac2eba6f02b3ad1f8e0756f
Author: jonross <jonross@chromium.org>
Date: Fri Aug 10 14:54:01 2018

Reland Update Linux Viz telemetry timeouts

Originally reviewed: https://chromium-review.googlesource.com/c/chromium/src/+/1168546
Reverted due to the great Gerrit tree closure of 2018:
https://chromium-review.googlesource.com/c/chromium/src/+/1169793

TBR=thakis@chromium.org

Update Linux Viz telemetry timeouts

We occassionally see telemetry_unittests_viz and telemetry_perf_unittests_viz
timeout on the FYI Linux Viz bot.

The logs point to no specific test taking longer than on a normal run. Nor of
any test hanging. The bot also reports 22/22 done on the shard, and browser
teardown, but never compiles the final success tally.

We would like to speculatively increase the timeouts here to see if that
stabilizes the bot.

TEST=telemetry_unittests_viz and telemetry_perf_unittests_viz

Bug:  871751 
Change-Id: I4a3da0e45e5006673a5a3967e9e33d602d38794e
Reviewed-on: https://chromium-review.googlesource.com/1168546
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1170092
Reviewed-by: Jonathan Ross <jonross@chromium.org>
Commit-Queue: Jonathan Ross <jonross@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582161}
[modify] https://crrev.com/a86a8a6bcfa438fa3ac2eba6f02b3ad1f8e0756f/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/a86a8a6bcfa438fa3ac2eba6f02b3ad1f8e0756f/testing/buildbot/test_suites.pyl

Removing OOP-D beta blocker - this does appear to be an understood test-only timeout.

If it's going to cause failures on bots then it's likely a beta blocker too. We have to turn on the field trial testing configuration before beta finch which switches all bots over.

So after updating the timeouts on our FYI bot, the rate of this failure format dropped to 7/200

However on several of the main CQs telemetry_perf_unittests have been marked experimental due to other issues.

With the failure occurring after the tests have completed, we'd need to begin debugging the shutdown of chrome, and the telemetry scrips. Steps we can consider:
   -  More logging (Both Chrome and Telem)
   -  Enable tracing on these builds, and add more instrumentation of shutdown.

Now that we are in field trials:

The test we were concerned about in #1 has had no failures recently.

Looking at Android coverage, only 2 failures of telemetry_perf_unittests in the last 200 runs. Both appear to be infra related to a lost device.

For Linux Viz, I've removed these test from the FYI bot, as we have CQ coverage directly for these tests now.

No more work here.