I've seen this on Nexus 5X as well, on the following benchmarks:
blink_perf.dom
blink_perf.layou
blink_perf.svg

Looks like it's random, from what I've seen.

Randy, do you know how we're supposed to be disabling this?

This has been going on for a while and make many tests flaky. Not sure what's going on here

Is there any point disabling media.mse_cases when this is a Telemetry bug not a bug in the media.mse_cases test cases? It seems like it would be better to keep this running and try our best to fix it.

Agree, disabling the story is of no use, it probably will only cause other stories to start failing with this. The failure happens when launching the browser, way before any story-specific code gets to execute.

And again the issue appears to be Chrome not picking up the correct flags: devtools is not enabled (we can't connect to it), and the first-run experience is not skipped as it should.

Note that now from Telemetry's side we both write the command line file and *read it back* before launching Chrome to ensure the contents are as expected.

The following bit from the logcat shows what was happening at around the time when Chrome got started:

	I/ActivityManager(  554): Resuming delayed broadcast
	I/ActivityManager(  554): Start proc 5248:org.chromium.chrome/u0a74 for broadcast org.chromium.chrome/com.google.android.apps.chrome.appwidget.bookmarks.BookmarkThumbnailWidgetProvider
	D/AndroidRuntime( 5289): >>>>>> START com.android.internal.os.RuntimeInit uid 0 <<<<<<
	D/AndroidRuntime( 5289): CheckJNI is OFF
	E/memtrack( 5289): Couldn't load memtrack module (No such file or directory)
	E/android.os.Debug( 5289): failed to load memtrack module: -2
	D/AndroidRuntime( 5289): Calling main entry com.android.commands.am.Am
	I/ActivityManager(  554): START u0 {dat=about:blank flg=0x10000000 cmp=org.chromium.chrome/com.google.android.apps.chrome.Main} from uid 0 on display 0
	V/WindowManager(  554): addAppToken: AppWindowToken{1ddb3486 token=Token{6006161 ActivityRecord{11b1cc8 u0 org.chromium.chrome/com.google.android.apps.chrome.Main t49911}}} to stack=1 task=49911 at 0
	I/cr_InstantAppsHandler( 5248): Not handling with Instant Apps because Chrome is not default or there's a specialized handler
	I/ActivityManager(  554): START u0 {cmp=org.chromium.chrome/.browser.firstrun.FirstRunActivity (has extras)} from uid 10074 on display 0
	V/WindowManager(  554): addAppToken: AppWindowToken{2300b29d token=Token{39293d74 ActivityRecord{25fa6147 u0 org.chromium.chrome/.browser.firstrun.FirstRunActivity t49912}}} to stack=1 task=49912 at 0
	V/WindowManager(  554): Based on layer: Adding window Window{10549855 u0 Starting org.chromium.chrome} at 2 of 7
	W/cr_ChildProcLH( 5248): Create a new ChildConnectionAllocator with package name = org.chromium.chrome, sandboxed = true
	I/cr_LibraryLoader( 5248): Using linker: org.chromium.base.library_loader.LegacyLinker
	W/linker  ( 5248): libchromium_android_linker.so: unused DT entry: type 0x6ffffffe arg 0xce8
	W/linker  ( 5248): libchromium_android_linker.so: unused DT entry: type 0x6fffffff arg 0x3
	I/cr_base ( 5248): Extracting resource /data/data/org.chromium.chrome/app_chrome/paks/en-GB.pak@15cd54872b3
	D/OpenGLRenderer( 5248): Use EGL_SWAP_BEHAVIOR_PRESERVED: false
	W/cr_LibraryLoader( 5248): Low-memory device: shared RELROs used in all processes
	I/cr_base ( 5248): Extracting resource /data/data/org.chromium.chrome/app_chrome/paks/en-US.pak@15cd54872b3
	D/Atlas   ( 5248): Validating map...
	I/cr_LibraryLoader( 5248): Loading chrome
	V/WindowManager(  554): Adding window Window{369564c2 u0 org.chromium.chrome/org.chromium.chrome.browser.firstrun.FirstRunActivity} at 2 of 8 (before Window{10549855 u0 Starting org.chromium.chrome})
	W/art     ( 5248): Suspending all threads took: 7.723ms
	W/cr_VariationsSeedFetch( 5248): UnknownHostException fetching first run seed: 
	I/art     ( 5248): Background partial concurrent mark sweep GC freed 5337(249KB) AllocSpace objects, 1(13KB) LOS objects, 24% free, 5MB/7MB, paused 12.599ms total 103.974ms
	I/ActivityManager(  554): Start proc 5318:org.chromium.chrome:sandboxed_process0/u0i1 for service org.chromium.chrome/org.chromium.content.app.SandboxedProcessService0

+primiano, can you help us parse this? Is there some extra debugging we could add (either Chrome or Telemetry side) to figure out why command line flags aren't being honored?

Ah I think this might be the culprit:
----
I/ActivityManager(  554): Start proc 5248:org.chromium.chrome/u0a74 for broadcast org.chromium.chrome/com.google.android.apps.chrome.appwidget.bookmarks.BookmarkThumbnailWidgetProvider
-----

Looks like chrome is started before telemetry sends the intent by the system because chrome declares itself to be a bookmark thumbnail widget provider.
So I think the following sequence of actions is happening:

1. telemetry checks that there are no chrome running (eventually killing existing instances)
2. the system (very likely the homescreen app) decides to fetch the bookmarks thumbnails. By doing so, it "accidentally" starts chrome, because chrome in its manifest declares itself a bookmark thumbnail provider.
3. telemetry sets the command line
4. telemetry starts chrome via intent

At this point, this bug happens, because we read the commandline in 2 (during the accidental start), but the cmdline is not ready at that point.

I discussed this offline with perezju@ and we should be able to solve this by inverting the relative order of  [check if there are any chrome instances running and kill them if there is any] and [set the cmdline file].
If we swap the order of these two, the accidental startup should pick the right commandline

Yep, I can make this change.

Thanks for detective work of you both! Fixing this should help a lot with our Android bots stability!

Though the fact that Android can start Chrome whenever it likes during Telemetry benchmark seems a bit worrisome to me.

Does starting chrome through intent also kill existing Chrome that was "accidentally" started as bookmark thumbnail provider?

As I understand starting Chrome through an intent does not kill an existing instance.

And that's exactly what is happening here:

Story 1:
- Telemetry writes command line file
- Telemetry launches Chrome through intent
- Telemetry removes command line file
- Telemetry runs story
- Telemetry stops Chrome

Story 2:
- Android triggers thumbnail intent, this launches Chrome 
  -> But command line was not ready! So Chrome uses defaults.
- Telemetry writes command line file
- Telemetry launches Chrome through intent
  -> Android sees Chrome is already up, so sends intent to
     existing process.
- Telemetry can't run story :-(

I also agree it's a bit worrisome that Android can send random intents to Chrome while tests are running. This could surely cause flakes in all sorts of measurements.

Charlie: see #5 & #9, this surely can add lots of noise in Power tests. 

I think we may need a deeper fix to make sure that Telemetry is the only actor that send intents during the tests run here. I filed another bug to track this: issue 736755

What the heck's a bookmark thumbnail provider?

So the main problem is that chrome has a bunch of content provider implementations (Re #11: essentially the OS exposes some generic provider APIs that any app can override *, and chrome is one of them. This is for things like bookmarks). I think specifically BookmarkThumbnailWidgetProvider is the implementation of the content provider for the Android bookmarks widget.

The problem is that these content provider implementations are hosted in the same process that hosts chrome, and they also share some code with chrome.
So whenever they are triggered, they initialize base stuff like the command line (if not the rest of content, really depends on the specific API).

I am not sure how to prevent the framework to request updates of bookmarks and widgets. I checked the screenshot from a failing bot and there doesn't seem to be any widget in the homescreen. Despite that we seem to still receive some calls from the framework.

CC-ing some clank folks to see if they have any idea to increase predictability for benchmarks and avoid these. 


* https://developer.android.com/guide/topics/providers/content-providers.html 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4b80f573a1e1f1e174f817a156bde3d1e55df9d1

commit 4b80f573a1e1f1e174f817a156bde3d1e55df9d1
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Mon Jun 26 16:59:53 2017

Roll src/third_party/catapult/ b4eb70c31..6af9db22b (1 commit)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/b4eb70c319cb..6af9db22b57c

$ git log b4eb70c31..6af9db22b --date=short --no-merges --format='%ad %ae %s'
2017-06-26 perezju [Telemetry] Stop existing browser after command line flags are set

Created with:
  roll-dep src/third_party/catapult
BUG= 736516 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: Ie1f90814a1aa98c83de0123f01ce34090fa4c551
Reviewed-on: https://chromium-review.googlesource.com/548436
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#482298}
[modify] https://crrev.com/4b80f573a1e1f1e174f817a156bde3d1e55df9d1/DEPS

 Issue 736849  has been merged into this issue.

None of the named tests in #0 and #1 have failed in a while. Closing this a fixed. Do re-open or file a new bug if you see the issue again.