> Egor, could that be the culprit?

In my understanding the metrics code is executed after the importing, so it cannot be the metrics.

Started here: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus5%20Perf/2703

This is the likely reason, which is in the blame range:

commit 973cf57b1f0da5d6833b680b0b8efe8c6337eed6
Author:     Egor Pasko <pasko@chromium.org>
AuthorDate: Wed Oct 10 15:25:24 2018 +0000
Commit:     Commit Bot <commit-bot@chromium.org>
CommitDate: Wed Oct 10 15:25:24 2018 +0000

    startup.mobile benchmark: enable atrace temporarily
    
    I am investigating occasional startup delays in 2nd runs (see the bug).
    The extra systrace information may help.
    
    Currently traces merged like this cannot compute messageloop_start_time. This
    problem is addressed in the catapult-side change:
    https://chromium-review.googlesource.com/c/catapult/+/1270856
    
    Bug: 891797
    Change-Id: I70434fdd627101b29a720c3deca988e3c2318049
    Reviewed-on: https://chromium-review.googlesource.com/c/1264543

It is strange that this works on the downstream N5 bot and fails on the upstream one. Are the upstream devices provisioned differently?

I'd like to disable the benchmark on the upstream bots. Juan, does that SG as a workaround while investigating crbug.com/891797 ? After that bug is fixed I am planning to disable the systrace.

+Ned just in case, sorry for spam :)

Disabling the benchmark upstream sgtm.

> It is strange that this works on the downstream N5 bot and fails on the upstream one. Are the upstream devices provisioned differently?

They have different Android versions, upstream are K, downstream are on M. Could that be the reason?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5c7e8891018642b37e6aa00b4e4d2c6a2eec1e1b

commit 5c7e8891018642b37e6aa00b4e4d2c6a2eec1e1b
Author: Egor Pasko <pasko@chromium.org>
Date: Tue Oct 16 08:01:17 2018

benchmarks/startup.mobile: disable on upstream bots

Currently we are investigating some noisy characteristics of the
benchmark (http://crbug.com/891797), for that we enabled systrace.

Unfortunately, on upstream bots running Android K the trace merging
fails (reasons are unknown). After the investigation is done the
systrace will be turned off anyways because it noticeably affects the
metrics (in non-representative ways).

Disabling the benchmark upstream. This. Is. Temporary. Yes.

Bug:  894744 
Change-Id: I9f607e4da1e10727cd2b31b30d0ad15d53ab37c8
Reviewed-on: https://chromium-review.googlesource.com/c/1278214
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Cr-Commit-Position: refs/heads/master@{#599905}
[modify] https://crrev.com/5c7e8891018642b37e6aa00b4e4d2c6a2eec1e1b/tools/perf/expectations.config

 Issue 896850  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/277b01f27c92075420bd8fa5af807e5b1d7c2492

commit 277b01f27c92075420bd8fa5af807e5b1d7c2492
Author: Annie Sullivan <sullivan@chromium.org>
Date: Thu Oct 18 21:05:34 2018

Disable failing loading.mobile/TribunNews_3g and system_health.common_mobile/background:tools:gmail on N5, and fix startup.mobile disables.

Bug:  894744 , 896851, 896088
Change-Id: I08abf22dc487dd54fe0d3de9170f3f6cac74888c
TBR: nednguyen@google.com
NOTRY: true
Reviewed-on: https://chromium-review.googlesource.com/c/1289573
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Annie Sullivan <sullivan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#600904}
[modify] https://crrev.com/277b01f27c92075420bd8fa5af807e5b1d7c2492/tools/perf/expectations.config

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/44e9ffbb8818275a8bc49f45cc2f98ee9a04e6ed

commit 44e9ffbb8818275a8bc49f45cc2f98ee9a04e6ed
Author: Egor Pasko <pasko@chromium.org>
Date: Thu Oct 25 16:05:02 2018

startup.mobile benchmark: Expand inline HOWTO

Covering a couple of non-obvious things:
* necessity to --also-run-disabled-tests
* the fact that --browser=android-chrome matches monochrome_apk as a
  target, even for M, where Play Store does not install Monochrome

Bug:  894744 
Change-Id: I442ea775c337bb5ba62547c6ef33e4dd39f2dae6
Reviewed-on: https://chromium-review.googlesource.com/c/1299138
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Egor Pasko <pasko@chromium.org>
Cr-Commit-Position: refs/heads/master@{#602728}
[modify] https://crrev.com/44e9ffbb8818275a8bc49f45cc2f98ee9a04e6ed/tools/perf/benchmarks/startup_mobile.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/clank/internal/apps/+/e9ee155a9b1f98c8ee5cbeb34d091996754d91e6

commit e9ee155a9b1f98c8ee5cbeb34d091996754d91e6
Author: Juan Antonio Navarro Perez <perezju@chromium.org>
Date: Fri Oct 26 15:07:28 2018

The benchmark is disabled on upstream bots (including the "Android Nexus5 Perf").

I think then we should be able to close this now.