https://chromeperf.appspot.com/report?sid=09274a24876778d5dfc69c470959b4ba87c8cdd63ecf1c9c77a1d4483750694e&start_rev=461989&end_rev=462859 is a graph comparing swarmed and non swarmed data about this test.

We don't have benchmark duration data, because the bot assigned to the benchmark appears to have been taken offline. bpastene@, have you been looking at why build245-m4--device1 is offline? Should I file a bug for it?

Actually ccc-ing ben.

See t/26152838

An update; we're still waiting for the device to be back online :/ It's still broken, as of today, and there hasn't been an update on the bug in a while.

I'll ping the ticket.

If you need the data now, could you just break that device's affinity and distribute its tests to the other 20 N5Xs?

I'll make a CL to break device affinity for this bot.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1e2d5ea4965411024f7682ce7c441ab38239263f

commit 1e2d5ea4965411024f7682ce7c441ab38239263f
Author: martiniss <martiniss@chromium.org>
Date: Wed Apr 19 00:35:16 2017

//tools/perf: Remove bad device for android swarming test bot

Generally not recommended, but this bot is on FYI so we don't care.

BUG= 705135 

Review-Url: https://codereview.chromium.org/2827573006
Cr-Commit-Position: refs/heads/master@{#465437}

[modify] https://crrev.com/1e2d5ea4965411024f7682ce7c441ab38239263f/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/1e2d5ea4965411024f7682ce7c441ab38239263f/tools/perf/core/perf_data_generator.py

The device is back up and running again:
https://chromium-swarm.appspot.com/bot?id=build245-m4--device1

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d32fd6ec97a878ebd9a7065f246091d16a2e2394

commit d32fd6ec97a878ebd9a7065f246091d16a2e2394
Author: martiniss <martiniss@chromium.org>
Date: Wed Apr 19 19:44:45 2017

Revert "//tools/perf: Remove bad device for android swarming test bot"

This reverts commit 1e2d5ea4965411024f7682ce7c441ab38239263f.

BUG= 705135 

Reason: Bot came back online
Review-Url: https://codereview.chromium.org/2831693002
Cr-Commit-Position: refs/heads/master@{#465713}

[modify] https://crrev.com/d32fd6ec97a878ebd9a7065f246091d16a2e2394/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/d32fd6ec97a878ebd9a7065f246091d16a2e2394/tools/perf/core/perf_data_generator.py

The test is now running on swarming, but it's taking 2 hours and timing out. Is this expected for system_health.memory_mobile to take 2 hours!? Does it take that long right now?

Looks like the test takes about 140 minutes off of swarming. We need to bump up the timeout for the task :/

Same thing was happening to loading.mobile. I bumped its timeout a bit ago:
https://chromium.googlesource.com/chromium/src/+/431d50287a50b50bbc65b9074ce6044405879a95/tools/perf/core/perf_data_generator.py#662

+Juan: can you help Stephen with comparing the results here as well?

I spent the day looking at the logs and writing scripts to compare the timings of individual events ... being confused because the swarming bot seemed to be *faster* (stories and other events took less time).

I should have paid more attention to this graph:
https://chromeperf.appspot.com/report?sid=d81ddce902142437a828648e7e5aa8574a106199f2eb4b3b267c5f0d92ea6ab7

Yep, system_health.memory_mobile has been taking over 2 hours for a few days, and recently jumped to nearly 3 hours.

The swarmed bot is indeed somewhat faster, but is killed at the 2 hours mark.

Btw, do we track somewhere the duration of individual stories?

No, we haven't tracked the duration of individual stories yet. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1cda587abd16e09e3e25768c98e4c656d3f816be

commit 1cda587abd16e09e3e25768c98e4c656d3f816be
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Apr 21 19:30:17 2017

//tools/perf: Increase timeout for system_health.memory_mobile

This test currently takes a long time to run, so bump the timeout on
swarming.

Bug:  705135 
Change-Id: I325a18948f4cafb67c64f897dfeb4e2e6f2efa16
Reviewed-on: https://chromium-review.googlesource.com/483844
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#466413}
[modify] https://crrev.com/1cda587abd16e09e3e25768c98e4c656d3f816be/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/1cda587abd16e09e3e25768c98e4c656d3f816be/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/1cda587abd16e09e3e25768c98e4c656d3f816be/tools/perf/core/perf_data_generator.py

Given that memory.top_10_mobile gives reasonable memory data. I think we can close this as WontFix.

Timeout probably won't be that big of a problem as when we deploy this in production, we have a lot more devices.