I just want to point out we are already running system_health_loading tests with power metrics.

search for battor.system_health on the following bots:
Windows:
https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Win%20Power%20High-DPI%20Perf/builds/59

Android:
https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Power%20Nexus%205X%20Perf%20%281%29/builds/211

Mac:
https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Mac%20Power%20Dual-GPU%20Perf/builds/1186
https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Mac%20Power%20Low-End%20Perf/builds/1182

(marking this as "blocking issue 589726" to simplify keeping track of the SH effort)

Just for background, I think the thing we really want here is to get stories with real web pages + significant idle time to catch the problems we've been seeing about work during idle, right Charlie?

Adding the 10 second waiting period in https://codereview.chromium.org/2248693002 is causing out of memory errors on the mac BattOr bots. For example, this is the first build it broke on the mac FYI bots https://build.chromium.org/p/chromium.perf.fyi/builders/Mac%20Power%20Low-End%20Perf/builds/1342.

I think the battor benchmarks should be updated to use Alex's rail tracing categories to have leaner footprint.

Randy: can you update the tracing categories in the benchmarks? Alex can help reviewing.



+Nat, Fadi: this is another case of d8 OOM after JSON.parse(...):

[ RUN      ] /tmp/tmpbQLMm0.html

#
# Fatal error in MarkCompactCollector: semi-space copy, fallback in old gen

# Allocation failed - process out of memory
#

Unknown stack
[  FAILED  ] /tmp/tmpbQLMm0.html (51044 ms)
Traceback (most recent call last):
  File "/b/c/b/Mac_Power_Low_End_Perf/src/third_party/catapult/telemetry/telemetry/value/failure.py", line 41, in _GetExcInfoFromMessage
    raise Exception(message)
Exception: Unknown stack


Trace: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_26-2016-08-19_10-04-14-64807.html

The switch to rail only traces does not seem to fix the OOM problem.
https://codereview.chromium.org/2265083002/

Did you grab a trace with rail categories only? How many trace events are we able to reduce using "rail" category vs everything?

I was just running them, they look almost identical. I'm not suer if there is something wrong with how I'm enabling rail only tracing.


https://x20.corp.google.com/users/rn/rnephew/rail_test

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2cbfbe277ed9b712ac9b28763cd1e247bc5e89f8

commit 2cbfbe277ed9b712ac9b28763cd1e247bc5e89f8
Author: rnephew <rnephew@chromium.org>
Date: Tue Aug 23 22:46:56 2016

[Telemetry] Change default BattOr benchmark chrome tracing categories to be rail.

BUG= 637158 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq

Review-Url: https://codereview.chromium.org/2265083002
Cr-Commit-Position: refs/heads/master@{#413874}

[modify] https://crrev.com/2cbfbe277ed9b712ac9b28763cd1e247bc5e89f8/tools/perf/benchmarks/battor.py

Ok, should be working now that only rail catagory is set.

Any objections for getting a CL out running the browsing:*:* SH stories for battor? Currently we only have the loading:*:* stories running.

That sounds like a great idea. I think it's possible htey'll be pretty noisy and we won't actually want to alert on them, but it seems like it's worthwhile to turn them on to see what kind of noise we see.

I had a CL out for it, but the consensus was your work on https://codereview.chromium.org/2239053003/ would enable power numbers on those same tests so it was unnecessary. 

Doh, sorry. That CL was really supposed to land before I left :-/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/71253911d21dfc93fdeccb6c46e518c2a078a3b9

commit 71253911d21dfc93fdeccb6c46e518c2a078a3b9
Author: charliea <charliea@chromium.org>
Date: Wed Aug 31 04:40:17 2016

[system health] Add power system health benchmarks

Even though the metrics that we have aren't ideal quite yet (total
energy consumed, average power draw), we still want system health
benchmarks running ASAP in order to prevent major regressions.

We discussed having two runs of the system health benchmarks:
memory and non-memory, with memory being isolated due to its high
overhead. At least for the time being, that's not possible with power
benchmarks because we need to decide in the benchmark constructor
whether to enable BattOr tracing, even though we don't have any ability
to know whether a BattOr is attached. Because of that, we're forced to
disable the test on any devices without BattOrs in the @ShouldDisable
predicate.

BUG= 637158 ,589726
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq

Review-Url: https://codereview.chromium.org/2239053003
Cr-Commit-Position: refs/heads/master@{#415561}

[modify] https://crrev.com/71253911d21dfc93fdeccb6c46e518c2a078a3b9/tools/perf/benchmarks/system_health.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7c742ced7137db2e6785822d2a2f8299178b3537

commit 7c742ced7137db2e6785822d2a2f8299178b3537
Author: charliea <charliea@chromium.org>
Date: Tue Oct 11 20:41:26 2016

[system health] Compute power and clock sync latency metrics

I added the "common system health benchmark" (i.e. non-memory
benchmark), but forgot to have it compute any metrics. This fixes that.

I also removed the old system health loading stories from the "battor"
benchmark set. Eventually, I imagine that this set will go away
completely as BattOr metrics are computed in other benchmarks.

BUG= 637158 
NOTRY=true

Review-Url: https://codereview.chromium.org/2338343002
Cr-Commit-Position: refs/heads/master@{#424536}

[modify] https://crrev.com/7c742ced7137db2e6785822d2a2f8299178b3537/tools/perf/benchmarks/battor.py
[modify] https://crrev.com/7c742ced7137db2e6785822d2a2f8299178b3537/tools/perf/benchmarks/system_health.py