Expanding this to include battor.steady_state, which is also owned by Erik and also has a BattOr-based equivalent.

> - It's only enabled on Mac, where we're really striving for all benchmarks to cover multiple platforms.
The Mac is a unique platform - it will have platform-specific benchmarks.

> - In the last 200 runs, it's failed on ToT 31 times and the ref build 28 times.
That's not good.

> - It hasn't caught any regressions within the last 30 days.
Are you saying it has missed regressions, or that there just hasn't been any regression activity?

I found this +3% CPU regression the other day:

https://chromeperf.appspot.com/report?sid=fd8f68c380343b951366767a9db25da63f5ec481293556a728d1320c589ed9d0

I think the corresponding power regression is in the noise - not sure if that means power it's not relevant, or if the power units are hiding it.

> - We already have a BattOr version of this benchmark running (battor.trivial_pages)
> I've checked with erikchen@, the owner, and he said that he's okay with deleting it.

??? As erikchen@ knows, I just added a page to power.trivial_pages. It's not in battor.trivial_pages. It also seems like battor.trivial_pages is a completely different set of pages, so the test cases in power... will be lost?

> ??? As erikchen@ knows, I just added a page to power.trivial_pages. It's not in battor.trivial_pages. It also seems like battor.trivial_pages is a completely different set of pages, so the test cases in power... will be lost?

The new page you added was to TrivialSitesStorySet:
https://codereview.chromium.org/2766763004/diff/80001/tools/perf/page_sets/trivial_sites.py

This is the same story set used by battor.trivial_pages:
https://cs.chromium.org/chromium/src/tools/perf/benchmarks/battor.py?type=cs&q=TrivialSitesStorySet+package:%5Echromium$&l=45

battor.trivial_pages is similar to power.trivial_pages, but it uses battor instead of MSRs, and it has TBMv2 metrics, which power.trivial_pages doesn't have.

But thank you for bringing this up, it reminds me that there is one metric that battor doesn't have: idle_wakeups. I filed a bug about this in crbug.com/674963, but it was never resolved.

charlie: Can we block this on getting issue 674963 resolved, and then can you make sure that issue is appropriately prioritized and gets an owner?

On these perf tests, how large a deviation from the norm is considered a regression? 10%?

(CCing shrike@, who should have been CCed on this bug from the start)

The logic to determine what's considered a regression is a little bit complicated:

- The regression has to be at least 1%
- The regression has to have a "steppiness" factor of at least 0.5, where 0.0 is a perfectly flat line or random noise and 1.0 is a perfect step in the data
- The regression has to be greater than 2.5 standard deviations of the time series
- The series off of which these statistics are computed on each side of the regression has to consist of no fewer than 6 data points

All of this is configurable, though.

Not sure why I marked this as Fixed when it's not yet done, but reopening this: it's slated to be done this Wednesday. If it's urgent enough, we can implement a TBMv2 idle wakeups metric, but the maintenance costs imposed through bugs like  issue 763263  can't really be justified given the low number of regressions that these benchmarks alone find.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/03db80e03327e91832af3fb0bb40d3a057c8582f

commit 03db80e03327e91832af3fb0bb40d3a057c8582f
Author: Charlie Andrews <charliea@chromium.org>
Date: Wed Sep 13 16:10:23 2017

Delete power.steady_state and power.trivial_pages benchmarks

Bug:  705042 
Change-Id: Iec0fe8718b3a0510bb73d2e20efa4c732b539940
Reviewed-on: https://chromium-review.googlesource.com/665098
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#501646}
[modify] https://crrev.com/03db80e03327e91832af3fb0bb40d3a057c8582f/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/03db80e03327e91832af3fb0bb40d3a057c8582f/tools/perf/benchmark.csv
[modify] https://crrev.com/03db80e03327e91832af3fb0bb40d3a057c8582f/tools/perf/benchmarks/power.py
[modify] https://crrev.com/03db80e03327e91832af3fb0bb40d3a057c8582f/tools/perf/unowned_benchmarks.txt