We have clock frequency in Android traces, but I'm not sure if we do in Linux.

+Some folks who may have thoughts.

Is this something we should do for rendering benchmarks too? So that the cpu-time we get from the benchmark is more reliable?

There's always a tradeoff between being representative of real user experience, and being stable and easy to understand.

I don't have a good intuition for whether the tradeoff would make sense in this case.

Perhaps we should be counting CPU cycles instead of task durations? I know Android folks are getting a fair bit of mileage out of counting CPU cycles.

(Adding brucedawson@, power benchmarks owner)

tdresser@ that's super interesting - do you understand how they calculate this in practice? It does seem like the ideal solution.

It seems like you'd need to know:

1) The frequencies that all CPUs are running at at all times. This is easy to get with systrace on Android and ftrace on Linux, but I'm not sure about on Mac and Win

2) What processor each Chrome thread was running on at each point in time. I'm not sure if we currently collect this or how hard it would be to get. Maybe this is also available via systrace?

This would at least be enough to get us an approximation of CPU cycles.

There is trickiness, though. We only have a single "cpu time" number for a trace event. If the wall time for an event was 100 seconds, the CPU time is 80s, and we know that the processor was running at 0.2 GHz for the first 90s and 1.0 GHz for the last 10s: it's impossible to precisely know what the total clock cycles are. It turns out, though, that we already have this problem (when we have a range of interest that terminates in the middle of an event) and we already assume that the CPU time is evenly distributed (https://cs.chromium.org/chromium/src/third_party/catapult/tracing/tracing/model/thread.html?type=cs&q=getcputimeforrange&sq=package:chromium&g=0&l=258), which is probably a reasonable estimate in practice.

Frankly, I think the biggest thing holding us back from implementing this is a willingness to commit the time to get it done. I'd guess this would take O(weeks) to implement, and I'm just not sure how many problems we currently experience due to this.

I think that QueryProcessCycleTime is supposed to measure CPU cycles consumed by a process and it should account for the constantly change CPU speeds (although even then it would miss some things, because TurboBoost is managed by the CPU itself). I would want to do some testing of this function before trusting it.

If this function doesn't work then trying to recreate this would be crazy hard. All of the information should be available in ETW traces but a) recording ETW traces requires admin privileges and b) I am not convinced that the CPU frequency data in ETW traces can be trusted.

Disabling CPU scaling is also a reasonable solution. It is imperfect, to be sure, but it will give us a much more sensitive measurement of whether we are increasing or decreasing the CPU load.

This isn't just power.desktop. I was hit by similar "regressions" with media.desktop test suites as well. I think that this issue is the cause of much flake with CPU usage based tests.

*nudge*

Trying disabling CPU scaling SGTM.

Ned, who is the right owner for this now?

#10 that would fit into core automation team. But we don't have any bandwidth to take this now, so just keep the components "Speed > Benchmarks" for now for us to backlog this.