Filing this to keep track on a more persistent medium.
In [1] I did some quick research about tuning various parameters (scaling governor, frequency, P states) in order to get less noisy benchmarks.
There are two potentially action items here:
 - 1. Having a perf_envsetup script in catapult shared by bots and developers (see nduca suggestion).
 - 2. Improving the kind of tweaks we do in the perf bots today.
 - 3. Figuring out the Windows equivalent of this black magic. I can only do Linux black magic.

TL;DR
On a Z620 cc_perftests's stddev went from 2% to 0.07% using some tweaks. Full data in [2]. The summary is:

- Affinity makes a big difference
- SCHED_FIFO a bit, but not that much
- performance governor seems to make things actually worse
- when you use powersave governor the scaling_max_freq seems to be ignored. Proof:
$ for cpu in /sys/devices/system/cpu/cpu*; do sudo sh -c "echo performance > $cpu/cpufreq/scaling_governor";
$ for cpu in /sys/devices/system/cpu/cpu*; do sudo sh -c "echo 1200000 > $cpu/cpufreq/scaling_min_freq"; done
$ for cpu in /sys/devices/system/cpu/cpu*; do echo -en "$cpu\t"; sudo cat $cpu/cpufreq/scaling_cur_freq; 
  /sys/devices/system/cpu/cpu0    3100015
  /sys/devices/system/cpu/cpu1    3100015
  /sys/devices/system/cpu/cpu10   3099906
  /sys/devices/system/cpu/cpu11   3100015

- The max_freq seems to be respected instead when using "powersave". Also powersave seems to respect the scaling_max_freq. Proof:
for cpu in /sys/devices/system/cpu/cpu*; do echo -en "$cpu\t"; sudo cat $cpu/cpufreq/scaling_cur_freq; done
  /sys/devices/system/cpu/cpu0    1199953
  /sys/devices/system/cpu/cpu1    1199953
  /sys/devices/system/cpu/cpu10   1199953
  /sys/devices/system/cpu/cpu11   1199953
  /sys/devices/system/cpu/cpu12   1199953

- Disabling all the P-state (but P0) makes the biggest difference (1 order of magnitude in stddev)
sudo sh -c "echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
for cpu in /sys/devices/system/cpu/cpu*; do for p in $(seq 4); do sudo sh -c "echo 1 > $cpu/cpuidle/state$p/disable"; done; do


nduca@ suggested in a follow up: 
"""From my very high level view of things, I think we should move toward a world where catapult has a subsystem in it ("environment setup"?) that sets up the local machine for the best possible perf run possible. It might be parameterized in a few ways to accommodate power, for instance, EnvironmentSetup.Init(is_power_test=False). Then we'd use that in all the places, telemetry of course, but other harnesses as well.
We know that local dev machines are varied and often problematic. I think its great for the environment setup code to detect bad-for-perf things and try to disable them. But, if they can't be disabled, it should at least try to stop the developer from running. I think true success here looks like us steering people to use perf trybots more and use local runs less. Eg better if environment setup say "we can't make your machine produce good results, you're running some stuff we can't disable. please use the perf bots. if you are sure, you can bypass this by saying --ignore-environment-setup-failure."""

I personally really like the idea of having one perf_envsetup script which is shared by humans and bots, so that everybody tests under the same conditions.

[1] https://groups.google.com/a/chromium.org/d/msg/project-trim/uhtD-UIvjLA/RtAd1k6KBQAJ
[2] https://docs.google.com/spreadsheets/d/1xAxScjSHWht-ftiag1ppiAedM2aLON_BJcafWEC9Xck/edit?usp=sharing