The way this is done in Telemetry is the binary under test (BUT) is a controllable variable. It could be a TOT browser, or it could be a reference browser stored in cloud storage.

For C++ tests like media_perftests and angle_perftest, there is no detachment between BUT & the test driver, hence the same model doesn't work here. So to make it work, I think there is two options:

1) Detach the BUT & the test driver (the current telemetry way). I am not sure whether we can do some dynamical linking magic here. (primiano?)

Another option is create a lightweight test driver, something python script that does:
if options.ref_binary:
   # Download the ref binary from cloud storage
  subprocess.Popen('ref_media_perftests')
else:
  subprocess.Popen('out/media_perftests')

2) Redesign the way ref build work from ground up at the recipe level. A ref build run is a swarming run specified by a fixed revision. Then we can use the swarming service to kick off a run on that fixed revision (very similar to how pinpoint would trigger the job).

The nice thing about this option is updating the ref build is probably as simple as updating that fixed revision. 

Isn't this a problem we can solve with bisect? i.e. if the bisect cannot repro, ignore?
Honestly, at least as far tracing_perftest is concerned, ref builds is IMHO not worth the effort and is going to increase waterfall maintenance cost.

 issue 710017  is an example of lack of a reference build causing additional work for sheriffs. When there isn't a reference build, it takes a good bit of work to understand what happened (1. run bisect, 2. see that culprit wasn't found, 3. re-run bisect, 4. figure out that it was a device change maybe, 5. close bug WontFix). When there is a reference build, it only takes a glance.

It is correct that this will increase costs shortterm, but generally gtest perf tests are very quick, especially compared to Telemetry tests, so the increase should be small. media_perftests for example only takes ~17 seconds to run all of its cases. In general, we should be encouraging the use of gtest perf tests wherever possible because they tend to isolate regressions in an easily reproducible way and they are quicker to run. If you think about it in terms of the testing pyramid, Telemetry tests are E2E tests, and gtest perf tests are unit or integration tests. If we can provide better features like reference build runs for the gtests, then developers will write those instead of Telemetry tests where possible. So actually, increasing the features of the gtest perf tests with *reduce* waterfall maintenance longterm because we will have more maintainable test cases.

nednyguyen@'s solutions both seem reasonable and not too difficult.

+ Yuri for performance_browser_tests

Yes, I'd love to see this for performance_browser_tests as well. :)

BTW--I'm also a perf sheriff and I having the ref signals seems to eliminate a HUGE number of false-positives. When we don't have the ref signals, the bisects have to be run multiple times, with varying ranges, just to be sure there isn't a regression. It's very time-consuming to manage.

I hit this during my media perf sheriff rotation today --  bug 738085  is an example where hopefully the bisect will find a culprit. If it doesn't, more human time will be spent per the 1st paragraph of https://crbugcom/714812#c5.

Hit again :  bug 812708 .