After discussing two overlapping efforts going on in the Chrome Speed Operations team (histogram set and one buildbot step) we have concluded that we should at least pause and think about the right chrome as a whole path forward in regards to the relationship between infra side scripts (runtest.py, upload_perf_dashboard_results.py, etc), recipe modules (chromium_tests steps.py specifically), and src side scripts (I can't remember if we have a current one that is invoked from our chromium_tests module?  Ethan, maybe your new catapult binary to generate histogram sets?)



For our current use case in chrome speed operations (one buildbot and histogram sets), we need to refactor the perf dashboard upload slightly to support our next overlapping milestones current state of the world:

Potentially 1 to n result files, each containing 1 to n benchmarks.  Each one of these files could contain legacy json (output currently by c++ perf tests), chart json, histogram set data, or a combination of the above. 

We need to collect all this data for one run of one perf waterfall configuration (that could come from n machines in the swarming pool) safely (oauth tokens could be necessary for histogram sets) to both the perf dashboard and the flakiness dashboard in 1 to n rpcs where n is the number of benchmarks (still figuring out what apis are currently available for each dashboard).  


Ok that is our use case, lets tie it back to how to design this right in recipe land.  We currently outsource to two infra side scripts: upload_perf_dashboard_results.py and results_dashboard.py (why are there two?) to upload data to the perf dashboard (not sure where we actually upload to the flakiness dashboard).  

We also need to be able to talk to catapult somewhere with specific builder information (where a builder here is information about the swarming bot it ran on ) to create the histogram sets.  We may or may not have to do this on every run, but we are moving to a model (2018 goal?) where we only support histogram set results.

That being said, my first gut feeling is that we should stop using infra side scripts (they can't be tested easily) and only use src side scripts.  But do we even need a src side script at all?  Can we do all this uploading in a perf specific recipe module that the chromium recipe calls into?   We stumbled upon this when migrating perf to swarming months ago but I don't think we ever really figured out who used it or exactly what it did or if we could re-purpose it. 

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/perf_dashboard/



Ok this is a lot of info for a bug, but I wanted to get the ball rolling.  Dirk I am initially assigning to you for initial thoughts, feel free to ignore and I can bring it up in our weekly sync to make sure we are on track for one buildbot starting August 30th.  

Anyone else, please chime in with additional thoughts or things I missed.

Ashley, I included you in case you have any thoughts around the flakiness dashboard since I have little context there.  Please feel free to remove and add the right person for flakiness dashboard insights.