Look at this line in catapult: https://cs.chromium.org/chromium/src/third_party/catapult/tracing/tracing/metrics/compare_samples_cmdline.html?rcl=0&l=55

The issue here is that we expect three parts for composing interaction name record. Ethan might be able to explain the code if you have any questions.

Is there any reason compare_samples only reads list_of_scalar_values and histograms? In crbug.com/668165, Octane generates scalar values which don't seem to be getting read.

https://codereview.chromium.org/2528083002/ adds support for scalars.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c60f3321df06eb2945441f7953e1747efadb943a

commit c60f3321df06eb2945441f7953e1747efadb943a
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Fri Nov 25 00:25:59 2016

Roll src/third_party/catapult/ 3276375d2..671a65473 (7 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/3276375d2563..671a654736c2

$ git log 3276375d2..671a65473 --date=short --no-merges --format='%ad %ae %s'
2016-11-24 simonhatch Bisect - Fix compare_samples to read scalar values.
2016-11-24 hjd Fix getBoundingVisibleRect with offscreen elements
2016-11-24 ulan Helper functions for computing expected queueing time.
2016-11-23 achuith Autotest extension fix.
2016-11-23 benjhayden Document customizeSummaryOptions in how-to-write-metrics.
2016-11-23 fmeawad runtime_stats metric: only display the average in the dashboard
2016-11-23 simonhatch [bisect] - Clearly state bisect type in output.

BUG= 668536 , 664515 ,625701, 654525 , 667813 

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2530973002
Cr-Commit-Position: refs/heads/master@{#434413}

[modify] https://crrev.com/c60f3321df06eb2945441f7953e1747efadb943a/DEPS

So the octane bisect works now, but there are still bisects failing with missing data.

https://bugs.chromium.org/p/chromium/issues/detail?id=667313#c8
Metric: memory:chrome:renderer_processes:reported_by_chrome:code_and_metadata_size_avg/browse_social
JSON has the data:

"browse_social@@memory:chrome:renderer_processes:reported_by_chrome:code_and_metadata_size_avg": {
      "browse:social:facebook": {
        "description": "total size of code and metadata reported by Chrome for renderer processes in Chrome", 
        "grouping_keys": {
          "case": "browse", 
          "group": "social"
        }, 
        "important": false, 
        "improvement_direction": "down", 
        "name": "memory:chrome:renderer_processes:reported_by_chrome:code_and_metadata_size_avg", 
        "page_id": 1, 
        "std": 0.0, 
        "tir_label": "browse_social", 
        "type": "list_of_scalar_values", 
        "units": "sizeInBytes", 
        "values": [
          1885922.7848101268
        ]
      }, 
      "summary": {
        "description": "total size of code and metadata reported by Chrome for renderer processes in Chrome", 
        "grouping_keys": {
          "case": "browse", 
          "group": "social"
        }, 
        "important": false, 
        "improvement_direction": "down", 
        "name": "memory:chrome:renderer_processes:reported_by_chrome:code_and_metadata_size_avg", 
        "std": 0.0, 
        "tir_label": "browse_social", 
        "type": "list_of_scalar_values", 
        "units": "sizeInBytes", 
        "values": [
          1885922.7848101268
        ]
      }
    }, 

Looks like the actual metric is memory:chrome:renderer_processes:reported_by_chrome:code_and_metadata_size_avg/browse_social/summary

But as Roberto pointed out, the code in compare_samples expects 3 parts to compose the proper name. Using bin/compare_samples a.json b.json memory:chrome:.../browse_social/summary --chartjson works fine.

Looking at dashboard/add_point.py:362 (https://github.com/catapult-project/catapult/blob/25e660b/dashboard/dashboard/add_point.py#L362) I can see that "summary" gets special cased out. We probably need to add it back either in the dashboard when we generate the metric name to pass to the bisect script, or we can just try looking for it in compare_samples if the first lookup fails. Maybe Ethan has some thoughts on this?


Threw together a quick CL trying that out and it seems to work: https://codereview.chromium.org/2527263004/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2d25f0981c4bd17eb7a3ee32320774d2774a8588

commit 2d25f0981c4bd17eb7a3ee32320774d2774a8588
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Wed Nov 30 17:22:30 2016

Roll src/third_party/catapult/ b38578d92..59c9c1d73 (2 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/b38578d92404..59c9c1d73830

$ git log b38578d92..59c9c1d73 --date=short --no-merges --format='%ad %ae %s'
2016-11-30 simonhatch [bisect] - Have compare_samples check for missing "summary" tracename.
2016-11-30 charliea Change benchmark owners link in perf dashboard to go/chrome-benchmarks

BUG= 668536 

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2540813004
Cr-Commit-Position: refs/heads/master@{#435308}

[modify] https://crrev.com/2d25f0981c4bd17eb7a3ee32320774d2774a8588/DEPS