(kraynov, for context: Maria and I synced on this and I asked her to file this on us)

I think the real challenge here is really get the bot up and running (tracked on Issue 670826). The test itself should be quite trivial.

Ah just realized that this bug is also about: --enable-heap-profiling. I think Maria just forgot to mention it.
We already have the test described in #0 (without --enable-heap-profiling) in place.

Ah, apologies for leaving that off. Would it be sufficient to run the original test with this flag added or do you think we need additional actions in the new test, Primiano?

One thing to figure out as part of this bug (related with #3) is why we didn't catch things like  Issue 675600 , which repros without --enable-heap-profiling and just with the trace config.
Regardles of --enable-heap-profiling it feels we are still missing some coverage

Actually implemented https://build.chromium.org/p/chromium.android.fyi/builders/Memory%20Infra%20Tester and bot is running system_health story with --enable-heap-profiling
Not closing before will figure out why  crbug.com/688651  case passes this test.

well I think there is a big piece missing to the test: nothing is consuming the trace and ensuring that the heap profiler data is there. Which means that if we break it entirely at any point we won't realize.

perezju, kraynov and I just had a chat offline. I think the best way foward is:

1) Extend the existing memory metric (third_party/catapult/tracing/tracing/metrics/system_health/memory_metric.html) so that it can parse and summarize the heap profiler data. 
   This would be useful not just for this test but also for the general idea of re-running benchmarks with the full heap profiler.
   I am confident that a combination of you two (perezju and kraynov) can figure out the right pattern for the metric name (memory.chrome.browser_process.reported_by_chrome.heap_profiler.{malloc, partition_alloc, blinkgc}.{total, num_objects} or similar might be a good start)
2) For the test part, ideally something should ./run_bechmark --output-format=chartjson, consume the chartjson and assert that the above metrics are present and non-zero. Not sure what is the best pattern here (invoke a script that calls run_benchmark and then consumes the output from the recipe?). You folks figure out.

+nednguyen - on the part 2 of #10 above, what would be the best way to create a fail/pass test case that: opens a browser with tracing and memory infra enabled, loads a page, performs some interactions, stops tracing, grabs the trace, runs memory_infra metrics on the trace, and then asserts some properties that must be true on the metrics gathered?

Would this kind of test be a good fit for something like serially_executed_browser_test_case [1]? Or is there a better place for it?

[1]: https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/testing/serially_executed_browser_test_case.py

CL where I'm writing this "fail/pass metric" https://crrev.com/2687593003

re #12: that is precisely what I would expect NOT to happen: i.e. having to re-parse the trace dict ourselves.
The metric system (And the corresponding outputjson) is supposed to take care of parsing and importing all that. we shouldn't have two code paths in the same system to parse the trace.

re #13:
Sure, it will be replaced. It's just a CL link.

#11: you may find the example in https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/web_perf/timeline_based_page_test_unittest.py#L152 useful

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dbc456d4d716612c740256fc91f928983da98cc7

commit dbc456d4d716612c740256fc91f928983da98cc7
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Mon Feb 20 16:35:10 2017

Roll src/third_party/catapult/ df581f5fc..71c4c9aba (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/df581f5fc8de..71c4c9aba898

$ git log df581f5fc..71c4c9aba --date=short --no-merges --format='%ad %ae %s'
2017-02-20 kraynov Add dump_count:heap_profiler in memory metric.

Created with:
  roll-dep src/third_party/catapult
BUG= 670828 

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2709513003
Cr-Commit-Position: refs/heads/master@{#451636}

[modify] https://crrev.com/dbc456d4d716612c740256fc91f928983da98cc7/DEPS

The NextAction date has arrived: 2017-08-15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/54468efa2d28668d7547404902aa4915a79b09a4

commit 54468efa2d28668d7547404902aa4915a79b09a4
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Thu Sep 07 18:48:20 2017

Roll src/third_party/catapult/ c9667ecd2..29f450ae4 (16 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/c9667ecd29cb..29f450ae4dcb

$ git log c9667ecd2..29f450ae4 --date=short --no-merges --format='%ad %ae %s'
2017-09-07 littlecvr [Telemetry] Add ChromeOS to desktop platform list
2017-09-07 dtu [pinpoint] Separate Execution exceptions from result_values.
2017-09-07 lalitm Update memtrack binary version for ARMv7 devices
2017-09-07 nednguyen Revert of Smoke test for heap profiler. (patchset #2 id:20001 of https://codereview.chromium.org/3010173002/ )
2017-09-06 rnephew [Telemetry] Fully get rid of PermanentlyDisableBenchmark.
2017-09-06 benjhayden Add traceUrls to CSVs.
2017-09-06 benjhayden Format traceUrls in generic-set-span.
2017-09-06 benjhayden Remove incorrect test testGet_WithFinish_LabelsBugWithLowestMilestonePossible
2017-09-06 benjhayden Produce all statistics in CSVBuilder.
2017-09-06 sullivan Add CORS headers for whitelisted origins.
2017-09-06 xunjieli [wpr-go] Update comment in telemtry/bin/update_wpr_go_binary
2017-09-06 simonhatch Dashboard - Add calls to graph_revisions and find_anomalies in add_histograms_queue
2017-09-06 loloangela Fix errors related to invalid-name pt. 10
2017-09-06 dtu [pinpoint] Limit executions to one test run each + device sharding.
2017-09-06 xunjieli [Telemetry] Use --ignore-certificate-errors-spki-list to bypass cert errors
2017-09-06 kraynov Smoke test for heap profiler.

Created with:
  roll-dep src/third_party/catapult
BUG=753279, 670828 ,713222, 753948 , 670828 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I81717857fe493d9edb734becf08f69534f3adbb5
Reviewed-on: https://chromium-review.googlesource.com/655362
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#500345}
[modify] https://crrev.com/54468efa2d28668d7547404902aa4915a79b09a4/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d4eee2e6bec26130d10402d0eca2ff3d5ad28817

commit d4eee2e6bec26130d10402d0eca2ff3d5ad28817
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Wed Sep 13 18:15:40 2017

Roll src/third_party/catapult/ c562de2ba..c5be5cb62 (4 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/c562de2ba25b..c5be5cb62e76

$ git log c562de2ba..c5be5cb62 --date=short --no-merges --format='%ad %ae %s'
2017-09-13 dtu [pinpoint] Fix race condition in building + isolate finding.
2017-09-13 perezju [telemetry_mini] Fix websocket method name
2017-09-13 dtu [dashboard] Refactor buildbucket_service.py.
2017-09-13 kraynov Re-land: Smoke test for heap profiler.

Created with:
  roll-dep src/third_party/catapult
BUG=752963, 670828 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: Ibe9a9db9971dfd585628e0518246c1cd0952e4ba
Reviewed-on: https://chromium-review.googlesource.com/665299
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#501693}
[modify] https://crrev.com/d4eee2e6bec26130d10402d0eca2ff3d5ad28817/DEPS

Telemetry smoke test is up and running