@ paul - how much work would this be? 

Paul, can you update the bug with some idea about how long this may take? Un-assign at that point to put back in triage queue.

This would probably take about 1-3 days.

We would need an upstart daemon which looks up CPU information from /proc every minute for a whitelisted set of processes, and report that to Monarch.

sysmon should do this

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/51bb61229e4465c573673e959dcc4b7bdd4d0106

commit 51bb61229e4465c573673e959dcc4b7bdd4d0106
Author: Allen Li <ayatane@chromium.org>
Date: Thu Jun 22 06:38:04 2017

sysmon: Move process metrics to its own module

Well be adding more metrics, so split it out first before it gets
bigger.

BUG= chromium:728911 
TEST=None

Change-Id: I9d3948bab815eff69bfde159ed524f7f060bc9ea
Reviewed-on: https://chromium-review.googlesource.com/543898
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/51bb61229e4465c573673e959dcc4b7bdd4d0106/scripts/sysmon/__main__.py
[add] https://crrev.com/51bb61229e4465c573673e959dcc4b7bdd4d0106/scripts/sysmon/proc_metrics.py
[modify] https://crrev.com/51bb61229e4465c573673e959dcc4b7bdd4d0106/scripts/sysmon/system_metrics.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/fd10384504481a7fd40da7b5407831ea4ac95dea

commit fd10384504481a7fd40da7b5407831ea4ac95dea
Author: Allen Li <ayatane@chromium.org>
Date: Thu Jun 22 06:38:05 2017

Add a venv python interpreter entry point

This makes it easier to start a python interpreter for poking around,
since it has the same venv environment that is used by chromite venv
scripts.

BUG= chromium:728911 
TEST=None

Change-Id: I6bd9a5a530a922acb7b2503409b2a8a48eb545e4
Reviewed-on: https://chromium-review.googlesource.com/543899
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Paul Hobbs <phobbs@google.com>

[add] https://crrev.com/fd10384504481a7fd40da7b5407831ea4ac95dea/bin/venv_python

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f051d034c8de308f64478aa07a01b68c68cd4646

commit f051d034c8de308f64478aa07a01b68c68cd4646
Author: Allen Li <ayatane@chromium.org>
Date: Thu Jun 22 06:38:05 2017

sysmon: Remove cpu_time_percent() primer

1. Based on manual tests, importing psutil already primes it.
2. This logic shouldnt go here; it should go in the collection
   function.
3. At worst, this will result in a single 0 CPU data point every few
   months when sysmon restarts.

BUG= chromium:728911 
TEST=None

Change-Id: I83c6cec702aef0e1c63da333c47a1dc382b4acbd
Reviewed-on: https://chromium-review.googlesource.com/543900
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/f051d034c8de308f64478aa07a01b68c68cd4646/scripts/sysmon/__main__.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/6bb74d56612b3c7c8819d6ae47fd4e1a0e1f95e3

commit 6bb74d56612b3c7c8819d6ae47fd4e1a0e1f95e3
Author: Allen Li <ayatane@chromium.org>
Date: Sat Jun 24 05:56:46 2017

sysmon: Add CPU percent per process metric

BUG= chromium:728911 
TEST=None

This commit is the combination of multiple commits.  Their commit
messages are as follows:

commit c25ee1049101d6dba3211644586ed918be0881b6
Author: Allen Li <ayatane@chromium.org>
Date:   Wed Jun 21 12:23:46 2017 -0700

    sysmon: Make a class for process metric collection

    The current code structure encourages linear growth of the function
    size.

    BUG= chromium:728911 
    TEST=None

    Change-Id: Ic9a1c027f6e0d47d6e915b76879663d28c63632a

commit 291db06dfa626142f9eec8ee82bee82b179fdba9
Author: Allen Li <ayatane@chromium.org>
Date:   Wed Jun 21 12:39:22 2017 -0700

    sysmon: Extract per-process metrics into class

    BUG= chromium:728911 
    TEST=None

    Change-Id: Ib688a21336cde877bea86e1bbaa9c4746e7f073e

commit 84a7ae65e88bbcf16dab2904f1c15fb56717592f
Author: Allen Li <ayatane@chromium.org>
Date:   Wed Jun 21 13:03:10 2017 -0700

    sysmon: Use Monarch streams for process count

    BUG= chromium:728911 
    TEST=None

    Change-Id: I31d12e561a98632b5aa2993cfc8ffa2d49b2669c

commit 16a01ee2197656d670db7346437124bd532f450f
Author: Allen Li <ayatane@chromium.org>
Date:   Wed Jun 21 14:25:00 2017 -0700

    sysmon: Add CPU percent per process metric

    Im not sure how accurate summing the CPU percent for all processes is
    with respect to threads and such, but we can push this out first and
    augment the logic if the metrics are too far off.

    BUG= chromium:728911 
    TEST=None

    Change-Id: Ibe3ea7fa90940bba8c32af65e55290391eae7c49

Change-Id: I67cf7bed920700df4fc7989c181e0f19960f3f87
Reviewed-on: https://chromium-review.googlesource.com/545102
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/6bb74d56612b3c7c8819d6ae47fd4e1a0e1f95e3/scripts/sysmon/proc_metrics.py