Specifically, are you suggesting sending 1 metric per stage of the scheduler tick with a field indicating the stage name?

Re 2, yes (probably).

I'm thinking either:
 - A floating point gauge metric with subfield 'scheduler_stage'.
 - A floating point distribution metric with 'schdeuler_stage'.

Each has its own relative merit. Gauge is easier to graph with a simple query, but can be missing a bit of information, especially if the tick time is short relative to the ts_mon flush period. Realistically gauge is probably fine.

I think we want distribution, right? Seems like the perfect fit - we are repeatedly sampling a random variable.

Yes, distribution would be the more pedantically "correct" choice. It does make the queries slightly harder, and means we can't showing all the sub-times and the aggregate time using a single metric.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f2c9ac9f514b2c56799974c98c896ae03119c34d

commit f2c9ac9f514b2c56799974c98c896ae03119c34d
Author: Aviv Keshet <akeshet@chromium.org>
Date: Fri Sep 16 22:44:26 2016

autotest: add stats around db cleanup time

Note, this change requires 386284 to be pushed to prod before it can
land.

CQ-DEPEND=CL:386284
BUG= chromium:626423 
TEST=None

Change-Id: Ie78b731a2797574c89f3a0f855ed2f80768395a2
Reviewed-on: https://chromium-review.googlesource.com/386333
Commit-Ready: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/f2c9ac9f514b2c56799974c98c896ae03119c34d/scheduler/monitor_db_cleanup.py