Hi there, not super familiar with how to figure out which metrics. From looking at your cl and finding accessibility_metric.html in github, it kinda looks like:

system_health.*/browser_accessibility_*
system_health.*/render_accessibility_*

and then some specific stories:

system_health.*/*/browse_accessibility_*
system_health.*/*/load_accessibility_*

Does that look about right?

To send the alerts to a specific list, I think I need to create a new sheriff rotation, which I think would mean you'd need to triage the alerts yourself.

+sullivan to confirm

That's right.

The metrics are:

browser_accessibility_events
render_accessibility_events
render_accessibility_locations

These are all time durations in ms, and we're interested in the sum, not average. I don't believe these are underneath system_health.

The specific stories look exactly right. I hope to add more but most likely they'd all fall under:

system_health.*/*/browse_accessibility*
system_health.*/*/load_accessibility*

Our team would be happy to triage the alerts as part of a new sheriff rotation. You can send them to chrome-a11y-core@google.com, or you could add these usernames:

dmazzoni     (@chromium.org or @google.com)
dtseng       (@chromium.org or @google.com)
aboxhall     (@chromium.org or @google.com)
nektar       (@chromium.org or @google.com)
aleventhal   (@chromium.org or @google.com)
dougt@chromium.org / dft@google.com

Thanks!

Comments in #2 and #3 look right. We generally do per-page alerting to make regressions easier to bisect and isolate. The chromium perf sheriff rotation will handle general-case regressions to these pages (regressions to page load time, memory, power, etc). You'll probably want to do a daily summary mail, since regressions tend to be bursty (a bunch at the same time)

So I added a new accessibility sheriff, daily emails sent to chrome-ally-core@google.com, test patterns:

ChromiumPerf/*/*/browser_accessibility*/*/*
ChromiumPerf/*/*/render_accessibility*/*/*
ChromiumPerf/*/system_health.*/*/browse_accessibility*/*
ChromiumPerf/*/system_health.*/*/load_accessibility*/*

Annie, do you know if this takes effect immediately or we need to have new data points come in so that the tests update their sheriffs? I'm not seeing any monitored tests for this sheriff yet.

I'm hitting the 1 minute timeout trying to query for paths like ChromiumPerf/*/*/browser_accessibility*/*/*. Are you sure that /edit_sheriffs succeeded when you typed them in? It should have printed a message saying it's adding a list of paths.

Ok think this is all properly submitted now. Ended up rewriting some of the internals of the endpoint to optimize it, since all the queries are done serially. Manually specified the metric/story names and resaved as:

ChromiumPerf/*/*/browser_accessibility_events_sum/*/*
ChromiumPerf/*/*/render_accessibility_events_sum/*/*
ChromiumPerf/*/*/render_accessibility_locations_sum/*/*
ChromiumPerf/*/system_health.*/*/browse_accessibility_tech/*
ChromiumPerf/*/system_health.*/*/browse_accessibility_tools/*
ChromiumPerf/*/system_health.*/*/load_accessibility_media/*
ChromiumPerf/*/system_health.*/*/load_accessibility_shopping/*

These queries are still heavy, I could probably make them faster by specifying the specific benchmarks instead of (ChromiumPerf/*/system_health.*/*/browse_accessibility_tech/*) but this works for now.