We do have the tests for that. For example
https://chromeperf.appspot.com/report?sid=ac1bd398bb823319db3b87d18ee83c4e7af20eab525dea2ff5651ef2dff9a354

Although the names are old that don't quite match 1:1 with the metrics names. I feel like we had something for scroll update as well:
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/metrics/smoothness_unittest.py?type=cs&q=gesture_scroll_update_latency&sq=package:chromium&g=0&l=20

But I don't seem to find it in the dashboard.

Another issue is that some of the regressions happen on 99th percentile only and there is no guarantee for a written perf test to reproduce these regressions.

Even for the regression at 99th %ile, I think reproing in test devices isn't impossible (e.g. https://bugs.chromium.org/p/chromium/issues/detail?id=867581#c9). So having some coverage in telemetry would be useful (would also allow bisecting using pinpoint).

For  issue 867581 , it doesn't look like the telemetry metric changed much around the time of the offending CL [1]. Is there anything we could do so regressions like this are easier to detect from chromeperf?

[1] https://chromeperf.appspot.com/report?sid=6475328454d3250a35ea46a102af0c45d8d81417709d8169e6c1186bf0316619&start_rev=556352&end_rev=571551

>Even for the regression at 99th %ile, I think reproing in test devices isn't impossible (e.g. https://bugs.chromium.org/p/chromium/issues/detail?id=867581#c9). So having some coverage in telemetry would be useful (would also allow bisecting using pinpoint).

The repo in that case was possible since we checked the UKM data to find a device and a URL that was showing the regression and then tried to come up with a repo. We already have telemetry tests for scroll latency (https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/metrics/smoothness.py?type=cs&sq=package:chromium&q=ComputeFirstGestureScrollUpdateLatencies&g=0&l=208) which gets executed over some popular websites, but there is no guarantee that this test (or any similar tests) catch the 99th percentile regressions. 

That particular metric seems to report only the first scroll-update. If we reported all scroll-updates, would that better represent the performance?

As far as I have noticed the scroll regression that we see are either only related to the first update (regression on ScrollBegin metrics) or related to updates in general (regressions both on ScrollBegin and ScrollUpdate metrics), either way we would see the regression on ScrollBegin metrics and first-update telemetry should be enough. nzolghadr@ or tdresser@ might know better though: Is it possible to have regressions that affect only non-first scroll updates? (e.i. regression on ScrollUpdate metrics that we don't see on ScrollBegin metrics.)

I'm not aware of any regressions which would only effect updates.

However, a large proportional regression in updates is likely to be small relative to the absolute latency for a scroll begin, which means there are regressions for updates (and scroll begins) which get lost in the noise for scroll begins.

Navid, do you have any update here?

This will be done in this quarter.