Forgot to add the link:
https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg8qvEqQsM

speedometer's regression in the link goes from 1.1 min to 1.3 min

It kinda looks like benchmark_duration is reported in seconds not minutes, so the anomaly config

{
  "min_absolute_change": 5.0
}

Would alert on changes of 5s or more. I can switch that to 300, just want to confirm first that the benchmark_duration is in seconds.

benchmark_duration is in minutes, I am pretty sure: https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/internal/story_runner.py#L375

Though what may happen here is the dashboar automatically convert minute to seconds?

Ah my bad, yes it does look like it's in minutes, I'll dig in a bit and see what's up then.

One thing I noticed digging through the datastore is that, the example link has 2 benchmarks, both speedometer, which alert on < 5 mins. The anomaly config on TestMetadata('ChromiumPerf/win-high-dpi/speedometer/benchmark_duration') is actually assigned to the v8-speedometer anomaly config. The code in TestMetadata that assigns anomaly configs grabs all configs and sorts alphabetically, but is case sensitive so 'benchmark_duration' would fall after 'V8-speedometer' due to case.

Perhaps we should prefer the more specific pattern matching over the general one as a potential fix.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d997518a95d124b2deebb67e37c96cff7fe5c1c2

commit d997518a95d124b2deebb67e37c96cff7fe5c1c2
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Thu May 11 17:11:56 2017

Roll src/third_party/catapult/ 23c9f59f4..59e58189c (1 commit)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/23c9f59f4c6b..59e58189c55d

$ git log 23c9f59f4..59e58189c --date=short --no-merges --format='%ad %ae %s'
2017-05-11 simonhatch Dashboard - Choose anomaly config which most specifically matches.

Created with:
  roll-dep src/third_party/catapult
BUG= 718981 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: Icf38804f7c158bfc919f3e72f124aa576f8f53f6
Reviewed-on: https://chromium-review.googlesource.com/502633
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#470985}
[modify] https://crrev.com/d997518a95d124b2deebb67e37c96cff7fe5c1c2/DEPS

Next step would be to re-put() the affected TestMetadata's.

Ok they've been resaved and I checked a few in datastore and their configs have switched over to benchmark_duration. Can keep an eye out for the alerts and see if anything is alerting on < 5 mins in the future.

Thanks Simon!

I think I just forgot to close this, closing now.