From https://sponge.corp.google.com/invocations?searchFor=video_PlaybackPerf.h264%20squawks%20R70%20after%3A2018-07-24%20before%3A2018-07-25

The test results uploaded to chromeperf right after they were executed. It seems the delay is mainly from chromeperf anomaly detection. Annie, do you have any insight of the issue?



https://screenshot.googleplex.com/aJJxwe1S1aK from crosboltv2

Reassigning to TL simonhatch

Seems it is out of scope from TE end as it is related to perf dash, adding TE-NeedsTraige-help label to move this out of our triaging bucket.

Thanks..!

Hmm this seems to happen periodically but we've never figured out why. Often by the time it's reported, we have trouble going back that far in the logs. I'll see if I can add some additional logging and a notification to the team when this happens so we can dig into this better.

Actually, I've been digging into this and I think I know what's wrong here.

So although it's not visible on the UI, the alert entity itself was generated on Aug 8, the day you were emailed, so there was no delay there. As for why it was alerting so late, from what I can see it was having trouble getting positive signal for the alert. 

New points were streaming in both before and after the alert point, some of them were huge spikes which threw off the alerting algorithm, making it wait for more points before generating the alert.

From https://dev-simonhatch-193e9338-dot-chromeperf.appspot.com/debug_alert?test_path=ChromeOSVideo%2Fcros-squawks%2Fvideo_PlaybackPerf%2Fhw_video_cpu_usage_h264_1080p&rev=35010001090600000&num_before=200&num_after=30&config=%7B%7D

This is a version of the dashboard with the spikes removed, and with default parameters you can see that the alert fired pretty quickly.


https://chromeperf.appspot.com/debug_alert?test_path=ChromeOSVideo%2Fcros-squawks%2Fvideo_PlaybackPerf%2Fhw_video_cpu_usage_h264_1080p&rev=35010001090600000&num_before=200&num_after=30&config=

Here's the untouched data, and even with quite a few data points no alert is generated yet. We could tweak the parameters a bit to have worked better in this case, but I think those kind of spikes always throw off the current implementation.

Thanks for the clear explanation. Actually since spikes may be treated as outliers, what we care about more is the step changing point, the alert latency would make us more difficult to investigate. It would be great if we could shorten the latency.

You have mentioned we could improve this case by tweaking parameters, so do we need to provide anything to you for doing this?

Thanks

 

+dtu for ideas

Playing with some values like max_window_size=30, seems to get the alert considerably faster. Might also result in some spurious alerts though. We could create a new anomaly config for you that overrides the defaults.

Removing myself since investigation is done.

Setting defect without priority to Pri-2.