New issue
Advanced search Search tips

Issue 889551 link

Starred by 1 user
Status: Fixed
Owner:
dtu@chromium.org
Closed: Sep 28
Cc:
simonhatch@chromium.org
st...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

[📍] Significance thresholds for smaller regressions

Project Member Reported by dtu@chromium.org, Sep 26 
Currently, Pinpoint's loosest performance threshold can detect regressions as small as 1.0x the interquartile range (IQR). I've seen a few cases where the regression size is smaller than that, so we need to generate thresholds that can detect smaller regressions.

https://pinpoint-dot-chromeperf.appspot.com/job/103f3547640000
https://pinpoint-dot-chromeperf.appspot.com/job/1624a6fb640000
https://pinpoint-dot-chromeperf.appspot.com/job/10ad7c98e40000
Project Member

Comment 1 by bugdroid1@chromium.org, Sep 27 

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/7453eba4feb2d90136a9415a6670ad41b866c93f

commit 7453eba4feb2d90136a9415a6670ad41b866c93f
Author: Dave Tu <dtu@chromium.org>
Date: Thu Sep 27 23:28:10 2018

[pinpoint] Add performance thresholds for smaller regressions.

This lets us detect regressions as small as 0.5x IQR (0.37x std dev).

Bug:  chromium:889551 
Change-Id: Ibc2a6dee08efcb0554db41590eede7e38ee96316
Reviewed-on: https://chromium-review.googlesource.com/1246154
Commit-Queue: Dave Tu <dtu@chromium.org>
Auto-Submit: Dave Tu <dtu@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>

[modify] https://crrev.com/7453eba4feb2d90136a9415a6670ad41b866c93f/dashboard/dashboard/pinpoint/models/compare/thresholds.py
[modify] https://crrev.com/7453eba4feb2d90136a9415a6670ad41b866c93f/dashboard/dashboard/pinpoint/models/compare/thresholds_performance.py
[modify] https://crrev.com/7453eba4feb2d90136a9415a6670ad41b866c93f/dashboard/dashboard/pinpoint/models/compare/thresholds_test.py

Comment 2 by dtu@chromium.org, Sep 28

Status: Fixed (was: Started)
Project Member

Comment 3 by bugdroid1@chromium.org, Sep 28 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a003c5b2a9ed1837d27b5d6b2f9b735a2d6b4c0a

commit a003c5b2a9ed1837d27b5d6b2f9b735a2d6b4c0a
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Sep 28 01:48:21 2018

Roll src/third_party/catapult 920acc565706..7453eba4feb2 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/920acc565706..7453eba4feb2


git log 920acc565706..7453eba4feb2 --date=short --no-merges --format='%ad %ae %s'
2018-09-27 dtu@chromium.org [pinpoint] Add performance thresholds for smaller regressions.


Created with:
  gclient setdep -r src/third_party/catapult@7453eba4feb2

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:889551 
TBR=sullivan@chromium.org

Change-Id: Iaf3ebbb1e33d4148e0c9249a5d34d686b94c97c4
Reviewed-on: https://chromium-review.googlesource.com/1250109
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#594951}
[modify] https://crrev.com/a003c5b2a9ed1837d27b5d6b2f9b735a2d6b4c0a/DEPS
►

Sign in to add a comment