Some more examples of graphs that look like they should have been flagged (very clear regressions): https://chromeperf.appspot.com/report?sid=f56184ed0fd9f0df20ceab60371f75902e92cd2cfd52477eefc55b396f4e7e33&start_rev=399679&end_rev=402150

One other thing to note, it looks like confidence interval data cuts off on all these graphs towards the end - not sure if this is related.

Assigning to vmiura, owner of smoothness.top_25_smooth benchmark, to weigh in on monitoring and reassign to me if changes are needed.

We're only monitoring the following metrics for smoothness.top_25_smooth:
  first_gesture_scroll_update_latency
  frame_times
  mean_input_event_latency

Victor, should we be monitoring percentage_smooth as well? I know you mentioned some caveats on another thread:

"we compute it differently on Android & other platforms.  Android has ground truth data from SurfaceFlinger, so we can know if we really missed Vsync.  Other platforms including Windows are using the Renderer / Browser swap timestamps, which can move forward & backward inside a frame and still meet Vsync. So, take with a pinch of salt and investigate percentage_smooth regressions by examining the traces."

Oh, re comment 2: the confidence interval changes are unrelated, they're a result of https://codereview.chromium.org/2087463002/

It would be good to monitor percentage_smooth.

I think the caveat mentioned in #3 applies to this.  I took a look at the traces for the Blogger page which shows a regression from 79% to 62% smooth.  It looks to me like in reality this is almost 100% smooth if you look at the swap times relative to vsync.

While a sampling of timestamps for BenchmarkInstrumentation::ImplThreadRenderingStats look like:

Timestanp(ms)   Delta(ms)
6,481.72	16.00
6,498.29	16.57
6,516.60	18.31
6,535.78	19.18
6,548.65	12.87
6,566.50	17.85
6,582.01	15.51
6,597.94	15.93
6,614.51	16.57
6,631.88	17.36
6,648.15	16.28
6,666.68	18.53
6,682.03	15.35
6,699.82	17.78
6,719.76	19.94
6,731.53	11.77
6,752.98	21.45
6,764.88	11.90
6,783.70	18.83
6,797.84	14.14
6,820.84	23.00
6,832.80	11.96
6,848.76	15.96

We count any frame delta over 17ms as a jank / unsmooth frame, which is bogus here.

Note, there is a 0.2ms regression in mean_frame_time which is concerning.

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Refactor flash component installer to use DefaultComponentInstaller.
Author  : waffles
Commit description:
  This enables it to use differential updates.

BUG= 601928 

Review-Url: https://codereview.chromium.org/2041573002
Cr-Commit-Position: refs/heads/master@{#401973}
Commit  : 5a2538963cbd28c53ed0f4befc24bd9b4d99f77c
Date    : Fri Jun 24 21:24:53 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev    N  Good?
chromium@401951  16.6594  0.0128374  5  good
chromium@401967  16.6836  0.05321    5  good
chromium@401971  16.7114  0.0684346  5  good
chromium@401972  16.667   0.0115758  5  good
chromium@401973  21.3682  0.61642    5  bad    <--
chromium@401975  22.1506  1.35735    5  bad
chromium@401983  21.3198  0.710155   5  bad

Bisect job ran on: mac_10_11_perf_bisect
Bug ID: 623640

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests smoothness.top_25_smooth
Test Metric: mean_frame_time/ESPN
Relative Change: 27.97%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_10_11_perf_bisect/builds/704
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9008476826246730016


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6380609794998272

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Re-opening as the alerting issue isn't fixed.

The regression does seem to be from the change in #7.

Sorry for the delay, was OOO. Monitoring updated.