Bisect: https://chromeperf.appspot.com/buildbucket_job_status/9016931360340923328

===== BISECT JOB RESULTS =====
Status: completed


===== TESTED REVISIONS =====
Revision                Exit Code   Std. Dev.   Num Values  Good?
chromium@382200         0           N/A         5           good
chromium@382203         0           N/A         5           bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 598332

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --also-run-disabled-tests smoothness.scrolling_tough_ad_cases
Test Metric: first_gesture_scroll_update_latency/http___www.androidauthority.com_reduce-data-use-turn-on-data-compression-in-chrome-630064_
Relative Change: 0.00%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/3528
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9016931360340923328


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=598332

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Looking at the results, the metric isn't present in any of the chartjson (links to build 1 and 5):
https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/3528/steps/Gathering%20reference%20values.Reading%20chartjson%20results/logs/json.output
https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/3528/steps/Gathering%20reference%20values.Reading%20chartjson%20results%20%285%29/logs/json.output

I realized a return_code bisect isn't actually what I want, since the return_code is 0 whether this value is output or not. I tried a mean bisect here: https://chromeperf.appspot.com/buildbucket_job_status/9016921865120956272

But strangely, the value wasn't output in the bisect for the last revision that sent data to the perf dashboard. I'm not sure what's going on here.

Overall, this metric seems to be really flaky. It stops appearing on different dates on about half our android devices:
https://chromeperf.appspot.com/report?sid=62dfbed5f229ce45bca65af58d908869cc058de15d853654393ed0a9c92ff89d

(It's fine on nexus5x, nexus7v2, and nexus9).

Assigning to reveman, owner of smoothness.tough_scrolling_cases benchmark: do we need this page in the pageset?

I don't think we care about this metric for tough_scrolling_cases.

Ned: This metric is pretty flaky on this specific benchmark, and the owner doesn't care about it for this benchmark. Is there an easy way to just disable it?

We can filter out the metric by overriding CanValueBeAddedPredicate: https://code.google.com/p/chromium/codesearch#chromium/src/third_party/catapult/telemetry/telemetry/benchmark.py&l=171

Assign to benchmark owner to filter out the metrics.

Looks like smoothness.tough_scrolling_cases should already be limited to the metrics we care about:

https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/benchmarks/smoothness.py&l=407

not sure how to eliminate this first_gesture_scroll_update_latency issue if it's already setup as a metric that we don't care about.

I just run the benchmark smoothness.tough_scrolling_cases locally and can verify that it already filters out these metrics:
queueing_durations
frame_time_discrepancy
mean_input_event_latency
input_event_latency_discrepancy
first_gesture_scroll_update_latency
frame_times
mean_frame_time
percentage_smooth


However, the "http://www.androidauthority.com/" page Annie mention in #1 is from smoothness.tough_ad_cases benchmark. So the question is whether we want to filter out first_gesture_scroll_update_latency metric from smoothness.tough_ad_cases

skyostil@, assigning to you as test owner. Should we limit the smoothness.tough_ad_cases to pixels_approximated/checkerboarder metrics as we do for touch_scrolling_cases?

Measuring first_gesture_scroll_update_latency in smoothness.tough_ad_cases doesn't make sense because that test doesn't scroll -- it just animates a bunch of ads.

I've got a patch to filter this out but I'm not quite sure why I need to do this explicitly -- the test shouldn't be generating this value in the first place.

https://codereview.chromium.org/2304703003

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cf4d11d63dc2da3b3801e9d63c08f7ab5ec3b7d6

commit cf4d11d63dc2da3b3801e9d63c08f7ab5ec3b7d6
Author: skyostil <skyostil@chromium.org>
Date: Sat Sep 03 18:17:26 2016

perf: Don't record first_gesture_scroll_update_latency in smoothness.tough_ad_cases

BUG= 598332 

Review-Url: https://codereview.chromium.org/2304703003
Cr-Commit-Position: refs/heads/master@{#416455}

[modify] https://crrev.com/cf4d11d63dc2da3b3801e9d63c08f7ab5ec3b7d6/tools/perf/benchmarks/smoothness.py

smoothness.tough_ad_cases seems to be passing successfully now:

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%282%29/builds/3809/steps/smoothness.tough_ad_cases/logs/stdio