All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=702175

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZymAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImpDaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL2xvYWRpbmcubW9iaWxlL3RpbWVUb0ZpcnN0Q29udGVudGZ1bFBhaW50X2F2Zy9SZWd1bGFyLTNHL2h0dHBfX193d3cuc2JzLmNvLmtyDAsSDVN0b3BwYWdlQWxlcnQYg8obDA,agxzfmNocm9tZXBlcmZymQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImtDaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL2xvYWRpbmcubW9iaWxlL3RpbWVUb0ZpcnN0Q29udGVudGZ1bFBhaW50X2F2Zy9SZWd1bGFyLTNHL2h0dHBfX193d3cubG9jYW50by5pbgwLEg1TdG9wcGFnZUFsZXJ0GIPKGww,agxzfmNocm9tZXBlcmZymQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImtDaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL2xvYWRpbmcubW9iaWxlL3RpbWVUb0ZpcnN0Q29udGVudGZ1bFBhaW50X2F2Zy9SZWd1bGFyLTNHL2h0dHBfX193d3cuZ3NzaG9wLmNvbQwLEg1TdG9wcGFnZUFsZXJ0GIPKGww,agxzfmNocm9tZXBlcmZylQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImdDaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL2xvYWRpbmcubW9iaWxlL3RpbWVUb0ZpcnN0Q29udGVudGZ1bFBhaW50X2F2Zy9SZWd1bGFyLTNHL2h0dHBfX193d3cuZWJzLmluDAsSDVN0b3BwYWdlQWxlcnQYg8obDA,agxzfmNocm9tZXBlcmZynAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im5DaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL2xvYWRpbmcubW9iaWxlL3RpbWVUb0ZpcnN0Q29udGVudGZ1bFBhaW50X2F2Zy9SZWd1bGFyLTNHL2h0dHBfX193d3cuZHJhbWFxLmNvbS50dwwLEg1TdG9wcGFnZUFsZXJ0GIPKGww,agxzfmNocm9tZXBlcmZyngELEhNTdG9wcGFnZUFsZXJ0UGFyZW50InBDaHJvbWl1bVBlcmYvYW5kcm9pZC1uZXh1czVYL2xvYWRpbmcubW9iaWxlL3RpbWVUb0ZpcnN0Q29udGVudGZ1bFBhaW50X2F2Zy9SZWd1bGFyLTNHL2h0dHBfX193d3cuYnJhZGVzY28uY29tLmJyDAsSDVN0b3BwYWdlQWxlcnQYg8obDA


Bot(s) for this bug's original alert(s):

android-nexus5X

ccing perf bot sheriff.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8984962777978033984

I kicked off a return_code bisect since this went from green to red.

This seems like the benchmark was timed out, hence did not produce any data to me. Stephen: can you take a look?

=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: android_nexus5X_perf_bisect
  Benchmark    : loading.mobile
  Metric       : timeToFirstContentfulPaint_avg/Regular-3G/http___www.sbs.co.kr

Revision             Exit Code      N
chromium@451843      0 +- N/A       20      good
chromium@452846      0 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.sbs.co.kr loading.mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8984962777978033984

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5785982080647168


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Yep. Its timing out from the android test runners internal timeout. I can bump it up some, but its already sitting at 5 hours.


I 18002.971s device_shard_helper(0)  loading.mobile : exit_code=-1 in 18000 secs on device 01e14fdcc2bb4526

Kouhei, Ksakamoto: can we remove a few urls from the loading.mobile. 5 hours is too much for our infrastructure to handle at the moment.

Assign to kouhei@ for gathering list of URLs we can punt from loading.mobile.

P1 because we are losing perf coverage here.

The runs I looked at were running ~180 out of the 200 stories, fyi.

50 stories, each run with and without traffic shapping. Each config is run 2 times.

We would have to eliminate somewhere around 5 urls to get this passing reliably. Less if we have a few 'bad apples' that are driving it higher, I have not looked into run times of individual pages though.

I think another solution worth at least talking about is splitting loading.mobile into two benchmarks. One that runs with traffic shaping and one that runs without. This would also allow for better sharding of tests since loading.mobile is the longest running test, and if we run it on two shards it should decrease cycle time as well as stop timeouts. This would yield 2 benchmarks, perhaps with names similar to 'loading.mobile' and 'loading.mobile.3g'.

Adding martiniss@ who has done a lot of work looking at perf cycle times.

I'm not sure why it was decided to run loading.mobile w/ and w/o traffic shaping as one benchmark, so maybe this has already been discussed and decided against.

Also since TTI is still being developed & no one is monitoring it. I am thinking that we should stop reporting TTI & ignore the TTI wait in https://cs.chromium.org/chromium/src/tools/perf/page_sets/page_cycler_story.py?q=page_cycler_story&dr&l=10

Tim & Kouhei: is this ok?

Here is running time of each page from a Nexus5X run [1], and from a Nexus6 run [2] for comparison.

https://docs.google.com/spreadsheets/d/1DI8xr6ul3Ws702NnXtpFetB7CWctnkeS1fDiL58IaBE/edit?usp=sharing

Nexus5X 3G takes almost 5 times longer than no traffic shaping. It is 2x on Nexus6 (sheet 2). Why traffic-shaped run on Nexus5X is so slow (or fast on Nexus6)?

I don't see clear 'bad apples'.
+1 to remove some url, I think we can reduce the number of {easy,tough}_{ttfmp,tti} cases.


[1] https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%283%29/builds/3535
[2] https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%283%29/builds/5189

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/671b10cba57b286d7a38d0f1f84a200353c8fe6b

commit 671b10cba57b286d7a38d0f1f84a200353c8fe6b
Author: ksakamoto <ksakamoto@chromium.org>
Date: Fri Mar 17 05:35:32 2017

Reduce number of URLs in LoadingMobileStorySet

Running time of loading.mobile test on nexus5X bot is exceeding 5 hours
timeout. This patch reduces number of each {easy,tough}_{ttfmp,tti} case
from 5 to 3, total stories from 50 to 42.

BUG= 702175 

Review-Url: https://codereview.chromium.org/2754883003
Cr-Commit-Position: refs/heads/master@{#457696}

[modify] https://crrev.com/671b10cba57b286d7a38d0f1f84a200353c8fe6b/tools/perf/page_sets/loading_mobile.py

Dropping TTI for now SGTM.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70661d59960a9a3cc2175eafb66db3c4e537c979

commit 70661d59960a9a3cc2175eafb66db3c4e537c979
Author: nednguyen <nednguyen@google.com>
Date: Tue Mar 21 00:45:15 2017

Remove TTI wait for loading.mobile pages

Rationale: TTI metrics are still being developed & not tracked by any team.
Mean while, wait extra for loading is consuming too much of bot' cycle time
without much benefits.

BUG= 702175 

Review-Url: https://codereview.chromium.org/2756183002
Cr-Commit-Position: refs/heads/master@{#458259}

[modify] https://crrev.com/70661d59960a9a3cc2175eafb66db3c4e537c979/tools/perf/page_sets/page_cycler_story.py

It's passing now.