Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995356386799571808

=== Auto-CCing suspected CL author lgarron@chromium.org ===

Hi lgarron@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : HSTS preload list additions for Chrome 56.
Author  : Lucas Garron
Commit description:
  
BUG=
R=estark@chromium.org
TBR=palmer@chromium.org

Review URL: https://codereview.chromium.org/2446433003 .

Cr-Commit-Position: refs/heads/master@{#433018}
Commit  : 6f704bd822ebe8599498e5f27af401634f08fc38
Date    : Fri Nov 18 01:41:21 2016


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N  Good?
chromium@433003  0          N/A      2  good
chromium@433017  0          N/A      2  good
chromium@433018  1          N/A      2  bad    <--
chromium@433019  1          N/A      2  bad
chromium@433021  1          N/A      2  bad
chromium@433024  1          N/A      2  bad
chromium@433030  1          N/A      2  bad
chromium@433056  1          N/A      2  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 667432

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.amazon.com.gp.aw.s.ref.is.box..k.nicolas.cage oilpan_gc_times.sync_scroll.key_mobile_sites_smooth
Test Metric: oilpan_idle_mark/http___www.amazon.com_gp_aw_s_ref_is_box_?k_nicolas+cage
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4362
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995356386799571808


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5831849090220032

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Benchmark disabled in https://codereview.chromium.org/2522503003/, although I have no idea why that CL wasn't automatically linked when it was committed.

https://codereview.chromium.org/2446433003 is expected to have a minor performance regression. Would you mind helping me understand how much the performance on this test changed (it says "0.00%"?), and if this is out of the ordinary compared to previous changes to net/http/transport_security_state_static.{json, h}?

Sorry for the confusing message in #3. The bisect actually found that your CL caused a telemetry test failure. The telemetry test is attempting to load a recorded version of http://www.amazon.com/gp/aw/s/ref=is_box_?k=nicolas+cage using web-page-replay and scroll it on Android. It stopped working between r433003 and r433056, and we did a bisect and the bisect shows it failing at your CL.

I'm not sure what HSTS does, but could there be a bad interaction between it and web-page-replay?

Oh! That's the right question to ask.
Indeed, this is not a performance issue, it's actually an expected functionality change.

HSTS upgrades sites to HTTPS automatically. Amazon is now using HTTPS, which means that Chrome automatically redirects the test link to https://www.amazon.com/gp/aw/s/ref=is_box_?k=nicolas+cage as of 6f704bd822ebe8599498e5f27af401634f08fc38 . This is expected, desired (requested directly by Amazon), and semi-permanent.

Presumably, the telemetry test needs to be upgraded to cache the redirected HTTPS page – unless it was meant to test HTTP only, in which case a page on a different domain should be used for this test.

nednguyen@, could you determine which adaptation is appropriate for this test?

To #6: rerecord the page with https version is the correct action here.

Okay, then who is the maintainer for the telemetry test?

Or if you want me to do this, is there a straightforward playbook I can follow to update the test without spending too much time on it?

The instruction for rerecord the page set is in https://www.chromium.org/developers/telemetry/record_a_page_set

Or you can reassign this to the benchmark owner in https://docs.google.com/spreadsheets/d/1R_1BAOd3xeVtR0jn6wB5HHJ2K25mIbKp3iIRQKkX38o/edit#gid=0

tdresser@, could you take this on?

Patch here: https://codereview.chromium.org/2526583002.

I haven't figured out how to get the benchmark to rerecord correctly, I spent some time with Ned trying to figure it out, but I think something in the recording logic isn't working (story filter appears to be filtering out things it shouldn't be sometimes, but not others).

Ned, can you take this over? It looks like our guidance on rerecording benchmarks needs updating.

Yes, I will take over this. Sorry for your hassle to make the WPR working. 

 Issue 669195  has been merged into this issue.

 Issue 669199  has been merged into this issue.

Note: I'm still seeing data stoppage alerts about this.

Hi,

Just checking back on this. We are not able be get data for a couple of performance metrics for Chrome OS for a while now. Any ETA on a fix for this. Thanks!

Annie - this is still valid. Can you find someone to look at this?

Since this is related to WPR recording, I assign this to Ashley.

Checking back to see what the next steps here are. It's been a while since we had this metric. Would it be reasonable to skip the test on amazon's recorded web page and run it on others while someone figures out how to fix the problem?

Pining to check back if there is an update.

Sorry about the delay krk@, it looks like that specific site has been being skipped since November: https://codereview.chromium.org/2522503003 and sync_scroll in particular doesn't seem disabled to me: https://cs.chromium.org/chromium/src/tools/perf/benchmarks/smoothness.py?l=285

Is there a particular disabled metric you want reenabled?

Ned, did someone else end up taking on WPR recording stuff?

#26: no one has redo the WPR recording, iirc. One way to figure out is looking at the git log of WPR json file.

From what I understand, There are issues with running smoothness.top_25_smooth/percentage_smooth.percentage_smooth on Amazon's web page which has been causing test failures and resulting in no performance data for scroll smoothness. Is that correct?

From CrOS standpoint, this is an important metric we track during releases and it would be great to have the data flowing again. 

So, the next steps are to re-record the failing web page? Any short term/quick fixes possible (like disabling tests on the failing web page etc?)

Ohhhh I understand now
I was under the impression it was specifically failing for sync_scroll where it's been disabled at this point. smoothness.top_25_smooth uses Top25SmoothPageSet  instead of KeyMobileSitesSmoothPageSet where it's already been disabled.
We could disable amazon.com here while we work on rerecording the test:
https://cs.chromium.org/chromium/src/tools/perf/page_sets/top_25_smooth.py?type=cs&l=160
I'll disable that now

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/aa3f1ca32be88440c7f71f0956e5427037b36123

commit aa3f1ca32be88440c7f71f0956e5427037b36123
Author: ashleymarie <ashleymarie@chromium.org>
Date: Fri Jun 16 17:10:03 2017

Disabling amazon.com in top_25_smooth page set until we rerecord

Need to rerecord http://amazon.com to https://amazon.com

BUG= chromium:667432 

Review-Url: https://codereview.chromium.org/2940333002
Cr-Commit-Position: refs/heads/master@{#480082}

[modify] https://crrev.com/aa3f1ca32be88440c7f71f0956e5427037b36123/tools/perf/page_sets/top_25_smooth.py

This issue was created > 6 months ago. The perf waterfall has changed significantly since then. If this bug is still relevant, please re-open.

Reopen it.

It seems the amazon.com is turned on again so we still do not have this metrics of smoothness.top_25_smooth

https://chromium.googlesource.com/chromium/src/+/038bd90be90e8fb7ccc2e2b801166cb93a05b4ea%5E%21/tools/perf/page_sets/top_25_smooth.py

It should not be running anywhere.Test expectations moved the disabling of stories. It is disabled here now:
https://cs.chromium.org/chromium/src/tools/perf/benchmarks/smoothness.py?q=smoothness.p&sq=package:chromium&l=70

Is it running somewhere it shouldn't be? If so can you link to some log or buildbot pages?

Hi

It does look like Amazon.com page is excluded from the scroll smoothness test. However, the test itself doesn't seem to be producing results. May be there are other failures? https://wmatrix.googleplex.com/failures/unfiltered?suites=crosbolt_perf_perbuild&tests=telemetry_Benchmarks.smoothness.top_25_smooth&days_back=10&releases=62

Please let me know if I should file a new bug for this.

We have to disable Pinterest to have 'valid' test results in summary for ChromeOS crosbolt dashboard. CL - https://chromium-review.googlesource.com/c/614933.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1480fab6a19a19c4a3f515423cf1dc18bd67bb26

commit 1480fab6a19a19c4a3f515423cf1dc18bd67bb26
Author: Chung-yih Wang <cywang@chromium.org>
Date: Wed Aug 16 20:44:33 2017

[Telemetry] smoothness.top_25_smooth: Disable Pinterest

TBR=achuith@chromium.org

The patch removes the Pinterest page from the top_25_smooth story set as
it is a 404 error during test on ChromeOS.

BUG= 667432 
TEST=test_that --args local=True DUT_IP telemetry_Benchmarks.smoothness.top_25_smooth
and no Pinterest entries in final results.

Signed-off-by: Chung-yih Wang <cywang@chromium.org>
Change-Id: I12f4428e37ae001eae49c7f6ab74d7d9eb8f66d1
Reviewed-on: https://chromium-review.googlesource.com/614933
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Victor Miura <vmiura@chromium.org>
Commit-Queue: Victor Miura <vmiura@chromium.org>
Cr-Commit-Position: refs/heads/master@{#494936}
[modify] https://crrev.com/1480fab6a19a19c4a3f515423cf1dc18bd67bb26/tools/perf/benchmarks/smoothness.py

It's not totally clear what needs to happen in this bug. Should we re-record the pageset?

#39: yes, but it would be easier to do it after we merge them in issue 760553

Sounds like this is not P1.

The following pages have now been re-recorded and re-enabled:

rendering.desktop/amazon
rendering.mobile/amazon
rendering.mobile/amazon_desktop_gpu_raster
rendering.mobile/pinterest_desktop_gpu_raster
rendering.mobile/amazon_mobile
rendering.mobile/amazon_mobile_sync_scroll