Bisect: https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2940

===== BISECT JOB RESULTS =====
Status: completed


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@389738  1          N/A      10  good
chromium@389810  1          N/A      10  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 607616

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests smoothness.scrolling_tough_ad_cases
Test Metric: frame_times/frame_times
Relative Change: 0.00%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2940
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9014116364709012064


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5814357821227008

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Noticed this in log output:

Console output:
(log) http://cdnapi.kaltura.com/html5/html5lib/v2.32/mwEmbedLoader.php/p/591531/uiconf_id/30181102:49: kWidget: Kaltura HTML5 Version: 2.32
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(warning) http://ads.nexage.com/js/admax/admax_api.js:1: Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/.
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) :1: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) :1: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://s1.2mdn.net/ads/richmedia/studio/pv2/37428067/20150722093412483/assets/primarySmoke.png:0: Failed to load resource: the server responded with a status of 404 (Not Found)
(error) http://s1.2mdn.net/ads/richmedia/studio/pv2/37428067/20150722093412483/assets/glassmark.png:0: Failed to load resource: the server responded with a status of 404 (Not Found)
(error) http://ox-d.pubgears.com/w/1.0/acj?ee=mul&ai=7a07d606-9c07-4ecf-4a73-4071a6ae3872&o=3619999999&callback=OX_3619999999&ju=http%3A//m.tmz.com/&jr=&auid=538004706&dims=600x791&adxy=0%2C0&res=600x960x32&plg=pm&ch=UTF-8&tz=480&ws=600x791&vmt=1&sd=1:1: Uncaught ReferenceError: OX_4619999999 is not defined
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://js.moatads.com/tmz695915919965/moatad.js:30: Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).
(error) http://ll-assets.tmz.com/mobile-web/v1.6.5/Scripts/app.js:7: Uncaught ReferenceError: pSUPERFLY is not defined
(error) http://metrics.tmz.com/b/ss/wbrostmz/1/H.20.3/s46620000000000?AQB=1&ndh=1&t=29/1/2008%202%3A26%3A9%205%20480&vmt=4A98566A&ns=warnerbros&pageName=Mobile%20Site%20Homepage%20-%201&g=http%3A//m.tmz.com/%23Articles/all&cc=USD&ch=Mobile%20Site%20Homepage&events=event6&c1=TMZ.mobileweb&v1=TMZ.mobileweb&c2=Mobile%20Website&v2=Mobile%20Website&c3=Mobile%20Site%20Homepage&v3=Mobile%20Site%20Homepage&c14=Data%20Not%20Available&v14=Data%20Not%20Available&c15=Data%20Not%20Available&v15=Data%20Not%20Available&c16=Data%20Not%20Available&v16=Data%20Not%20Available&s=600x960&c=32&j=1.6&v=N&k=Y&bw=600&bh=791&AQE=1:0: Failed to load resource: the server responded with a status of 404 (Not Found)
(error) https://app.tmz.com/mobileweb/v1.2/articles?page=1:0: Failed to load resource: net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH
(log) http://s1.2mdn.net/ads/richmedia/studio/pv2/37428067/20150722093412483/app.js:376: 51

Attempt #2: https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2941

#2 failed on provision_device, trying again.

Attempt #3: https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2946

===== BISECT JOB RESULTS =====
Status: completed


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@389461  1          N/A      10  good
chromium@389810  1          N/A      10  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 607616

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests smoothness.scrolling_tough_ad_cases
Test Metric: frame_times/frame_times
Relative Change: 0.00%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2946
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9014015715398223680


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5863387121057792

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Update: Still haven't found the cause, or when this started happening. Hard to tell exactly because of the other issues around the same time. The last bisect job showed that on android_nexus7_perf_bisect, it failed for revision 389461.

Trying another return code bisect job: https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2957

Meanwhile, maybe this should be disabled? Disabling CL: https://codereview.chromium.org/1949843003/

Ned, any clues about the secure origin output in #3?

The failing page is http://m.tmz.com/#news/ 
I guess this is another case which we have some code in chromium that do "this feature is not secured and hence will be deprecated on date X/Y/Z"

I just check the page m.tmz.com, they updated those urls to https (e.g: https://z.moatads.com/kargodfp680570393645/moatad.js). I think all we need to do is to rerecord this  http://m.tmz.com/#news/ page.

re #8: Yeah, looks like my disable failed on one of the s5 cq bots, removing that and trying again.
re #10: Should I continue with disabling?

No, disabling the whole benchmark will make it impossible to bisect the regression during the whole time it's disabled. For now, just comment out https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/page_sets/tough_ad_cases.py&l=218 instead. Bonus point if you can rerecord that page :P

+dtu, prasadv, robertocn

Re #12: Shouldn't bisect always run with --also-run-disabled-benchmarks?

https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2957/steps/config/logs/stdio

Bisect always includes --also-run-disabled while running benchmarks.
This flag is set by the dashboard here: https://github.com/catapult-project/catapult/blob/master/dashboard/dashboard/start_try_job.py#L457

sgtm

Looks like everyone beat me to asking about --also-run-disabled-tests

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1f8f1a5127b61b565bc772ba41cec3200a43ebe9

commit 1f8f1a5127b61b565bc772ba41cec3200a43ebe9
Author: simonhatch <simonhatch@chromium.org>
Date: Wed May 04 20:02:17 2016

Disable tmz.com page in tough_ad_cases

TBR=skyostil, nednguyen
BUG= 607616 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Review-Url: https://codereview.chromium.org/1936293002
Cr-Commit-Position: refs/heads/master@{#391607}

[modify] https://crrev.com/1f8f1a5127b61b565bc772ba41cec3200a43ebe9/tools/perf/page_sets/tough_ad_cases.py

===== BISECT JOB RESULTS =====
Status: completed


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@389000  1          N/A      20  good
chromium@389788  1          N/A      20  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 607616

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests smoothness.scrolling_tough_ad_cases
Test Metric: frame_times/frame_times
Relative Change: 0.00%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2957
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9013575396962379040


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5781970647252992

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

According #10, it looks like the next step is to re-record that page and re-enable.

Friendly sheriff ping

(According to the Perf Bot Sheriffing guide, Pri-1 build bot bugs should be pinged daily. Please close the issue if it's already been fixed or lower its priority.)

I just try to re-recording m.tmz.com but it's no longer work because the page now has a pop-up ad. We will need to expand the page to include the logic of closing the pop-up add so it's scrollable again. 

Not sure whether the benchmark owner still interested in this page though (+Sami)

To keep things simple we could just drop tmz.com. The test would still have good coverage.

 Issue 609245  has been merged into this issue.