Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8962385983184100864

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8962385949285990528

=== Auto-CCing suspected CL author herb@google.com ===

Hi herb@google.com, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : Herb Derby
  Commit : b8b3086de30b3b0f09c6c5ffc176482a538e9337
  Date   : Thu Nov 16 21:39:19 2017
  Subject: Fix Sk8b reading too many bytes

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:webview:all_processes:reported_by_chrome:cc:locked_size_avg/load_media/load_media_soundcloud

Revision                             Exit Code      N
chromium@517139                      0 +- N/A       20      good
chromium@517283                      0 +- N/A       20      good
chromium@517319                      0 +- N/A       20      good
chromium@517324                      0 +- N/A       20      good
chromium@517326                      0 +- N/A       20      good
chromium@517327                      0 +- N/A       20      good
chromium@517327,skia@3bc00fe810      0 +- N/A       20      good
chromium@517327,skia@a94970ba4b      0 +- N/A       20      good
chromium@517327,skia@45b806eec4      0 +- N/A       20      good
chromium@517327,skia@b8b3086de3      1 +- N/A       20      bad       <--
chromium@517328                      1 +- N/A       20      bad
chromium@517337                      1 +- N/A       20      bad
chromium@517355                      1 +- N/A       20      bad
chromium@517426                      1 +- N/A       20      bad
chromium@517713                      1 +- N/A       20      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.media.soundcloud system_health.memory_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8962385983184100864


For feedback, file a bug with component Speed>Bisection

It seems like this might be fixing a different bug so I'm a little hesitant to revert it.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8962279774346700800

=== Auto-CCing suspected CL author herb@google.com ===

Hi herb@google.com, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : Herb Derby
  Commit : 35c556f33e2c9316055537e5683c28ea1645ff49
  Date   : Thu Nov 16 22:46:09 2017
  Subject: Revert "Revert "Direct evaluation of gaussian""

Bisect Details
  Configuration: android_webview_nexus6_aosp_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:webview:all_processes:dump_count_avg/load_media/load_media_soundcloud

Revision                             Exit Code      N
chromium@517130                      0 +- N/A       10      good
chromium@517275                      0 +- N/A       10      good
chromium@517312                      0 +- N/A       10      good
chromium@517321                      0 +- N/A       10      good
chromium@517326                      0 +- N/A       10      good
chromium@517327                      0 +- N/A       10      good
chromium@517327,skia@3bc00fe810      0 +- N/A       10      good
chromium@517327,skia@b8b3086de3      0 +- N/A       10      good
chromium@517327,skia@1e09e461d2      0 +- N/A       10      good
chromium@517327,skia@35c556f33e      1 +- N/A       10      bad       <--
chromium@517328                      1 +- N/A       10      bad
chromium@517330                      1 +- N/A       10      bad
chromium@517348                      1 +- N/A       10      bad
chromium@517420                      1 +- N/A       10      bad
chromium@517710                      1 +- N/A       10      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.media.soundcloud system_health.memory_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8962279774346700800


For feedback, file a bug with component Speed>Bisection

I don't think it is this CL. The code for this CL is not turned on for chrome. It is controlled by the skia configuration flag: SK_USE_LEGACY_INTERP_BLUR.

Yeah, the test is failing flakily. I think whats going on is that its highly likely a CL in that roll, or in a CL immediately before the roll, is whats causing the flakiness. You just keep winning the lottery and having random CLs of yours getting flagged as the culprit. I'll fire off another bisect. 

dtu@ would pinpoint be able to better find the culprit, and if so do you know if there are any plans to move the webview bots to pinpoint soonish?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9f0a9da2e97d5bc7b1a75451994a074f2971c8f3

commit 9f0a9da2e97d5bc7b1a75451994a074f2971c8f3
Author: rnephew <rnephew@chromium.org>
Date: Wed Nov 22 20:45:41 2017

[Telemetry] Disable flaky soundcloud benchmark on webview.

TBR: nednguyen@google.com
Bug: 787001
Change-Id: I9af0a363fd6e4c0e62086e62d1afdaef1e8852ce
Reviewed-on: https://chromium-review.googlesource.com/786291
Reviewed-by: rnephew <rnephew@chromium.org>
Commit-Queue: rnephew <rnephew@chromium.org>
Cr-Commit-Position: refs/heads/master@{#518736}
[modify] https://crrev.com/9f0a9da2e97d5bc7b1a75451994a074f2971c8f3/tools/perf/page_sets/system_health/expectations.py

Going to kick off a perf try job to see if this story magically fixed itself in the last 8 months.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14ca6969a40000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/15fc6e71a40000

📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/14ca6969a40000

📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/15fc6e71a40000

passing 👍

Actually, looks like it failed 5 out of 40 runs

There's no information in the logs about that failure, though. I'm going to try re-enabling.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cfdb39526529c855565e2507f558a557e44c6b08

commit cfdb39526529c855565e2507f558a557e44c6b08
Author: Annie Sullivan <sullivan@chromium.org>
Date: Tue Aug 14 16:51:50 2018

Re-enable load:media:soundcloud for webview on system_health.common_mobile.

It's passing in perf tryjobs.

Bug: 787001
Change-Id: Ie9a74b141d6dd610a1bfab3d9d9ea79b40dafc48
TBR: tdresser@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/1174632
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Annie Sullivan <sullivan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582944}
[modify] https://crrev.com/cfdb39526529c855565e2507f558a557e44c6b08/tools/perf/expectations.config

Setting a NextAction date to swing back around and see how it's doing on the waterfall.

The NextAction date has arrived: 2018-08-16

Looks like soundcloud is still failing flakily (~20% fail rate) on 6X Webview, the only platform where it's running: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/b9eb9fb6-a14b-11e8-9c69-0242ac110004

Stack trace:

(ERROR) 2018-08-16 11:40:02,551 page_test_results.Fail:540  Failure recorded: TypeError: Cannot read property 'rendererHelpers' of undefined
    at new loadingMetric (/tracing/metrics/system_health/loading_metric.html:414:35)
    at runMetrics (/b/swarming/w/ir/third_party/catapult/tracing/tracing/metrics/metric_map_function.html:61:16)
    at metricMapFunction (/b/swarming/w/ir/third_party/catapult/tracing/tracing/metrics/metric_map_function.html:190:24)
    at Object.mapSingleTrace (/tracing/mre/map_single_trace.html:39:7)
    at eval (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:61:18)
    at Object.runAndConvertErrorsToFailures (/tracing/mre/map_single_trace.html:24:10)
    at mapSingleTraceWithResult (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:52:12)
    at Object.mapSingleTraceMain (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:77:20)
    at eval (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:90:15)
    at Object.loadHTMLFile (/b/swarming/w/ir/third_party/catapult/third_party/vinn/vinn/html_imports_loader.js:140:5)
(ERROR) 2018-08-16 11:40:02,551 page_test_results.Fail:540  Failure recorded: TypeError: Cannot read property 'rendererHelpers' of undefined
    at new expectedQueueingTimeMetric (/tracing/metrics/system_health/expected_queueing_time_metric.html:83:55)
    at runMetrics (/b/swarming/w/ir/third_party/catapult/tracing/tracing/metrics/metric_map_function.html:61:16)
    at metricMapFunction (/b/swarming/w/ir/third_party/catapult/tracing/tracing/metrics/metric_map_function.html:190:24)
    at Object.mapSingleTrace (/tracing/mre/map_single_trace.html:39:7)
    at eval (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:61:18)
    at Object.runAndConvertErrorsToFailures (/tracing/mre/map_single_trace.html:24:10)
    at mapSingleTraceWithResult (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:52:12)
    at Object.mapSingleTraceMain (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:77:20)
    at eval (/b/swarming/w/ir/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:90:15)

Deep, do you have any idea what's going wrong here?

Meanwhile, I'm going to revert the reenable.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0adefb35a3a68819f526bf010aa78848e2283d31

commit 0adefb35a3a68819f526bf010aa78848e2283d31
Author: Charlie Andrews <charliea@chromium.org>
Date: Thu Aug 16 17:16:11 2018

Revert "Re-enable load:media:soundcloud for webview on system_health.common_mobile."

This reverts commit cfdb39526529c855565e2507f558a557e44c6b08.

Reason for revert: Story is still failing

Original change's description:
> Re-enable load:media:soundcloud for webview on system_health.common_mobile.
> 
> It's passing in perf tryjobs.
> 
> Bug: 787001
> Change-Id: Ie9a74b141d6dd610a1bfab3d9d9ea79b40dafc48
> TBR: tdresser@chromium.org
> Reviewed-on: https://chromium-review.googlesource.com/1174632
> Reviewed-by: Annie Sullivan <sullivan@chromium.org>
> Commit-Queue: Annie Sullivan <sullivan@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#582944}

NOTRY=true
TBR=sullivan@chromium.org

Bug: 787001
Change-Id: If734162c4815ec7c1a988343859b80556cce6044
Reviewed-on: https://chromium-review.googlesource.com/1178001
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583705}
[modify] https://crrev.com/0adefb35a3a68819f526bf010aa78848e2283d31/tools/perf/expectations.config

/bump here

Deep, do you have any idea when you might be able to take a look at this?

Sorry for the delay - I just blocked off some time on my calendar to work
on this today afternoon. Will have some update by EOD.

Hey Deep,

Any update here?

When I tried last Friday I couldn't reproduce it using my local phone. I will try again this afternoon, and get back with updates. 

/bump

Any luck? Did you make sure to try to run the story on Webview, not just regular Chrome? (I'm actually not 100% sure how to do that, but it does seem to make a difference given. https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=performance_test_suite&tests=system_health.common_mobile%2Fload%3Amedia%3Asoundcloud shows that it's passing 100% of the time on Nexus 5X, but seems to be reliably failing on 5X Webview.)

(Even though I'm not 100% sure how to run a test on a webview browser, if you ping me tomorrow in the office I'm happy to look into it with you.)

Ah wow completely missed that I need webview. I just briefly searched for it, and looks like building webview is not straightforward.

I looked closer at the error message on #23, and it seems to me it's failing on this line, which means chromeHelper is undefined: https://cs.chromium.org/chromium/src/third_party/catapult/tracing/tracing/metrics/system_health/loading_metric.html?l=414&rcl=70783e3968e7bcda29fe37a70fc9a5218ae12b37

This in turn means ChromeModelHelper.supportsModel returned false for the model: https://cs.chromium.org/chromium/src/third_party/catapult/tracing/tracing/model/helpers/chrome_model_helper.html?l=70&rcl=70783e3968e7bcda29fe37a70fc9a5218ae12b37

This in turn means catapult fails to find a single browser process or a renderer process in the trace. This could be because we're not recording the trace properly, or it could be because on webview the processes have different names (which would be very surprising - I thought we had other metrics that we run against webview.) I can't say for sure what is happening unless I can get a trace.

Is there anyway you could help get me a trace? Would it be easy to hack the benchmark so a trace shows up on pinpoint and we get a trace from the failing bot? Otherwise I'll try by the end of this week to get webview running and repro this.

A similar issue cropped up in  Issue 886621 . It had the same issue where catapult could not recognize the correct processes. The issue has likely just been fixed by oysteine. Can we check the flakiness of this again? 

Hey Deep, it looks like this is still happening: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/aefead26-1056-11e9-80c9-0242ac110006 (from today)

Any update here?

Huh the bug is still there - that's unfortunate. Getting a trace here would be super nice, but I'm guessing that's not straightforward. I have a few questions to help me debug this:

(1) Where can I see the fails happening? I can't find this on the waterfall here: https://ci.chromium.org/p/chromium/g/chromium.perf/console

(2) If I want to reproduce the fails on a bot, can I run a pinpoint job to see this? Is this a good config for the pinpoint job? https://screenshot.googleplex.com/ww6KHpvs0he

(3) To get a trace out of pinpoint on the failing run, I'm thinking I can change the benchmark code to upload the trace and add it as a CL on pinpoint. Let me know if there's an easier way.