It seems that there are several culprits.

- One started from build 2619 (https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus6%20WebView%20Perf/2619), for which https://chromium-review.googlesource.com/c/catapult/+/1216504 is responsible ( crbug.com/884999 ).

- Another issue (or issues) started from build 2659 (https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus6%20WebView%20Perf/2659) and increased the number of failures.

- A fix (https://chromium-review.googlesource.com/c/catapult/+/1233939) for the first issue was landed in build 2676 (https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus6%20WebView%20Perf/2676) which decreased the number of failures.

So the remaining unknown culprit is in the revision range 591923-591934.

Is it possible to get a log from the remaining failures?

Suspecting one of the following CLs:

1- https://chromium-review.googlesource.com/c/chromium/src/+/1224676: Delete old GrContext resources in ScheduleGrContextCleanup.

2- https://chromium-review.googlesource.com/c/chromium/src/+/1227593: gpu: Make ServiceFontManager thread-safe.

3- https://chromium-review.googlesource.com/c/chromium/src/+/1229195: Android: Add chrome Build ID as metadata

The first two because they are GPU related and I'm seeing a lot of errors like the following in the logs (not sure if they are expected):

[ERROR:gles2_cmd_decoder.cc(17797)] [GroupMarkerNotSet( crbug.com/242999 )!:5061749D]GL ERROR :GL_INVALID_OPERATION : glCreateAndConsumeTextureCHROMIUM: invalid mailbox name

The third one because it's related to tracing on Android.

Here is a log file: https://chrome-swarming.appspot.com/task?id=400c7ffc29aa5e10&refresh=10&show_raw=1

But, Telemetry has failed to get a stack trace of the native crash.

I think this might be related to issue 887422 which can cause a GPU hang. I have a fix up for it: https://chromium-review.googlesource.com/c/chromium/src/+/1236455

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/84aec815ac3e956690aef30af88813d7053d49fe

commit 84aec815ac3e956690aef30af88813d7053d49fe
Author: Khushal <khushalsagar@chromium.org>
Date: Thu Sep 20 17:24:27 2018

gpu: Fix deserialization error in ServiceFontManager.

We deserialize skia data while holding a lock, which we block on if
skia tries to delete a handle as this data is creating new entries in
the strike cache.

R=ericrk@chromium.org

Bug: 887422, 887152
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: Id32cb071834186e7de0dac16902c452c3ccdb102
Reviewed-on: https://chromium-review.googlesource.com/1236455
Reviewed-by: Eric Karl <ericrk@chromium.org>
Commit-Queue: Khushal <khushalsagar@chromium.org>
Cr-Commit-Position: refs/heads/master@{#592847}
[modify] https://crrev.com/84aec815ac3e956690aef30af88813d7053d49fe/gpu/command_buffer/service/service_font_manager.cc

Looks like its fixed now, here is the first build with the change: https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus6%20WebView%20Perf/2685

The only test that failed is v8.browsing_mobile-future/browse:shopping:avito, which I think is a different issue. I just see a lot of logs for failed to load resource:

(WARNING) 2018-09-20 18:06:55,961 inspector_log._OnMessage:26  DevTools console [network]: Failed to load resource: the server responded with a status of 404 (Not Found) https://images-na.ssl-images-amazon.com/images/I/71DZLRD6l2L._AC_UL210_SR210,210_.jpg

Assigning back to chiniforooshan@ for triaging this.

dtu@, do you know why the bot is stopped?

+nednguyen@, I believe this is related to the LUCI migration.
The builder has moved from Buildbot to LUCI, so the above link is out of date. You can find the builds continuing at:
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus6%20WebView%20Perf

Yes, sorry for the ruckus. The new link to view is https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus6%20WebView%20Perf/2710

The failing/flaky tests are:
Unexpected Failures:
* loading.mobile/EnquiryIndianRail_3g
* loading.mobile/FlipBoard_hot_3g
* loading.mobile/FlipKart_hot
* loading.mobile/FlipKart_hot_3g
* loading.mobile/FlipKart_warm
* loading.mobile/GSShop
* loading.mobile/TheStar
* loading.mobile/VoiceMemos_cold_3g
* rendering.mobile/idle_power_animated_gif
* system_health.common_mobile/browse:shopping:avito
 
Unexpected Flakes:
* loading.mobile/QQNews_3g
* system_health.memory_mobile/browse:shopping:avito
 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a60c802709581c621f5bfef3e825525a70031f4e

commit a60c802709581c621f5bfef3e825525a70031f4e
Author: Khushal <khushalsagar@chromium.org>
Date: Tue Sep 25 00:20:06 2018

gpu: Fix deserialization error in ServiceFontManager.

We deserialize skia data while holding a lock, which we block on if
skia tries to delete a handle as this data is creating new entries in
the strike cache.

R=​ericrk@chromium.org

Bug: 887422, 887152
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: Id32cb071834186e7de0dac16902c452c3ccdb102
Reviewed-on: https://chromium-review.googlesource.com/1236455
Reviewed-by: Eric Karl <ericrk@chromium.org>
Commit-Queue: Khushal <khushalsagar@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#592847}(cherry picked from commit 84aec815ac3e956690aef30af88813d7053d49fe)
Reviewed-on: https://chromium-review.googlesource.com/1242153
Reviewed-by: Khushal <khushalsagar@chromium.org>
Cr-Commit-Position: refs/branch-heads/3538@{#634}
Cr-Branched-From: 79f7c91a2b2a2932cd447fa6f865cb6662fa8fa6-refs/heads/master@{#587811}
[modify] https://crrev.com/a60c802709581c621f5bfef3e825525a70031f4e/gpu/command_buffer/service/service_font_manager.cc

I started a pinpoint job (https://pinpoint-dot-chromeperf.appspot.com/job/16ae3797640000) on Android Nexus6 WebView, for loading.mobile/EnquiryIndianRail_3g. I hope that gives me a trace on which I can reproduce the loading.mobile failure and debug from there.