The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c81425e082608177faabe49198193c0a4252ada9

commit c81425e082608177faabe49198193c0a4252ada9
Author: boliu <boliu@chromium.org>
Date: Fri Mar 25 23:06:48 2016

gpu: disable_program_cache for adreno 530

BUG= 598060 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.win:win_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel

Review URL: https://codereview.chromium.org/1837563003

Cr-Commit-Position: refs/heads/master@{#383394}

[modify] https://crrev.com/c81425e082608177faabe49198193c0a4252ada9/gpu/config/gpu_driver_bug_list_json.cc

Your change meets the bar and is auto-approved for M50 (branch: 2661)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a63736136932497ff6447408674e0a582f1d8ec5

commit a63736136932497ff6447408674e0a582f1d8ec5
Author: Bo Liu <boliu@chromium.org>
Date: Sat Mar 26 01:07:51 2016

[Merge M50] gpu: disable_program_cache for adreno 530

BUG= 598060 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.win:win_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel

Review URL: https://codereview.chromium.org/1837563003

Cr-Commit-Position: refs/heads/master@{#383394}
(cherry picked from commit c81425e082608177faabe49198193c0a4252ada9)

Review URL: https://codereview.chromium.org/1838633003 .

Cr-Commit-Position: refs/branch-heads/2661@{#395}
Cr-Branched-From: ef6f6ae5e4c96622286b563658d5cd62a6cf1197-refs/heads/master@{#378081}

[modify] https://crrev.com/a63736136932497ff6447408674e0a582f1d8ec5/gpu/config/gpu_driver_bug_list_json.cc

Looking at heroqltetmo

Crashes started in 49 and 47 on the 20th of March.

Also that's the day that MMB29M crashes of any kind started on that device. Maybe there was an OTA? Could be that not only is the QC binary format ABI-dependent, but they have made no attempt to keep it stable across driver versions?

Just checked b/22046630 and it looks like the build fingerprint is used to check that the shader cache is valid.

Samsung started shipping s7 on march 20th?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/30bcb92ee59a6778c4025081d7da82d2af8cccc2

commit 30bcb92ee59a6778c4025081d7da82d2af8cccc2
Author: ericrk <ericrk@chromium.org>
Date: Thu Aug 25 18:47:33 2016

Re-enable program memory cache for Adreno 4/5xx

The program cache was disabled on Adreno 4/5xx due to crashes.

I'm hoping that this issue was caused by invalidations/issues with the
on-disk program cache, possibly with driver updates or other issues.

To test this, I'm re-enabling the in-memory portion of the cache
only. The cached objects will not outlive the GPU process.

I've also confirmed that the related Adreno 3xx bug, crbug.com/510673,
does not impact the Adreno 400 (and I'm guessing 500, but will track
down a device to double check).

We should watch crash rates for gfx::GLApiBase::glProgramBinaryFn to
make sure this change doesn't re-introduce these crashes.

BUG= 598060 , 486117
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel

Review-Url: https://codereview.chromium.org/2267163003
Cr-Commit-Position: refs/heads/master@{#414490}

[modify] https://crrev.com/30bcb92ee59a6778c4025081d7da82d2af8cccc2/gpu/command_buffer/service/in_process_command_buffer.cc
[modify] https://crrev.com/30bcb92ee59a6778c4025081d7da82d2af8cccc2/gpu/config/gpu_driver_bug_list_json.cc
[modify] https://crrev.com/30bcb92ee59a6778c4025081d7da82d2af8cccc2/gpu/config/gpu_driver_bug_workaround_type.h
[modify] https://crrev.com/30bcb92ee59a6778c4025081d7da82d2af8cccc2/gpu/ipc/service/gpu_channel_manager.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/db08f5848018102d93afbb554009292213a95c7c

commit db08f5848018102d93afbb554009292213a95c7c
Author: ericrk <ericrk@chromium.org>
Date: Sat Feb 04 00:37:05 2017

Re-enable on disk program cache for Adreno 4/5xx

Now that we've added a checksum of the cached data, I'd like to try re-
enabling the on-disk program cache for the blacklisted devices which
seemed to be experiencing crash-on-load behavior.

We should be on the lookout for an increase in the crash callstacks
from the linked bugs.

R=vmiura@chromium.org
BUG= 598060 ,486117
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2674803003
Cr-Commit-Position: refs/heads/master@{#448119}

[modify] https://crrev.com/db08f5848018102d93afbb554009292213a95c7c/gpu/config/gpu_driver_bug_list_json.cc

 Issue 736372  has been merged into this issue.

This has jumped a lot in the last 5 days (>5x)

https://crash.corp.google.com/browse?q=product.name%3D%27AndroidWebView%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27gpu%3A%3Agles2%3A%3AProgram%3A%3ALink%27%20AND%20upload_info.minidump_bytes%3C%3D1024&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-property-selector,samplereports:5,-clientid,+day

Given that there was speculation before that the cache might not be binary compatible across versions (in comment 7) - have any of the devices where this is crashing just gotten an OTA by any chance? :)

Also we should reopen this to keep track until we decide.

Issue 735151 has been merged into this issue.

Issue 734678 has been merged into this issue.

My console has exploded with these adreno crashes since my release one week ago. I can't reproduce and would be extremely grateful for any information along the lines of:

* What might be causing the crash
* What steps I can take that would be most likely to reproduce
* Any tips for mitigating

I'm thinking an analytics library that I recently upgraded might not be properly interacting with my webviews. That's what I'll be investigating today. Here are the top 4 adreno crashes in my console. They make up 80% of my total crashes: 

2949 reports / 526 users
backtrace:
  native: pc 0000000000af3860  /system/vendor/lib64/libllvm-glnext.so
  native: pc 0000000000a4a6d0  /system/vendor/lib64/libllvm-glnext.so
  native: pc 0000000000b0c760  /system/vendor/lib64/libllvm-glnext.so (_Z21QGLCLoadProgramBinaryPvS_mP23QGLC_LINKPROGRAM_RESULT+96)
  native: pc 000000000019bdb8  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN17EsxShaderCompiler21LoadProgramBinaryBlobEP10EsxContextP10EsxProgramPKvmP10EsxInfoLog+256)

911 reports / 155 users
backtrace:
  native: pc 0000000000997c50  /system/vendor/lib64/libllvm-glnext.so
  native: pc 00000000009982d8  /system/vendor/lib64/libllvm-glnext.so
  native: pc 000000000090b560  /system/vendor/lib64/libllvm-glnext.so
  native: pc 00000000009a90e4  /system/vendor/lib64/libllvm-glnext.so (_Z21QGLCLoadProgramBinaryPvS_mP23QGLC_LINKPROGRAM_RESULT+84)
  native: pc 000000000015a718  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN17EsxShaderCompiler21LoadProgramBinaryBlobEP10EsxContextP10EsxProgramPKvmP10EsxInfoLog+184)
  native: pc 0000000000141bac  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxProgram17LoadProgramBinaryEP10EsxContextjPKvi+172)
  native: pc 00000000000ddde8  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxContext15GlProgramBinaryEjjPKvi+136)
  native: pc 000000000011fdd0  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN21EsxGlApiParamValidate15GlProgramBinaryEP11EsxDispatchjjPKvi+592)
  native: pc 00000000000c35c4  /system/vendor/lib64/egl/libGLESv2_adreno.so (glProgramBinary+68)
  native: pc 0000000000d2173c  /data/app/com.android.chrome-2/base.apk

249 reports / 53 users
backtrace:
  native: pc 0000000000cef734  /system/vendor/lib64/libllvm-glnext.so (_ZN13ShaderObjects20adjustSymbolPointersEPcP11GLSL_SYMBOL+80)
  native: pc 0000000000cefddc  /system/vendor/lib64/libllvm-glnext.so (_ZN13ShaderObjects17loadProgramBinaryEP15CompilerContextPvmP23QGLC_LINKPROGRAM_RESULT+812)
  native: pc 0000000000c71e3c  /system/vendor/lib64/libllvm-glnext.so (_ZN15CompilerContext17loadProgramBinaryEPvmP23QGLC_LINKPROGRAM_RESULT+176)
  native: pc 0000000000d05868  /system/vendor/lib64/libllvm-glnext.so (_Z21QGLCLoadProgramBinaryPvS_mP23QGLC_LINKPROGRAM_RESULT+68)
  native: pc 0000000000156b70  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN17EsxShaderCompiler21LoadProgramBinaryBlobEP10EsxContextP10EsxProgramPKvmP10EsxInfoLog+144)
  native: pc 000000000013f164  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxProgram17LoadProgramBinaryEP10EsxContextjPKvi+164)
  native: pc 00000000000e05a8  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxContext15GlProgramBinaryEjjPKvi+136)
  native: pc 00000000000c49e4  /system/vendor/lib64/egl/libGLESv2_adreno.so (glProgramBinary+68)
  native: pc 0000000000d2173c  /data/app/com.google.android.webview-2/lib/arm64/libwebviewchromium.so

121 reports / 14 users
 backtrace:
  native: pc 0000000000924190  /system/vendor/lib64/libllvm-glnext.so
  native: pc 00000000009247d8  /system/vendor/lib64/libllvm-glnext.so
  native: pc 00000000008a5610  /system/vendor/lib64/libllvm-glnext.so
  native: pc 00000000009331c0  /system/vendor/lib64/libllvm-glnext.so (_Z21QGLCLoadProgramBinaryPvS_mP23QGLC_LINKPROGRAM_RESULT+64)
  native: pc 0000000000175bd0  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN17EsxShaderCompiler21LoadProgramBinaryBlobEP10EsxContextP10EsxProgramPKvmP10EsxInfoLog+208)
  native: pc 000000000015cea4  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxProgram17LoadProgramBinaryEP10EsxContextjPKvi+164)
  native: pc 00000000000fb428  /system/vendor/lib64/egl/libGLESv2_adreno.so (_ZN10EsxContext15GlProgramBinaryEjjPKvi+136)
  native: pc 00000000000e0984  /system/vendor/lib64/egl/libGLESv2_adreno.so (glProgramBinary+68)
  native: pc 0000000000d2173c  /data/app/com.google.android.webview-1/lib/arm64/libwebviewchromium.so

This is a graphics driver bug on the device; it's probably not caused by anything specific your app is doing, and you definitely won't be able to reproduce it unless you have the exact same graphics driver version and hardware as the users seeing the crash (and maybe then even not, if it's specific to a particular situation like OTAs as I speculated above). There's probably nothing you can to do to prevent it other than not using hardware rendering.

Toby/Bo, I think we should just re-blacklist these at this point (i.e. revert the CL from comment 11). WDYT?

Thanks for the quick response Torne.

> it's probably not caused by anything specific your app is doing

I'm hesitant to accept this idea, because I went from 0 to thousands of crashes the moment I released my newest version. Tobias mentioned this number has grown 5x in the last few days. Any speculation on the timing of this surge?

Eric, it seems like https://codereview.chromium.org/2744363002 doesn't account for cases where the GPU process isn't separated (i.e. WebView and Clank on svelte). Do we have a way to conditionally enable the blacklist when this is the case? Otherwise, do you have any other suggestions?

re #20: You've just been unlucky. The time period where we had this surge is the time period when we released a new version of WebView to 5% of android devices, and the new version contains a change which re-enables the code that provokes the graphics driver bug as Toby notes in #21.

Re #21, I'll re-disable the shader cache for non-separate GPU proces devices today. Thanks for tracking this down.

Do we need to merge this to stable? In this case we should just revert https://codereview.chromium.org/2743423013, then do a safer fix for beta/canary. If we're OK with this going in to Beta, I'll do the fix described in #23 directly.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/62cdea2d511f19967ffb31be31f078350fade283

commit 62cdea2d511f19967ffb31be31f078350fade283
Author: Eric Karl <ericrk@chromium.org>
Date: Tue Jun 27 19:54:22 2017

Revert "Re-enable shader disk cache on Adreno 4/5xx"

This reverts commit 42267f803ddaadca1d2d716a478ea5f8dd9f169f.

This was breaking in-process-gpu cases.

Bug:  598060 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I26476555b11292f228957465eb8b056947c47d37
Reviewed-on: https://chromium-review.googlesource.com/549966
Reviewed-by: Alexandre Elias <aelias@chromium.org>
Commit-Queue: Eric Karl <ericrk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#482716}
[modify] https://crrev.com/62cdea2d511f19967ffb31be31f078350fade283/gpu/config/gpu_driver_bug_list.json

+amineer@

The issue in this bug accounts for ~5% of WebView crashes in 59.0.3071.92 and 0.7% of WebView crashes in 59.0.3071.117. (https://crash.corp.google.com/browse?q=product.name%3D%27AndroidWebView%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27gpu%3A%3Agles2%3A%3AProgram%3A%3ALink%27%20AND%20upload_info.minidump_bytes%3C%3D1024&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-property-selector,samplereports:5,-clientid,+day)

If the rate stays below 1% (as it is in .117), this doesn't seem like it needs a rush fix, but wanted to make you aware of the issue and see how you thought we should proceed.

The "fix" (at least for beta and/or stable) is to just re-disable the feature in question (#25). 

One additional note on this crash (which makes it a bit worse) - if a user hits it, they will likely continue to hit it in subsequent loads. (although, as I don't believe we've ever found a local repro, I'm just basing this on the crash stats and my understanding of the bug).

Merge approved for M59 branch 3071 and M60 branch 3112.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/62f3a54b38649d7004b5c6acda61378bc38a184c

commit 62f3a54b38649d7004b5c6acda61378bc38a184c
Author: Eric Karl <ericrk@chromium.org>
Date: Tue Jun 27 22:19:22 2017

Revert "Re-enable shader disk cache on Adreno 4/5xx"

This reverts commit 42267f803ddaadca1d2d716a478ea5f8dd9f169f.

This was breaking in-process-gpu cases.

Bug:  598060 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I26476555b11292f228957465eb8b056947c47d37
Reviewed-on: https://chromium-review.googlesource.com/549966
Reviewed-by: Alexandre Elias <aelias@chromium.org>
Commit-Queue: Eric Karl <ericrk@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#482716}
Review-Url: https://codereview.chromium.org/2962543004 .
Cr-Commit-Position: refs/branch-heads/3071@{#829}
Cr-Branched-From: a106f0abbf69dad349d4aaf4bcc4f5d376dd2377-refs/heads/master@{#464641}

[modify] https://crrev.com/62f3a54b38649d7004b5c6acda61378bc38a184c/gpu/config/gpu_driver_bug_list.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d3224f75d745815157778db0845cdba887d9bf22

commit d3224f75d745815157778db0845cdba887d9bf22
Author: Eric Karl <ericrk@chromium.org>
Date: Tue Jun 27 22:21:02 2017

Revert "Re-enable shader disk cache on Adreno 4/5xx"

This reverts commit 42267f803ddaadca1d2d716a478ea5f8dd9f169f.

This was breaking in-process-gpu cases.

Bug:  598060 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I26476555b11292f228957465eb8b056947c47d37
Reviewed-on: https://chromium-review.googlesource.com/549966
Reviewed-by: Alexandre Elias <aelias@chromium.org>
Commit-Queue: Eric Karl <ericrk@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#482716}
Review-Url: https://codereview.chromium.org/2954863005 .
Cr-Commit-Position: refs/branch-heads/3112@{#484}
Cr-Branched-From: b6460e24cf59f429d69de255538d0fc7a425ccf9-refs/heads/master@{#474897}

[modify] https://crrev.com/d3224f75d745815157778db0845cdba887d9bf22/gpu/config/gpu_driver_bug_list.json

Verified bug fix went on latest M59 build on Adreno 4XX and Adreno 5XX devices.

Fix verified on the latest M60 build on Adreno 4XX and Adreno 5XX devices.

To be more clear on comments #31 and #32 :
we were not able to repro the crash on previous 59.0.3071.117.
We did the sanity testing on Webview apps on Adreno 4XX and Adreno 5XX devices with builds:59.0.3071.122 and 60.0.3112.50 and didnt see any crashes.

We also verified in chrome "about:gpu"" has "disable_program_disk_cache" entry under "Driver Bug Workarounds".

No longer seeing this crash after 59.0.3071.117 and 60.0.3112.43 so far:

https://crash.corp.google.com/browse?q=product.name%3D%27AndroidWebView%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27gpu%3A%3Agles2%3A%3AProgram%3A%3ALink%27%20OMIT%20RECORD%20IF%20SUM(CrashedStackTrace.StackFrame.FunctionName%3D%27gl%3A%3AGLApiBase%3A%3AglProgramBinaryFn(unsigned%20int%2C%20unsigned%20int%2C%20void%20const*%2C%20int)%27)%20%3D%200&ignore_case=false&enable_rewrite=true&omit_field_name=CrashedStackTrace.StackFrame.FunctionName&omit_field_value=gl%3A%3AGLApiBase%3A%3AglProgramBinaryFn(unsigned%20int%2C%20unsigned%20int%2C%20void%20const*%2C%20int)&omit_field_opt=%3D

Users experienced this crash on the following builds:

Android Dev 61.0.3142.0 -  0.34 CPM, 7 reports, 6 clients (signature LoadLinkedProgram)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

There are other crashes with LoadLinkedProgram in the magic signature but without glProgramBinaryFn in stack, which is why Fracas tagged it.  Due to the missing namespace in function names bug, it could show up as a new crash but it's been around for a long time.  For example, this crash in Chrome_Android:

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Android%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%20like%20%27%25LoadLinkedProgram%27&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=#samplereports:5,productversion:1000,magicsignature:50

Issue 745912 has been merged into this issue.

ericrk@: is there further work required here?

This is fixed.