Link to the list of the builds:
===============================
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27S32_D32_constX_shaderproc%27%20AND%20product.name%3D%27Chrome_Android%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-property-selector,productversion:1000

Note:
=====
1. Crashes are seen spiked in M-65 since the first M-65 dev(65.0.3299.6), ranked as #10 renderer crash on the latest Android Dev(65.0.3316.0).

Considering below as the changelog:

https://chromium.googlesource.com/chromium/src/+log/64.0.3282.0..65.0.3299.0?pretty=fuller&n=10000

Unable to pin point as there are too many skia rolls in the above changelog and based on code search of files from stack trace unable to find the exact CL aligining with the regression range.

Assigning to reed@ for earlier work on Issue 783205 and helping in further investigation of these crashes.

Users experienced this crash on the following builds:

Android Dev 65.0.3316.0 -  0.93 CPM, 41 reports, 35 clients (signature S32_D32_constX_shaderproc)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

reed@ Ping! This issue is marked as RB-Stable for M65, Could you please take a look in to this issue?

Thanks!

The following revision refers to this bug:
  https://skia.googlesource.com/skia/+/1fe0bcc4f92bfe104bda57aa2ae5d0cb5b4f8c09

commit 1fe0bcc4f92bfe104bda57aa2ae5d0cb5b4f8c09
Author: Mike Reed <reed@google.com>
Date: Mon Jan 22 22:33:11 2018

check for huge paths

Bug:802976
Change-Id: Ibb5930442f75ca8483afc8dfa5869cac98573904
Reviewed-on: https://skia-review.googlesource.com/98440
Reviewed-by: Cary Clark <caryclark@google.com>
Reviewed-by: Florin Malita <fmalita@chromium.org>
Commit-Queue: Mike Reed <reed@google.com>

[modify] https://crrev.com/1fe0bcc4f92bfe104bda57aa2ae5d0cb5b4f8c09/tests/PathTest.cpp
[modify] https://crrev.com/1fe0bcc4f92bfe104bda57aa2ae5d0cb5b4f8c09/src/core/SkScan_AntiPath.cpp
[modify] https://crrev.com/1fe0bcc4f92bfe104bda57aa2ae5d0cb5b4f8c09/src/core/SkPaint.cpp

Users experienced this crash on the following builds:

Win Canary 66.0.3334.0 -  0.14 CPM, 6 reports, 6 clients (signature S32_D32_constX_shaderproc)
Android Dev 65.0.3325.16 -  0.89 CPM, 24 reports, 20 clients (signature S32_D32_constX_shaderproc)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

No crash instances seen post-66.0.3338.0 on Android OS.

Link to the list of builds:
------------------------
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27S32_D32_constX_shaderproc%27%20AND%20product.name%3D%27Chrome_Android%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-property-selector,+productname,productversion:1000

Note: 
1.No instances on Android Canary-66.0.3340.,dev-66.0.3335.4,beta-65.0.3325.38 also.
2.No instances on Windows OS

reed@, Could you please change the status to fixed, if no other pending work from your end.
Thanks..!

reed@  please let us know if the CL in comment#5 was merged to M65, If not can you please merge request. Since we are still seeing crashes on M65 Beta.

The fix is not in M65 yet.

Your change meets the bar and is auto-approved for M65. Please go ahead and merge the CL to branch 3325 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Pls merge your change to M65 branch 3325 ASAP so we can pick it up for next M65 Beta release. Thank you.

The following revision refers to this bug:
  https://skia.googlesource.com/skia/+/567bd716a9ae4051c8629cc4e8c047be69e84cd6

commit 567bd716a9ae4051c8629cc4e8c047be69e84cd6
Author: Mike Reed <reed@google.com>
Date: Wed Feb 07 22:17:10 2018

check for huge paths

Bug:802976
Change-Id: Ibb5930442f75ca8483afc8dfa5869cac98573904
Reviewed-on: https://skia-review.googlesource.com/98440
Reviewed-by: Cary Clark <caryclark@google.com>
Reviewed-by: Florin Malita <fmalita@chromium.org>
Commit-Queue: Mike Reed <reed@google.com>
(cherry picked from commit 1fe0bcc4f92bfe104bda57aa2ae5d0cb5b4f8c09)
Reviewed-on: https://skia-review.googlesource.com/105423
Reviewed-by: Mike Reed <reed@google.com>

[modify] https://crrev.com/567bd716a9ae4051c8629cc4e8c047be69e84cd6/tests/PathTest.cpp
[modify] https://crrev.com/567bd716a9ae4051c8629cc4e8c047be69e84cd6/src/core/SkScan_AntiPath.cpp
[modify] https://crrev.com/567bd716a9ae4051c8629cc4e8c047be69e84cd6/src/core/SkPaint.cpp

Seems like this is already merged to M65 at #12. If nothing is pending for M65, pls remove "Merge-Approved-65" label. Thank you.

Just to update:

S32_D32_constX_shaderproc

Chrome:
--------
No instances seen on stable,beta & dev. only 2 instances seen on latest Canary-66.0.3345.0 (live for 1 day)

66.0.3345.0	0.86%	2	-Canary

Android:
-------
No instances seen on stable/beta. Only single instance seen on dev-66.0.3343.0 & canary-66.0.3345.0.

Link to the list of builds:
----------------------------
https://goto.google.com/ocnji

reed@, Can we remove blocker label as per above crash data?

Thanks..!

M65 Stable promotion is coming VERY soon. Your bug is labelled as Stable ReleaseBlock, pls make sure to land the fix and request a merge  into the release branch ASAP. Thank you.

Isn't this already all done?  See comment #12 and (your) comment #13?

I guess this just needs to be marked fixed

We are still seeing these crash on latest Chrome Beta on Windows and Android, hence changing the status back to Assigned.

Please find the crashes history here : https://goto.google.com/xatmf

will review with Heather

Yes, it looks like this is still happening across all releases, so perhaps the fix + short term downward trend we saw was not the resolution we thought.  

It looks like the crash was not a regression and present in releases before 65 but experienced a slight uptick around the opening of this bug and a greater rise starting in early Feb.

reed@, have you been looking into that RBS?
It really needs an investigation before we release stable today

Still do not have a repro case, but I will start writing (random) drawRects to try to trigger this locally.

+vmpstr

Looking at https://crash.corp.google.com/browse?q=reportid=%27782dfb78227a8b99%27, Ben was able to heroically scrape some local variables from the minidump (attached).

The interesting bits:


            [+0x078] fState           : 0xee37bfdcb8 [Type: SkBitmapProcState *]
                [+0x000] fProvider        [Type: SkBitmapProvider]
                    [+0x000] fImage           : 0x29a98be1100 [Type: SkImage *]
                        [+0x008] fRefCnt          [Type: std::atomic<int>]
                            [<Raw View>]     [Type: std::atomic<int>]
                            [value]          : Unable to read memory at Address 0x29a98be1108
                        [+0x010] fWidth           : Unable to read memory at Address 0x29a98be1110
                        [+0x014] fHeight          : Unable to read memory at Address 0x29a98be1114
                        [+0x018] fUniqueID        : Unable to read memory at Address 0x29a98be1118
                    [+0x008] fDstColorSpace   : 0x0 [Type: SkColorSpace *]
                [+0x010] fPixmap          [Type: SkPixmap]
>>>>                [+0x000] fPixels          : 0x29a92c43000 [Type: void *]   <<<<


Note that fPixels points to the crashing address (0x0000029a92c43000).  We crash on read, during a drawRect() call with an image shader.

Looking at SkBitmapProcState::fPixmap, it is the *source* bitmap for bilerp in the image shader.

It appears that sometimes, by the time we reach the bilerp shader stage, these bitmaps are pointing to invalid pixel addresses.

My guess is this is a race where (unlocked?) bitmaps are discarded prematurely either from Skia's internal caches or from CC's image cache.

@vmpstr: does Chrome ever request filter quality > kLow during rasterization?  I seem to recall we no longer do that -- in which case I suspect this might be a CC/IDC locking issue.

Deleted: windbg.txt 30.9 KB

windbg.txt 30.9 KB View Download

Any update here?

Users experienced this crash on the following builds:

Win Canary 67.0.3364.0 -  0.14 CPM, 3 reports, 3 clients (signature S32_D32_constX_shaderproc)
Android Dev 66.0.3356.0 -  2.34 CPM, 110 reports, 75 clients (signature S32_D32_constX_shaderproc)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Still no update here reed@ Are you still working on it. Please update

I have been pinging reed@ over chat as well and there is still no response yet. Hoping they get to tell us the status on this blocker soon.

Please see my comment #21, though there has been an uptick I don't think this is a true regression or that it should be RBS.

Skia team is investigating, need feedback from vmpstr/cc team as well.

Any update here?

Just to update:

S32_D32_constX_shaderproc

Seeing only 2 instances from 2 clients so far on Canary-67.0.3368.0(live for 1 day) on Windows OS.
Chrome:
------
67.0.3368.0	0.06%	2 - Canary
66.0.3359.22	0.50%	16  - Dev
65.0.3325.125	0.93%	30  -Beta

Android:
-------
67.0.3365.0	0.11%	10 --Canary (live for 4 days)
66.0.3359.14	0.49%	44 -- Dev
65.0.3325.144	1.59%	144 - Beta

Link to the list of builds:
-------------------------
https://goto.google.com/xatmf

Thanks..!

assigning to vmpster as per #26. Happy to relook after he has done an assessment.

Hmm, we should never be requesting > low filter quality. FWIW, this seems to be a software case.

-> khushalsagar@ for triage

Users experienced this crash on the following builds:

Android Dev 66.0.3359.14 -  1.92 CPM, 54 reports, 42 clients (signature S32_D32_constX_shaderproc)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Re #26, yeah we only request up-to kLow quality during raster, but the image remains locked in the cc cache for the duration of the draw call that uses it. So unless the operation that uses the image was being deferred in skia until after the drawRect returns (doesn't look like it from the stack), I wouldn't expect the image data to be prematurely unlocked.

Just to Update:

There are no crashes observed for this magic signature Since : 67.0.3368.0/.1

Link to the list of builds: 
https://crash.corp.google.com/browse?q=expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27S32_D32_constX_shaderproc%27#-property-selector,productversion:1000,-magicsignature:50,-magicsignature2:50,-stablesignature:50,-magicsignaturesorted:50

did we fix this in M67 then? Or the signature just changed.

Sending back to reed@.

I made a change recently with decoding of picture shaders in cc which I suspected may affect this if it were an image locking issue, but that rolled out in 67.0.3369.0.

reed@ can you check why we are not seeing any crash reports in M67?

Could the disappearance of crashes be due to DCHECKs added to paint in 67? (https://chromium.googlesource.com/chromium/src.git/+/bb5d0d6bdb6e29327bce812d1dcd61782001f539)

We haven't made any changes in the related Skia code here- this is still a mystery that seems to be coming, or going away, from upstack...

Re #c44: Unlikely. The CL didn't change anything in output operations.

Users experienced this crash on the following builds:

Win Dev 67.0.3371.0 -  0.10 CPM, 13 reports, 11 clients (signature S32_D32_constX_shaderproc)
Android Dev 66.0.3359.30 -  1.47 CPM, 40 reports, 34 clients (signature S32_D32_constX_shaderproc)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Just to update:

S32_D32_constX_shaderproc

Windows:
------

67.0.3371.0	0.36%	34 (From 29 clients)  -Dev
66.0.3359.33	0.44%	41 (From 35 clients)   -Beta
65.0.3325.181	0.50%	47 (From 47 clients)   -Stable

No instances on latest Canary-67.0.3376.1

Android:
--------
66.0.3359.30	0.49%	266 (From 179 clients) -Dev/beta
65.0.3325.109	93.75%	51084 (From 38994) - Stable

No instances seen on latest canary-67.0.3376.0

Link to the list of builds:
---------------------------
https://goto.google.com/zewsd

Thanks..!

Could some one from cc'ed dev please take a look & update the thread.
Thanks..!

Since this is a release blocker, We need someone to own this bug. Can anyone from the skia team own this?

enne@ can you please triage this? This really needs an owner as it's RBS, and it's not clear from the above comments if the bug is in skia or cc.

This smells similar to  issue 824518 , and I agree that it seems like a cc image decode cache issue.

The only changes in m65 were refactorings from vmpstr who unfortunately is no longer working on this:
https://chromium-review.googlesource.com/848497
https://chromium-review.googlesource.com/783636

I'll take a quick look and see if I can find anything obvious and maybe ericrk or khushalsagar who have more experience in SoftwareImageDecodeCache can take a look too.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fb48059ccdfe83de78d14b889cf734668ded071b

commit fb48059ccdfe83de78d14b889cf734668ded071b
Author: Khushal <khushalsagar@chromium.org>
Date: Mon Mar 26 04:10:20 2018

cc: Add some diagnostic CHECKs for sw image cache crashes.

Change a few DCHECKs to CHECKs to validate if some crashes are
resulting from using unlocked images, or incorrectly unlocking them
earlier.

R=enne@chromium.org, ericrk@chromium.org

Bug:  802976 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: Ibe0b9ef6ac52e6fafc9fd2670e20a9eefac17234
Reviewed-on: https://chromium-review.googlesource.com/979160
Commit-Queue: Khushal <khushalsagar@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#545724}
[modify] https://crrev.com/fb48059ccdfe83de78d14b889cf734668ded071b/cc/tiles/software_image_decode_cache.cc
[modify] https://crrev.com/fb48059ccdfe83de78d14b889cf734668ded071b/cc/tiles/software_image_decode_cache_utils.h

I strongly suspect this is a dupe of 825772.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/29ff29b8524efe68eac94675ae5b5d27a4e5f76d

commit 29ff29b8524efe68eac94675ae5b5d27a4e5f76d
Author: Khushal <khushalsagar@chromium.org>
Date: Tue Mar 27 17:14:48 2018

Revert "cc: Add some diagnostic CHECKs for sw image cache crashes."

This reverts commit fb48059ccdfe83de78d14b889cf734668ded071b.

Original change's description:
> cc: Add some diagnostic CHECKs for sw image cache crashes.
> 
> Change a few DCHECKs to CHECKs to validate if some crashes are
> resulting from using unlocked images, or incorrectly unlocking them
> earlier.
> 
> R=​enne@chromium.org, ericrk@chromium.org
> 
> Bug:  802976 
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
> Change-Id: Ibe0b9ef6ac52e6fafc9fd2670e20a9eefac17234
> Reviewed-on: https://chromium-review.googlesource.com/979160
> Commit-Queue: Khushal <khushalsagar@chromium.org>
> Reviewed-by: enne <enne@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#545724}

TBR=enne@chromium.org,khushalsagar@chromium.org,ericrk@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  802976 
Change-Id: I152d1354ba7f43b77ae95d6edcf9f579109f6d91
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Reviewed-on: https://chromium-review.googlesource.com/980721
Reviewed-by: Khushal <khushalsagar@chromium.org>
Cr-Commit-Position: refs/branch-heads/3381@{#1}
Cr-Branched-From: ae7ce2710f11dd78c2f787b888d2cc19b912bf49-refs/heads/master@{#545918}
[modify] https://crrev.com/29ff29b8524efe68eac94675ae5b5d27a4e5f76d/cc/tiles/software_image_decode_cache.cc
[modify] https://crrev.com/29ff29b8524efe68eac94675ae5b5d27a4e5f76d/cc/tiles/software_image_decode_cache_utils.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c308eef6bbbe70f8a64b37d5b0e87fb41d532020

commit c308eef6bbbe70f8a64b37d5b0e87fb41d532020
Author: Khushal <khushalsagar@google.com>
Date: Thu Mar 29 01:54:43 2018

cc: Fix software image cache handling for empty target size images.

The cache keys generated for an image with an empty target size can have
collisions. The ProcessingType for these cases is set to kOriginal,
indicating an original sized decode. If the cache is already populated
with an original sized decode, this will cause us to use the unlocked
memory for that entry.

Fix this by using a kSubrectAndScale Processing type for this case, so
the key comparison is accurate.

R=enne@chromium.org

Bug:  825772 ,  802976 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I26b40673975b64c6fb75a8363d925273407ed3f0
Reviewed-on: https://chromium-review.googlesource.com/981115
Commit-Queue: Khushal <khushalsagar@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#546697}
[modify] https://crrev.com/c308eef6bbbe70f8a64b37d5b0e87fb41d532020/cc/tiles/software_image_decode_cache.cc
[modify] https://crrev.com/c308eef6bbbe70f8a64b37d5b0e87fb41d532020/cc/tiles/software_image_decode_cache_unittest.cc
[modify] https://crrev.com/c308eef6bbbe70f8a64b37d5b0e87fb41d532020/cc/tiles/software_image_decode_cache_utils.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ac7ee269798b7add84a822ba79be1620e0a7b132

commit ac7ee269798b7add84a822ba79be1620e0a7b132
Author: Khushal <khushalsagar@google.com>
Date: Mon Apr 02 17:48:33 2018

cc: Fix software image cache handling for empty target size images.

The cache keys generated for an image with an empty target size can have
collisions. The ProcessingType for these cases is set to kOriginal,
indicating an original sized decode. If the cache is already populated
with an original sized decode, this will cause us to use the unlocked
memory for that entry.

Fix this by using a kSubrectAndScale Processing type for this case, so
the key comparison is accurate.

R=​enne@chromium.org

Bug:  825772 ,  802976 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I26b40673975b64c6fb75a8363d925273407ed3f0
Reviewed-on: https://chromium-review.googlesource.com/981115
Commit-Queue: Khushal <khushalsagar@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#546697}(cherry picked from commit c308eef6bbbe70f8a64b37d5b0e87fb41d532020)
Reviewed-on: https://chromium-review.googlesource.com/990272
Reviewed-by: Khushal <khushalsagar@chromium.org>
Cr-Commit-Position: refs/branch-heads/3359@{#527}
Cr-Branched-From: 66afc5e5d10127546cc4b98b9117aff588b5e66b-refs/heads/master@{#540276}
[modify] https://crrev.com/ac7ee269798b7add84a822ba79be1620e0a7b132/cc/tiles/software_image_decode_cache.cc
[modify] https://crrev.com/ac7ee269798b7add84a822ba79be1620e0a7b132/cc/tiles/software_image_decode_cache_unittest.cc
[modify] https://crrev.com/ac7ee269798b7add84a822ba79be1620e0a7b132/cc/tiles/software_image_decode_cache_utils.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c45788b380926947154ed18831522386a20ec731

commit c45788b380926947154ed18831522386a20ec731
Author: Khushal <khushalsagar@chromium.org>
Date: Mon Apr 02 19:08:04 2018

Revert "cc: Add some diagnostic CHECKs for sw image cache crashes."

This reverts commit fb48059ccdfe83de78d14b889cf734668ded071b.

Original change's description:
> cc: Add some diagnostic CHECKs for sw image cache crashes.
>
> Change a few DCHECKs to CHECKs to validate if some crashes are
> resulting from using unlocked images, or incorrectly unlocking them
> earlier.
>
> R=​enne@chromium.org, ericrk@chromium.org
>
> Bug:  802976 
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
> Change-Id: Ibe0b9ef6ac52e6fafc9fd2670e20a9eefac17234
> Reviewed-on: https://chromium-review.googlesource.com/979160
> Commit-Queue: Khushal <khushalsagar@chromium.org>
> Reviewed-by: enne <enne@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#545724}

TBR=enne@chromium.org,khushalsagar@chromium.org,ericrk@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 826208
Change-Id: I152d1354ba7f43b77ae95d6edcf9f579109f6d91
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Reviewed-on: https://chromium-review.googlesource.com/980720
Commit-Queue: Khushal <khushalsagar@chromium.org>
Reviewed-by: Khushal <khushalsagar@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#547482}
[modify] https://crrev.com/c45788b380926947154ed18831522386a20ec731/cc/tiles/software_image_decode_cache.cc
[modify] https://crrev.com/c45788b380926947154ed18831522386a20ec731/cc/tiles/software_image_decode_cache_utils.h