Tried testing the issue on reported chrome version #70.0.3508.0 using Mac OS 10.13.5, by following the below steps.

Steps:
====
1.Launched chrome.
2.Opened multiple tabs, did not observe the black boxes on the close button on a tab, the bookmark star, and the back button.

Attached screencast for reference.
@reporter : Could you please review the screencast and let us know if anything is being missed from our end.Request you to retry this issue with fresh profile without any extensions/apps or reset all the flags and let us know if issue still persists.

Thanks.!

Deleted: 870317.mp4 2.2 MB

	870317.mp4 2.2 MB View Download

over to elly for macviews triage

ccameron@:

We're sort of in the dark here. Can you think of a way this might happen? Do you know where the black color might be coming from?

Feel free to kick this back to me for reappraisal after you reply.

I wish I could reproduce it, but the few times it occurred, Chrome had been running for a long time.
I haven't been able to use Canary for a few days, as it was crashing for me ( Issue 871058 ), but it is working again.

 Issue 871782  has been merged into this issue.

Just some ideas on how it could be debugged:

I would suggest looking at how these icons are drawn - e.g. is it through gfx::Image or some other abstraction and then audit the code for any failure modes?

The other interesting thing is they seem to fix themselves after a bit of time - so do we manually try to reload them or is there some async stuff going on or?

+khushalsagar -- looks like image caching issue perhaps? Any ideas of how to track this down?

Inconveniently, we can't get useful data from about:gpu's error logs because of issue 864905.

Are these buttons drawn by the UI using images in a display list? The only cases I can think of where we might mess up handling them during raster is:

1) PaintOp::QuickRejectDraw, incorrectly assuming the op to draw them is being clipped. But that would be very consistent.

2) Unpinning the texture from the GPU image cache before its drawn, I'd expect that to show as black. But all of this is carefully managed in the cache and I would've expected it show up for the renderer as well if there was a bug there.

Async image decoding may draw without an image but that's turned off for the UI. +ericrk@ in case he has any other ideas.

I have observed similar artifacts on a system with an Nvidea GPU - with hardware acceleration enabled - just before Chrome crashes - or freezes the system.

Re #12, that sounds like a Out of Memory situation, I don't think its the same as this bug.

Do you make anything of this spew?

Full context: https://paste.googleplex.com/6313320172748800

[42623:775:0730/085526.121362:ERROR:gles2_cmd_decoder.cc(20112)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glLockDiscardableTextureCHROMIUM: Texture ID not initialized
[42623:775:0730/085526.121668:ERROR:gles2_cmd_decoder.cc(6203)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_OPERATION : glBindTexture: id not generated by glGenTextures
[42623:775:0730/085526.121708:ERROR:gles2_cmd_decoder.cc(9350)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glTexParameteri: unknown texture
[42623:775:0730/085526.121754:ERROR:gles2_cmd_decoder.cc(9350)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glTexParameteri: unknown texture
[42623:775:0730/085526.121776:ERROR:gles2_cmd_decoder.cc(9350)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glTexParameteri: unknown texture
[42623:775:0730/085526.121796:ERROR:gles2_cmd_decoder.cc(9350)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glTexParameteri: unknown texture
[42623:775:0730/085526.121820:ERROR:gles2_cmd_decoder.cc(10176)] : [.BrowserWorker.GpuRasterization]RENDER WARNING: there is no texture bound to the unit 0
[42623:775:0730/085526.121856:ERROR:gles2_cmd_decoder.cc(20094)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glUnlockDiscardableTextureCHROMIUM: Texture ID not initialized
[42623:775:0730/085526.170586:ERROR:gles2_cmd_decoder.cc(20112)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glLockDiscardableTextureCHROMIUM: Texture ID not initialized

Thanks! That's definitely related, discardable textures are what the GPU cache uses to lock/unlock image uploads.

[42623:775:0730/085526.121362:ERROR:gles2_cmd_decoder.cc(20112)] : [.BrowserWorker.GpuRasterization]GL ERROR :GL_INVALID_VALUE : glLockDiscardableTextureCHROMIUM: Texture ID not initialized

^ This log makes it look like the browser assumed the texture is locked but it isn't on the GPU. Or  Assigning to ericrk@ for GPU discardable. I'll try digging some too.

 Issue 868994  has been merged into this issue.

We've been hitting this quite a bit. This needs to be fixed.

ericrk@ is this issue in the M69 branch?

Re #18, this should manifest if the UI is using GPU raster. And I think that's the case with mac-views which is targeting 69?

Re #19, yes, macOS is using GPU raster for 69.

 Issue 872819  has been merged into this issue.

In most logs where we're hitting this, I also see errors with CALayers having invalid textures and in each case its preceded by an ipc warning:

[30100:775:0807/161308.041575:WARNING:ipc_message_attachment_set.cc(49)] : MessageAttachmentSet destroyed with unconsumed attachments: 0/1
[30100:775:0807/161945.834742:ERROR:gles2_cmd_decoder.cc(17799)] : [.BrowserCompositor-0x7fdd56a81e00]GL ERROR :GL_INVALID_OPERATION : glCreateAndConsumeTextureCHROMIUM: invalid mailbox name
[30100:775:0807/161945.834953:ERROR:gles2_cmd_decoder.cc(12559)] : [.BrowserCompositor-0x7fdd56a81e00]GL ERROR :GL_INVALID_VALUE : glScheduleCALayerCHROMIUM: unsupported texture format

+rockot
"MessageAttachmentSet destroyed with unconsumed attachments". What exactly does this mean?

It means an IPC message is discarded without any receiving code taking ownership of one or more attached platform objects. On Mac, this means file descriptor(s) and/or Mach port(s).

The glCreateAndConsumeTextureCHROMIUM and glScheduleCALayerCHROMIUM errors are a red herring -- see issue 864905.

Adding some more folks for ideas.

The discardable texture logs can come from 2 things. Either we didn't initialize the texture in the discardable system before using it, or there is an error on the ref-counting where the browser assumes the texture is locked but it has actually been purged on the GPU. The second slightly more likely since the first would have been fairly consistent. Talking to robliao@ offline, the only thing that fixed it was an intentional gpu process crash in which case the browser re-creates everything.

What's puzzling for me is that this code has been used in the renderer for quite sometime and we are only seeing this come up with the browser switching to GPU raster. I can't think of anything different in the browser -> GPU connection for the worker context that could be related.

My only offhand thought is that maybe we've exhausted shared memory somehow, but in that case it seems like we'd fail to find the buffer when we created the gpu discardable and would then restart the gpu process because of the decoding error.  So, I'm a bit lost.

Were you able to repro this, ccameron?

I haven't seen a reliable repro -- it has happened once or twice.

Also, it only affects one given ui::Compositor -- if I open a new window, that window works fine. It seems sticky on the ui::Compositor.

Attaching a video -- see the "Chrome GPU Slack" icon is broken. Note that
- no other icons are broken
- the same icon works fine in a new window
- when the bookmark bar is hidden/shown the broken icon persists
- lots of "glLockDiscardableTextureCHROMIUM: Texture ID not initialized" errors

This behavior seems consistent with
- deleting a GL texture in the GpuImageDecodeCache
- not realizing that it has been deleted, and continuing to use it

Just to add a little anec-data ... I do a lot of plug/unplug to a Cinema display, and move Chrome between spaces (for personal / work profile separation). My gut tells me the adding/removing of displays exacerbates this problem.

I've noticed that this does seem to appear when un-plugging a display. Is your external display non-retina?

Maybe this has to do with switching DPI (and having different resources for different DPIs)?

Next time this happens, see if you can plug back in to the last-size monitor you were using, and see if the black boxes go away.

Correct. Internal display is retina. External display is older Cinema, non-retina.

 Issue 873413  has been merged into this issue.

This is /also/ now an emergency. I'll work to debug this with ericrk and khushalsagar on monday.

Uploaded a patch for a few potential issues I could spot in the gpu cache: https://chromium-review.googlesource.com/c/chromium/src/+/1171960

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f0951c8174fc4739eb3eac68f75bb516b385ac4a

commit f0951c8174fc4739eb3eac68f75bb516b385ac4a
Author: Christopher Cameron <ccameron@chromium.org>
Date: Sat Aug 11 18:24:40 2018

cc: Make EvictAllUIResources only evict resources, not destroy them

This is to limit the error spew in about:gpu (currently any useful
messages are easily lost in warnings about these blank-resource uses).

This function in SetVisible(false) to ensure that resources be dropped
at tab switch. The current implementation immediately deletes the GL
resource backing, even if it may be in use by the display. This results
in flashed-black scrollbars and lots of GL errors.

Change EvictAllUIResources to just evict the resources without
deleting their backing. The resources are then destroyed when they
are returned from the display (at OnUIResourceReleased).

Bug: 864905,  870317 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I1e079e1790275d3194b565457971e65f7d265587
Reviewed-on: https://chromium-review.googlesource.com/1171832
Commit-Queue: ccameron <ccameron@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582453}
[modify] https://crrev.com/f0951c8174fc4739eb3eac68f75bb516b385ac4a/cc/trees/layer_tree_host_impl.cc
[modify] https://crrev.com/f0951c8174fc4739eb3eac68f75bb516b385ac4a/cc/trees/layer_tree_host_impl.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9d0263ed2378ad50d692ea4ddbd954f4ca08170e

commit 9d0263ed2378ad50d692ea4ddbd954f4ca08170e
Author: Khushal <khushalsagar@chromium.org>
Date: Mon Aug 13 00:29:00 2018

cc: Fix potential texture lifetime issues in the GPU image cache.

This fixes 2 bugs which could result in using invalid textures from the
image cache, if an image is originally uploaded at original size and
mips are generated with a subsequent copy later.

1) When we generate mips the original texture is deleted, while if the
DrawImage is ref-ed and there is an external ref on the SkImage backed
by the original texture it could potentially be used after its deleted.

2) If skia fails to mip the texture for any reason, the "mipped" image
returned is backed by the original texture which we would subsequently
delete.

R=ccameron@chromium.org, ericrk@chromium.org
TBR=piman@chromium.org

Bug:  870317 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: Iec925f35c7880db89f9b68a9cee5e0b76b8ce4d9
Reviewed-on: https://chromium-review.googlesource.com/1171960
Reviewed-by: Khushal <khushalsagar@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Commit-Queue: Khushal <khushalsagar@chromium.org>
Cr-Commit-Position: refs/heads/master@{#582484}
[modify] https://crrev.com/9d0263ed2378ad50d692ea4ddbd954f4ca08170e/cc/tiles/gpu_image_decode_cache.cc
[modify] https://crrev.com/9d0263ed2378ad50d692ea4ddbd954f4ca08170e/cc/tiles/gpu_image_decode_cache.h
[modify] https://crrev.com/9d0263ed2378ad50d692ea4ddbd954f4ca08170e/cc/tiles/gpu_image_decode_cache_unittest.cc
[modify] https://crrev.com/9d0263ed2378ad50d692ea4ddbd954f4ca08170e/gpu/command_buffer/service/gles2_cmd_decoder.cc

I've been hitting this bug every few days.  It seems correlated with extremely high swap usage, so the "running out of shared memory" comment from #26 seems likely.  When I kill and restart canary, the issue goes away for a few days, and then whenever it comes back, I notice that swap usage is very high again (10GB+ swap used).

+erikchen was helping debug why so much swap was being used.

Adding M69 label, since the suspected root cause is present in 69.

Do we have any reports from M69 with this bug? The codepath the change in #36 affects was added in 69.0.3491.0 but all bugs duped here seem to be from M70.

And I think the UI GpuRaster experiment is turned on for M69.

M69 Stable promotion is coming VERY soon. Your bug is labelled as Stable ReleaseBlock, pls make sure to land the fix and request a merge into the release branch ASAP. Thank you.

Got the same bug in a renderer process.

After some discussion, we will definitely want to merge #36 to M69. Rationale is as follows:
- If this fixes the bug (we don't see DumpWithoutCrashing reports) then it fixes a bug present in M69.
- If this does not fix the bug (we see DumpWithoutCrashing reports) then we do not know if this issue affects M69 or not. By merging the DumpWithoutCrashing, we will know whether or not this is happening in M69 (and if we need to panic now or later).

Deleted: renderer-black-square.png 310 KB

	renderer-black-square.png 310 KB View Download

Based on #43, making a request to merge just the crash dump portion of the change in #36 to M69. This doesn't add any new logic, only generates a crash report for when we hit this bug.

Cl listed at #36 is not in canary yet. Pls update teh bug with canary result tomorrow. Thank you.

This bug requires manual review: M69 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), kariahda@(iOS), cindyb@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The NextAction date has arrived: 2018-08-14

This behavior is still occurring in 70.0.3522.0.

Issue 873990 has been merged into this issue.

We got several dump-without-crash instances already - see issue 873990, which I just duped into this one. :(

Requesting merge to M69, so we can see if the DumpWithoutCrashes happen there (if they don't then this is a M70 bug)

Approving  merge to M69 branch 3497 based on comment #44, #51 and per offline chat with ccameron@. Pls merge. Thank you.

Assigning over to Khushal who is actively investigating.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/68057e6b91afcd35842b704dee67cb87a3d0a8aa

commit 68057e6b91afcd35842b704dee67cb87a3d0a8aa
Author: Christopher Cameron <ccameron@chromium.org>
Date: Tue Aug 14 19:37:37 2018

cc: Fix potential texture lifetime issues in the GPU image cache.

This fixes 2 bugs which could result in using invalid textures from the
image cache, if an image is originally uploaded at original size and
mips are generated with a subsequent copy later.

1) When we generate mips the original texture is deleted, while if the
DrawImage is ref-ed and there is an external ref on the SkImage backed
by the original texture it could potentially be used after its deleted.

2) If skia fails to mip the texture for any reason, the "mipped" image
returned is backed by the original texture which we would subsequently
delete.

R=ccameron@chromium.org, ericrk@chromium.org
TBR=khushalsagar@chromium.org, piman@chromium.org

(cherry picked from commit 9d0263ed2378ad50d692ea4ddbd954f4ca08170e)

Bug:  870317 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: Iec925f35c7880db89f9b68a9cee5e0b76b8ce4d9
Reviewed-on: https://chromium-review.googlesource.com/1171960
Reviewed-by: Khushal <khushalsagar@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Commit-Queue: Khushal <khushalsagar@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#582484}
Reviewed-on: https://chromium-review.googlesource.com/1175003
Reviewed-by: ccameron <ccameron@chromium.org>
Cr-Commit-Position: refs/branch-heads/3497@{#624}
Cr-Branched-From: 271eaf50594eb818c9295dc78d364aea18c82ea8-refs/heads/master@{#576753}
[modify] https://crrev.com/68057e6b91afcd35842b704dee67cb87a3d0a8aa/cc/tiles/gpu_image_decode_cache.cc
[modify] https://crrev.com/68057e6b91afcd35842b704dee67cb87a3d0a8aa/cc/tiles/gpu_image_decode_cache.h
[modify] https://crrev.com/68057e6b91afcd35842b704dee67cb87a3d0a8aa/cc/tiles/gpu_image_decode_cache_unittest.cc
[modify] https://crrev.com/68057e6b91afcd35842b704dee67cb87a3d0a8aa/gpu/command_buffer/service/gles2_cmd_decoder.cc

Perhaps relevant:
https://bugs.chromium.org/p/chromium/issues/detail?id=870663#c23 points out
"""
Investigating a shared-memory handle leak (issue 867468) in M69+, sebmarchand@ reached this refactoring CL:
https://chromium-review.googlesource.com/c/chromium/src/+/1127948
"""

We had been discussing that this could be a symptom of share memory allocation failure.

It's possible, but seeing where we're dumping w/out crashing, I expect we've gotten past shm allocation... is it possible there's a bug in shm allocation though that leads to both a leak as well as incorrect allocation? If so, that could explain things.

Just saw this again on 70.0.3522.0 (Official Build) canary (64-bit) - macos, fyi.

Re #58, could you share your chrome://gpu logs too?

I restarted chrome... still useful? 

Ummm no. I don't think logs are retained across restarts.

A request for people seeing this bug: Can you try moving to Chrome Beta (69) and see if you see this bug?

c#61, will try to repro and get you logs.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3cfc77f104e88a795f8c0d240f3bf3522be13983

commit 3cfc77f104e88a795f8c0d240f3bf3522be13983
Author: Khushal <khushalsagar@chromium.org>
Date: Wed Aug 15 07:19:58 2018

cc: Guard GrContext access for upload tasks in the GPU image cache.

The RasterInterface should be notified when accessing the GrContext, so
it can make sure any state tracked in the implementation is reset if it
could be modified by using the GrContext. We are already doing this for
raster tasks in GpuRasterBufferProvider, also do this in upload tasks
which use the GrContext for uploading images.

R=ericrk@chromium.org

Bug:  870317 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I2b5039f2a2f64fb6986181639b81597c741103a5
Reviewed-on: https://chromium-review.googlesource.com/1175415
Commit-Queue: Khushal <khushalsagar@chromium.org>
Reviewed-by: Eric Karl <ericrk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583188}
[modify] https://crrev.com/3cfc77f104e88a795f8c0d240f3bf3522be13983/cc/BUILD.gn
[modify] https://crrev.com/3cfc77f104e88a795f8c0d240f3bf3522be13983/cc/raster/gpu_raster_buffer_provider.cc
[add] https://crrev.com/3cfc77f104e88a795f8c0d240f3bf3522be13983/cc/raster/scoped_grcontext_access.h
[modify] https://crrev.com/3cfc77f104e88a795f8c0d240f3bf3522be13983/cc/tiles/gpu_image_decode_cache.cc

Current theory: caused by crrev.com/577364, initially seen as a spike in gpu::Buffer::GetDataAddress crash reports, isn't in M69.

But we'll know definitively soon.

Back to Eric who is landing the tentative fix soon.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/960dfdf0c4e2e7a99d5db811e266af7ba316d1ea

commit 960dfdf0c4e2e7a99d5db811e266af7ba316d1ea
Author: Eric Karl <ericrk@chromium.org>
Date: Thu Aug 16 00:15:41 2018

Fix GPU discardable handle re-use bug

A change earlier in M70 allowed us to more aggressively re-use
discardable handles. Unfortunately, as handle IDs (even those with
outstanding refs) were re-used as well, leading to issues where the
same handle was being used for two textures.

This change moves to a globally-incrementing ID system for discardable
handles, which removes the possibility of accidentally referencing a
deleted but re-used handle.

Bug:  870317 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I6ba8999d5f90d3ddb9c909cbbbd7f66c30ae6613
Reviewed-on: https://chromium-review.googlesource.com/1176223
Reviewed-by: Khushal <khushalsagar@chromium.org>
Commit-Queue: Eric Karl <ericrk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583456}
[modify] https://crrev.com/960dfdf0c4e2e7a99d5db811e266af7ba316d1ea/gpu/command_buffer/client/client_discardable_manager.cc
[modify] https://crrev.com/960dfdf0c4e2e7a99d5db811e266af7ba316d1ea/gpu/command_buffer/client/client_discardable_manager_unittest.cc
[modify] https://crrev.com/960dfdf0c4e2e7a99d5db811e266af7ba316d1ea/gpu/command_buffer/common/discardable_handle.cc
[modify] https://crrev.com/960dfdf0c4e2e7a99d5db811e266af7ba316d1ea/gpu/command_buffer/common/discardable_handle.h

The DumpWithoutCrashing is not present in
- Canary 70.0.3525.0
- Beta 69.0.3497.42

So I'm calling this fixed -- thanks all!

Sound an alarm if you see this in either of those versions, or later.

 Issue 875722  has been merged into this issue.