The GL context is lost or broken at this point (lots of errors in the logs).

We don't usually want to DCHECK on a GL error, but in the case the gpu process crashes, you'd think it recovers but maybe SingleProcessMash is unable to recover. You could try visiting chrome://gpucrash and see if that recovers at all.

It's hard to say what caused the context to get into this state in the first place, since that's so removed from the eventual DCHECK on GlGetError().

+ccameron who wrote this DCHECK. +sky for mash. +sadrul for mash/gpu? Not sure who would be looking at gpu recovery on mash do one of you know?

Visiting chrome:gpucrash in SingleProcessMash makes all existing chrome, app, and WS client (ie KSV) windows render blank.
Resizing and minimizing/restoring broken windows does not cause them to recover.
The Ash system UI appears unaffected; opening new windows works fine (they paint content fine alongside old broken windows).

Is this TODO relevant: https://chromium.googlesource.com/chromium/src/+/master/ui/aura/mus/mus_context_factory.cc#49 ?

AFAICT when going to chrome://gpucrash MusContextFactory::OnEstablishedGpuChannel() is called and a new AsyncLayerTreeFrameSink is created and set on the Compositor. Further in the ash side HostFrameSinkManager::CreateCompositorFrameSink() is called. What else needs to happen when the client loses a connection to the gpu?

Do the clients (and thus ash) need to allocate a new LocalSurfaceId after a gpu-crash? Perhaps that is not happening correctly with smash?

I think you are right that they aren't generating LSIs after a crash, but a window resize (which should generate a new LSI) doesn't fix the issue.

AFAIK a new LocalSurfaceId should not be necessary.

What should happen when the gpu crashes? It seems we are creating a new AsyncLayerTreeFrameSink  and calling to the WindowService to attach it (HostFrameSinkManager::CreateCompositorFrameSink()).

That should be enough. To me it seems like we're not recreating the lost resources, but I always assumed LayerTreeHostImpl will automatically handle that. danakj@ might know why that might break?

Sorry which resources do you mean? (Also using invalid texture ids just draws black, shouldnt crash the gpu process or WebGL could trivially crash it)

I'm not saying that invalid texture ids are crashing the gpu process. It could be that the gpu process crashes for an unrelated reason and the client fails to recreate its resources. The snapshot in #0 is blank which could indicate invalid resources, and there is also lots of log messages complaining about textures.

I think this bug is now two issues:

. The original DCHECK being triggered on in feature_info.cc:1759. Dana indicated this shouldn't be a DCHECK. I will ping Chris to see if he knows why this is a DCHECK.

. Going to chrome://gpucrashes causes all mus clients to paint black. Resizing the window doesn't help. Clients seem to be creating a new AsyncLayerTreeFrameSink and calling to the WindowService to attach it (HostFrameSinkManager::CreateCompositorFrameSink()). What else needs to be done?

Chris is on OOO. Looks like the DCHECK in feature_info was added here https://codereview.chromium.org/139013008 , specifically https://codereview.chromium.org/139013008/diff/1690001/gpu/command_buffer/service/feature_info.cc . +piman/kbr as they reviewed this patch.

piman/kbr, Dana indicates in comment #1 that it seems weird to DCHECK a gl error like this. Should the DCHECK be updated?

The OpenGL calls in that function are not supposed to generate an error on any OpenGL implementation - they are conservative, and used only for feature detection. We should dig into the error and add more error checking on someone's developer build to figure out exactly what operation is causing the failure.

The OpenGL error, if not detected here, will almost surely cause errors later in the GPU process's execution.

It's a DCHECK because we must not generate GL errors service side, if we do it's a bug in our code, and causes problems later on.

That said this DCHECK often times picks up an error that happened much earlier.

Thanks for the info Ken/Antoine. Antoine, can you suggest places to add DCHECKs to help isolate the issue?

I'm separating out the gpu-crash part into 922201.

One good place to add one would be in GLES2DecoderImpl::DoCommandsImpl and RasterDecoderImpl::DoCommandsImpl in their respective inner loops. Note that if you run with --enable-gpu-debugging it will log GL errors that happen in the driver.

That said, it's very possible that the errors don't happen during command buffer command execution, so you can try to pepper DCHECKs in/around  GLContextEGL::MakeCurrent / GLContext::MakeVirtuallyCurrent to try to isolate the problem a bit.

I tried adding:

    DCHECK_EQ(glGetError(), static_cast<GLuint>(GL_NO_ERROR));

to the front of FeatureInfo::InitializeFeatures() and that does hit in certain cases. The error is 1282. If my web searching is correct that is GL_INVALID_OPERATION.

Right, you have to figure out which GL call generates the error (GL errors are part of the GL state, they are "sticky"), and that can hint at what is going on - GL_INVALID_OPERATION usually indicates an invalid combination of multiple states however.
You can also try --enable-gpu-driver-debug-logging, which makes us register a callback with the driver (if it supports a particular extension) that would log the error, with extra driver information, from the call that generated the error.

I ran with --enable-gpu-debugging --enable-gpu-driver-debug-logging .

It would seem I'm using GlContextGLX (I'm running this on the desktop, not an actual ChromeOS device). I added:

  DCHECK_EQ(glGetError(), static_cast<GLuint>(GL_NO_ERROR));

to GLContextGLX::MakeCurrent before it returns true (both at the start, IsCurrent(), and the end). Here's the output:


[214689:214689:0115/140026.102916:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_SYNC_FLUSH_COMMANDS_BIT_APPLE, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in vertex shader
[214689:214689:0115/140026.105450:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_CONTEXT_FLAG_DEBUG_BIT_KHR, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader
[214689:214689:0115/140026.155853:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_LINE_STRIP, High): 0:2(12) warning: extension `GL_ARB_gpu_shader5' unsupported in vertex shader
[214689:214689:0115/140026.157930:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_GEOMETRY_SHADER_BIT_OES, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader
[214689:214689:0115/140026.240435:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_TRIANGLE_STRIP, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in vertex shader
[214689:214689:0115/140026.241848:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_TRIANGLE_FAN, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader
[214689:214689:0115/140026.301134:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_QUADS_OES, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in vertex shader
[214689:214689:0115/140026.302351:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_CONTEXT_FLAG_NO_ERROR_BIT_KHR, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader
[214689:214689:0115/140026.333498:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_RELATIVE_VERTICAL_LINE_TO_NV, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in vertex shader
[214689:214689:0115/140026.334936:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_LINES_ADJACENCY_OES, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader
[214689:214689:0115/140026.444252:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_LINE_STRIP_ADJACENCY_OES, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in vertex shader
[214689:214689:0115/140026.445425:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0xbadbefd700]GL Driver Message (Shader Compiler, Other, GL_TRIANGLES_ADJACENCY_OES, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader

And closer to the crash I get:

[193822:193822:0115/135536.619856:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0x2239addd0500]GL Driver Message (Shader Compiler, Other, GL_RELATIVE_SMOOTH_QUADRATIC_CURVE_TO_NV, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in vertex shader
[193822:193822:0115/135536.622167:ERROR:gl_utils.cc(328)] [.BrowserCompositor-0x2239addd0500]GL Driver Message (Shader Compiler, Other, GL_TESS_EVALUATION_SHADER_BIT_OES, High): 0:2(12): warning: extension `GL_ARB_gpu_shader5' unsupported in fragment shader
[193822:193822:0115/135540.234615:FATAL:gl_context_glx.cc(246)] Check failed: ::gl::g_current_gl_context_tls->Get()->Api->glGetErrorFn() == static_cast<GLuint>(0x0) (1282 vs. 0)
#0 0x7f63b7bbfecf base::debug::StackTrace::StackTrace()
#1 0x7f63b7aecd2a logging::LogMessage::~LogMessage()
#2 0x7f63b2113848 gl::GLContextGLX::MakeCurrent()
#3 0x7f63a7dde258 gpu::SharedImageStub::MakeContextCurrent()
#4 0x7f63a7ddc6d3 gpu::SharedImageStub::MakeContextCurrentAndCreateFactory()
#5 0x7f63a7ddd93c gpu::SharedImageStub::OnUpdateSharedImage()
#6 0x7f63a7ddd862 IPC::MessageT<>::Dispatch<>()
#7 0x7f63a7ddc8f7 gpu::SharedImageStub::OnMessageReceived()
#8 0x7f63b617c412 IPC::MessageRouter::RouteMessage()
#9 0x7f63a7dce0b4 gpu::GpuChannel::HandleMessageHelper()
#10 0x7f63a7dcbb26 gpu::GpuChannel::HandleMessage()
#11 0x7f63a7dcf470 base::internal::Invoker<>::RunOnce()
#12 0x7f63b3dfb590 gpu::Scheduler::RunNextTask()
#13 0x7f63b3dfe0d7 base::internal::Invoker<>::RunOnce()
#14 0x7f63b7ad0fa1 base::debug::TaskAnnotator::RunTask()
#15 0x7f63b7b60d47 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
#16 0x7f63b7b61244 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
#17 0x7f63b7be3199 base::MessagePumpLibevent::Run()
#18 0x7f63b7b615bc base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
#19 0x7f63b7b2cfb5 base::RunLoop::Run()
#20 0x7f63b4579fc1 content::GpuMain()
#21 0x7f63b5511bac content::ContentMainRunnerImpl::Run()
#22 0x7f63a7b303b3 service_manager::Main()
#23 0x7f63b5510024 content::ContentMain()
#24 0x55bd7f0bc943 ChromeMain
#25 0x7f63a80772b1 __libc_start_main
#26 0x55bd7f0bc7ba _start

I did not see anything about GL_RELATIVE_SMOOTH_QUADRATIC_CURVE_TO_NV or GL_TESS_EVALUATION_SHADER_BIT_OES

sky@ can you try running this script with the following argument:

src/ui/gl/generate_bindings.py --generate-dchecks

then rebuilding and rerunning with --enable-gpu-service-logging?

see:

https://cs.chromium.org/chromium/src/ui/gl/generate_bindings.py?q=src/ui/gl/generate_bindings.py&sq=package:chromium&g=0&l=3756

That flag will crash the GPU process upon the first OpenGL error that happens. If you're set up to catch those crashes in the debugger then it may be the fastest way of tracking down exactly what command generated the error.

I think the DCHECKs I added are likely wrong.

I can't seem to repro with --enable-gpu-debugging --enable-gpu-driver-debug-logging .

I was able to repro with --generate-dchecks and --enable-gpu-service-logging. Here's the last part of the output:


[99653:99653:0117/102220.033748:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kBindTexture
[99653:99653:0117/102220.033775:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.033800:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.033825:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kDeleteTexturesImmediate
[99653:99653:0117/102220.033865:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kDeleteTexturesImmediate
[99653:99653:0117/102220.033897:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kBindTexture
[99653:99653:0117/102220.033923:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.033949:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.033974:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kDeleteTexturesImmediate
[99653:99653:0117/102220.034057:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kBindTexture
[99653:99653:0117/102220.034090:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034117:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034143:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kDeleteTexturesImmediate
[99653:99653:0117/102220.034213:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kBindTexture
[99653:99653:0117/102220.034245:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034272:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034297:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kDeleteTexturesImmediate
[99653:99653:0117/102220.034368:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kBindTexture
[99653:99653:0117/102220.034399:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034426:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034450:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kDeleteTexturesImmediate
[99653:99653:0117/102220.034511:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kBindTexture
[99653:99653:0117/102220.034542:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034568:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kTexParameteri
[99653:99653:0117/102220.034593:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kDeleteTexturesImmediate
[99653:99653:0117/102220.034633:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x22ccae885100]cmd: kInsertFenceSyncCHROMIUM
[99653:99653:0117/102220.034858:FATAL:gl_bindings_autogen_gl.cc(9583)] Check failed: error == 0.
#0 0x7fad74b6089f base::debug::StackTrace::StackTrace()
#1 0x7fad74a8d8ba logging::LogMessage::~LogMessage()
#2 0x7fad6f04a931 gl::DebugGLApi::glBindTextureFn()
#3 0x7fad6d671c2a gpu::SharedImageBackingGLTexture::Update()
#4 0x7fad6d672616 gpu::SharedImageFactory::UpdateSharedImage()
#5 0x7fad64d7494c gpu::SharedImageStub::OnUpdateSharedImage()
#6 0x7fad64d74862 IPC::MessageT<>::Dispatch<>()
#7 0x7fad64d738f7 gpu::SharedImageStub::OnMessageReceived()
#8 0x7fad7311c412 IPC::MessageRouter::RouteMessage()
#9 0x7fad64d650b4 gpu::GpuChannel::HandleMessageHelper()
#10 0x7fad64d62b26 gpu::GpuChannel::HandleMessage()
#11 0x7fad64d66470 base::internal::Invoker<>::RunOnce()
#12 0x7fad70d9b590 gpu::Scheduler::RunNextTask()
#13 0x7fad70d9e0d7 base::internal::Invoker<>::RunOnce()
#14 0x7fad74a71b31 base::debug::TaskAnnotator::RunTask()
#15 0x7fad74b01717 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
#16 0x7fad74b01c14 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
#17 0x7fad74b83b69 base::MessagePumpLibevent::Run()
#18 0x7fad74b01f8c base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
#19 0x7fad74acd9b5 base::RunLoop::Run()
#20 0x7fad71519f21 content::GpuMain()
#21 0x7fad724b1b7c content::ContentMainRunnerImpl::Run()
#22 0x7fad64ac73b3 service_manager::Main()
#23 0x7fad724afff4 content::ContentMain()
#24 0x55bdd443b803 ChromeMain
#25 0x7fad6500e2b1 __libc_start_main
#26 0x55bdd443b67a _start


Complete output is here: https://sky.users.x20web.corp.google.com/gpu.crash.log (WARNING: it's 18MB)


Also worth noting, I see a bunch of warnings like this:

[99653:99653:0117/102220.014478:ERROR:gles2_cmd_decoder.cc(10399)] [.BrowserCompositor-0x22ccae885100]RENDER WARNING: texture bound to texture unit 0 is not render\
able. It maybe non-power-of-2 and have incompatible texture filtering.


Ken/Antoine, any suggestions?

According to that stack trace the error is happening inside the SharedImageFactory. Can you add more detailed logging just before the point of failure to see what kind of texture we're attempting to bind? Really, one of the GPU folks who have been working on SharedImage should help with this - perhaps piman, ericrk, cblume, vikassoni.

In trying a couple more times, it seems I hit this more often:

[118699:118699:0117/110723.560029:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x372d5f0c3000]cmd: kDisable
[118699:118699:0117/110723.560073:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x372d5f0c3000]cmd: SetToken
[118699:118699:0117/110723.560100:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserCompositor-0x372d5f0c3000]cmd: kPostSubBufferCHROMIUM
[118699:118699:0117/110724.186428:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserWorker-0x372d5efd4800]cmd: kDeleteQueriesEXTImmediate
[118699:118699:0117/110724.186510:ERROR:gles2_cmd_decoder.cc(5807)] [.BrowserWorker-0x372d5efd4800]cmd: kDeleteQueriesEXTImmediate
[118699:118699:0117/110724.187677:ERROR:gles2_cmd_decoder.cc(5807)] [.MusClient-0x372d5f0c4500]cmd: kDeleteQueriesEXTImmediate
[118699:118699:0117/110724.187736:ERROR:gles2_cmd_decoder.cc(5807)] [.MusClient-0x372d5f0c4500]cmd: kDeleteQueriesEXTImmediate
[118699:118699:0117/110724.453384:ERROR:gles2_cmd_decoder.cc(5807)] [GroupMarkerNotSet( crbug.com/242999 )!:B088DC8254000000]cmd: SetBucketSize
[118699:118699:0117/110724.453484:ERROR:gles2_cmd_decoder.cc(5807)] [GroupMarkerNotSet( crbug.com/242999 )!:B088DC8254000000]cmd: SetBucketData
[118699:118699:0117/110724.453559:ERROR:gles2_cmd_decoder.cc(5807)] [GroupMarkerNotSet( crbug.com/242999 )!:B088DC8254000000]cmd: SetToken
[118699:118699:0117/110724.453639:ERROR:gles2_cmd_decoder.cc(5807)] [GroupMarkerNotSet( crbug.com/242999 )!:B088DC8254000000]cmd: SetBucketSize
[118699:118699:0117/110724.453728:ERROR:gles2_cmd_decoder.cc(5807)] [GroupMarkerNotSet( crbug.com/242999 )!:B088DC8254000000]cmd: SetBucketData
[118699:118699:0117/110724.453800:ERROR:gles2_cmd_decoder.cc(5807)] [GroupMarkerNotSet( crbug.com/242999 )!:B088DC8254000000]cmd: SetToken
[118699:118699:0117/110724.453859:ERROR:gles2_cmd_decoder.cc(5807)] [GroupMarkerNotSet( crbug.com/242999 )!:B088DC8254000000]cmd: kTraceBeginCHROMIUM
[118699:118699:0117/110724.453895:ERROR:gles2_cmd_decoder.cc(5807)] [.MusClient-0x372d5f896000]cmd: SetBucketSize
[118699:118699:0117/110724.453923:ERROR:gles2_cmd_decoder.cc(5807)] [.MusClient-0x372d5f896000]cmd: SetBucketSize
[118699:118699:0117/110724.462774:FATAL:gl_bindings_autogen_gl.cc(9583)] Check failed: error == 0.
#0 0x7f1b20f9889f base::debug::StackTrace::StackTrace()
#1 0x7f1b20ec58ba logging::LogMessage::~LogMessage()
#2 0x7f1b1b482931 gl::DebugGLApi::glBindTextureFn()
#3 0x7f1b19aa7f2e gpu::SharedImageBackingFactoryGLTexture::CreateSharedImage()
#4 0x7f1b19aa754b gpu::SharedImageBackingFactoryGLTexture::CreateSharedImage()
#5 0x7f1b19aaa18c gpu::SharedImageFactory::CreateSharedImage()
#6 0x7f1b111ababd gpu::SharedImageStub::OnCreateSharedImage()
#7 0x7f1b111ab9c1 IPC::MessageT<>::Dispatch<>()
#8 0x7f1b111ab8a0 gpu::SharedImageStub::OnMessageReceived()
#9 0x7f1b1f554412 IPC::MessageRouter::RouteMessage()
#10 0x7f1b1119d0b4 gpu::GpuChannel::HandleMessageHelper()
#11 0x7f1b1119ab26 gpu::GpuChannel::HandleMessage()
#12 0x7f1b1119e470 base::internal::Invoker<>::RunOnce()
#13 0x7f1b1d1d3590 gpu::Scheduler::RunNextTask()
#14 0x7f1b1d1d60d7 base::internal::Invoker<>::RunOnce()
#15 0x7f1b20ea9b31 base::debug::TaskAnnotator::RunTask()
#16 0x7f1b20f39717 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl()
#17 0x7f1b20f39c14 base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
#18 0x7f1b20fbbb69 base::MessagePumpLibevent::Run()
#19 0x7f1b20f39f8c base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run()
#20 0x7f1b20f059b5 base::RunLoop::Run()
#21 0x7f1b1d951f21 content::GpuMain()
#22 0x7f1b1e8e9b7c content::ContentMainRunnerImpl::Run()
#23 0x7f1b10eff3b3 service_manager::Main()
#24 0x7f1b1e8e7ff4 content::ContentMain()
#25 0x55e042437803 ChromeMain
#26 0x7f1b114462b1 __libc_start_main
#27 0x55e04243767a _start

It looks like that glBindTexture call is one of the first OpenGL API calls in SharedImageBackingFactoryGLTexture::CreateSharedImage (actually, MakeTextureAndSetParameters, called by it). Do you think you could add a DCHECK about glGetError being 0 before that? Could you log the arguments to the BindTexture call?

The crash in #23 is extremely surprising. The only way glBindTexture fails is if a texture is bound with an incorrect target. I really don't see how that could happen (for a given texture, the target is decided once and for all and effectively immutable in our tracking structures). I suppose one could build a theory with a use-after-free of the texture, but that's fairly far-fetched. Even more crazy would be the one in #25 where glBindTexture is called on a newly created texture (in which case it cannot fail, assuming we give it a proper target). Well, there's also the glBindTexture from ~ScopedRestoreTexture, but that just resets a binding that we looked up from the driver, so I can't build a theory of how that would fail.

Now the other possibility is that the error was preexisting the glBindTexture call. This would happen if something calls gl without going through the GLApi layer and generates and error. That would be the case for skia (it goes to the driver functions directly). Maybe we can add a DCHECK *before* the GL calls to confirm

I thought maybe there was corruption happening as well, and tried this under asan. I couldn't reproduce with an asan build.

When I hit the DCHECK from gpu::SharedImageBackingGLTexture::Update() texture_->GetType() returns 1.

I added:

  DCHECK_EQ(glGetError(), static_cast<GLuint>(GL_NO_ERROR));

as the first call in SharedImageBackingFactoryGLTexture::CreateSharedImage() as well as ScopedRestoreTexture's constructor. It doesn't hit. That is, I'm still seeing:

[222592:222592:0117/143053.303719:FATAL:gl_bindings_autogen_gl.cc(9583)] Check failed: error == 0.
#0 0x7f85104f289f base::debug::StackTrace::StackTrace()
#1 0x7f851041f8ba logging::LogMessage::~LogMessage()
#2 0x7f850a9dc931 gl::DebugGLApi::glBindTextureFn()
#3 0x7f8509001ee5 gpu::SharedImageBackingFactoryGLTexture::CreateSharedImage()
#4 0x7f850900154b gpu::SharedImageBackingFactoryGLTexture::CreateSharedImage()
#5 0x7f85090040dc gpu::SharedImageFactory::CreateSharedImage()
#6 0x7f8500705abd gpu::SharedImageStub::OnCreateSharedImage()
#7 0x7f85007059c1 IPC::MessageT<>::Dispatch<>()
#8 0x7f85007058a0 gpu::SharedImageStub::OnMessageReceived()

I'm at a loss. Can you confirm whether the failing glBindTexture in #28 is from MakeTextureAndSetParameters() or from the ScopedRestoreTexture destructor?
Can you show the target and service id in either case?


(texture_->GetType() returning 1 is expected, it just means that it's a gles2::Texture, which is consistent with the static type)

Actually, it looks like the DCHECK in ScopedRestoreTexture()'s constructor is hitting more reliably. 

After adding more DCHECKs and logging I seem to be consistently hitting the error in OnCreateGMBSharedImage. Specifically:

[72276:72276:0117/162056.962799:ERROR:shared_image_backing_factory_gl_texture.cc(175)] calling MakeTextureAndSetParameters
[72276:72276:0117/162056.962829:ERROR:shared_image_backing_factory_gl_texture.cc(177)] called MakeTextureAndSetParameters
[72276:72276:0117/162056.996156:ERROR:shared_image_backing_factory_gl_texture.cc(327)] SharedImageBackingGLTexture::Update texture_type=1
[72276:72276:0117/162057.011518:ERROR:shared_image_backing_factory_gl_texture.cc(327)] SharedImageBackingGLTexture::Update texture_type=1
[72276:72276:0117/162057.031836:ERROR:shared_image_backing_factory_gl_texture.cc(327)] SharedImageBackingGLTexture::Update texture_type=1
[72276:72276:0117/162057.046533:ERROR:shared_image_backing_factory_gl_texture.cc(327)] SharedImageBackingGLTexture::Update texture_type=1
[72276:72276:0117/162057.066238:ERROR:shared_image_backing_factory_gl_texture.cc(327)] SharedImageBackingGLTexture::Update texture_type=1
[72276:72276:0117/162057.112405:ERROR:shared_image_backing_factory_gl_texture.cc(327)] SharedImageBackingGLTexture::Update texture_type=1
[72276:72276:0117/162057.149673:ERROR:shared_image_backing_factory_gl_texture.cc(327)] SharedImageBackingGLTexture::Update texture_type=1
[72238:72238:0117/162058.928340:ERROR:browser_view.cc(380)] START
[72238:72238:0117/162058.930465:ERROR:window_event_dispatcher.cc(954)] PRESSED loc=1210,775
[72276:72276:0117/162058.939657:ERROR:shared_image_stub.cc(150)] OnCreateGMBSharedImage id=1 format=10 size=128x128 space={primaries:BT709, transfer:IEC61966_2_1, \
matrix:RGB, range:FULL}
[72276:72276:0117/162058.939800:ERROR:shared_image_backing_factory_gl_texture.cc(175)] calling MakeTextureAndSetParameters
[72276:72276:0117/162058.939856:ERROR:shared_image_backing_factory_gl_texture.cc(177)] called MakeTextureAndSetParameters
[72238:72238:0117/162058.960849:ERROR:browser_view.cc(387)] STEP1
[72276:72276:0117/162058.940360:FATAL:shared_image_stub.cc(147)] Check failed: ::gl::g_current_gl_context_tls->Get()->Api->glGetErrorFn() == static_cast<GLuint>(0x\
0) (1282 vs. 0)
#0 0x7f523f46989f base::debug::StackTrace::StackTrace()
#1 0x7f523f3968ba logging::LogMessage::~LogMessage()
#2 0x7f522f67d644 gpu::SharedImageStub::OnCreateGMBSharedImage()



Here's the arguments supplied to OnCreateGMBSharedImage() right before the DCHECK for a number of failures:

[65477:65477:0117/160738.789369:ERROR:shared_image_stub.cc(149)] OnCreateGMBSharedImage id=7 format=10 size=256x256 space={primaries:BT709, transfer:IEC61966_2_1, matrix:RGB, range:FULL}

[66151:66151:0117/160853.349957:ERROR:shared_image_stub.cc(149)] OnCreateGMBSharedImage id=1 format=10 size=128x64 space={primaries:BT709, transfer:IEC61966_2_1, matrix:RGB, range:FULL}

[65477:65477:0117/160738.789369:ERROR:shared_image_stub.cc(149)] OnCreateGMBSharedImage id=7 format=10 size=256x256 space={primaries:BT709, transfer:IEC61966_2_1, matrix:RGB, range:FULL}

[66151:66151:0117/160853.349957:ERROR:shared_image_stub.cc(149)] OnCreateGMBSharedImage id=1 format=10 size=128x64 space={primaries:BT709, transfer:IEC61966_2_1, matrix:RGB, range:FULL}


This is the DCHECK I added:

  DCHECK_EQ(glGetError(), static_cast<GLuint>(GL_NO_ERROR));

And I added it right after the early return if MakeContextCurrentAndCreateFactory() fails.

I tried to repro locally, and as far as I can tell the error is generated by ~ScopedRestoreTexture. That's really crazy, as we're rebinding a value that's coming from the driver. I will investigate some more.

FYI I've seen the original DCHECK a couple times when running *without* the SingleProcessMash flag.

linux-chromeos, release-with-dchecks build, at ToT r624160. I saw it when I had a fresh user data dir, ran with --login-manager and signed into a test account. It doesn't happen every time, unfortunately.

FWIWs here's a HACK patch that readily reproduces this (ChromeOS):
https://chromium-review.googlesource.com/c/chromium/src/+/1423344
Run it like --enable-features=SingleProcessMash --hack --user-data-dir=/tmp/tmp-profile

When you pass in --hack chrome generates events that should open the tray, open settings, close settings. This sequence typically triggers the crash pretty readily. Because mash isn't correctly updating things when the gpu crashes (922201) the first time you see something go black you know you hit it.

So here's what's happening. I'm dumping the calls to glDeleteTextures, and calls in ScopedRestoreTexture destructor/constructor, with associated GLX context.

[19136:19136:0118/131916.686113:ERROR:gl_bindings_autogen_gl.cc(3269)] glDeleteTextures 173 context=0xecbd082948
[19136:19136:0118/131916.694145:ERROR:shared_image_backing_factory_gl_texture.cc(158)] ScopedRestoreTexture target=3553 get_target=32873 old_binding=173 context=0xecbd081d48
[19136:19136:0118/131916.694357:ERROR:shared_image_backing_factory_gl_texture.cc(164)] ~ScopedRestoreTexture target=3553 old_binding=173 context=0xecbd081d48
[19136:19136:0118/131916.694408:FATAL:shared_image_backing_factory_gl_texture.cc(166)] Check failed: ::gl::g_current_gl_context_tls->Get()->Api->glGetErrorFn() == static_cast<GLuint>(0x0) (1282 vs. 0)

Basically, in context A (0xecbd082948), we delete a texture (id 173 here), that was still bound in context B (0xecbd081d48). This will reset all the bindings that point to that texture in context A, but (expectedly) doesn't do anything in context B. When we switch to context B, we look up the current texture binding, and get the id of that deleted texture. When we try to bind that id, we get a GL_INVALID_OPERATION.

This is a super subtle edge corner of the spec. It doesn't actually say anything about this case, all reasonable driver implementations have problems:
1- resetting context B's bindings to 0 would have race conditions if context B is bound to another thread
2- delaying deletion of the texture (internal refcounting) would violate the spec relative to id reuse, causing bad failures (incomplete/inconsistent textures).
3- marking the id as free (while still bound) would cause the (common) pattern used in ScopedRestoreTexture to leak (binding the freed id would recreate a texture, but no-one would be there to delete it as we lost any reference to that id)
4- marking the id as in limbo state, that is trying to bind it is an error (what we end up getting). It probably technically violates the spec, but probably in the least harmful way.


Ok, so what can we do about it? The problem is that we can't really practically track which context has which texture bound, so that we could manually clean them up. In particular skia does all its operations without involving chrome code, so we can't reasonably track if it has a texture id still bound that we would destroy on the chrome side. Or vice versa.


Truth is, in this case, the error is effectively harmless. We could ignore it (or, say, reset the binding if it happens) and be fine. It'd have a small cost. There is however a risk that a driver implements (3) in which case we could cause a slow leak, that would be virtually impossible to detect/properly investigate.

Otherwise, we would need to somehow guarantee that every subsystem never keeps a texture bound that might get deleted in another context. It would be the case today for GLES2DecoderImpl (basically, the underlying texture wouldn't be deleted until it's unbound), but I'm not positive about other subsystems (and obviously at least one doesn't), e.g. skia, RasterDecoderImpl, GLES2DecoderPassthroughImpl etc.

https://chromium-review.googlesource.com/c/chromium/src/+/1422961/ seems to fix this particular case for me, without strong guarantees, though.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2c9d3fbb17e42ddb1bd038cabe4deccac1b28a89

commit 2c9d3fbb17e42ddb1bd038cabe4deccac1b28a89
Author: Antoine Labour <piman@chromium.org>
Date: Sat Jan 19 08:43:43 2019

Always reset texture binding in ~ScopedTextureBinder

Leaving a texture bound can cause trouble if that texture ends up destroyed in
another context. So always reset to binding 0 after CopySubTexture.

Bug: 921046
Change-Id: I39b6c74d59198fc1a7151363119b4d1babac9710
Reviewed-on: https://chromium-review.googlesource.com/c/1422961
Commit-Queue: Antoine Labour <piman@chromium.org>
Reviewed-by: Peng Huang <penghuang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#624459}
[modify] https://crrev.com/2c9d3fbb17e42ddb1bd038cabe4deccac1b28a89/gpu/command_buffer/service/raster_decoder.cc
[modify] https://crrev.com/2c9d3fbb17e42ddb1bd038cabe4deccac1b28a89/gpu/command_buffer/service/raster_decoder_unittest_base.cc