New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 858879 link

Starred by 3 users

Issue metadata

Status: Verified
Owner:
Closed: Jul 26
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: Bug-Regression

Blocked on:
issue 802379
issue 856858

Blocking:
issue 850257



Sign in to add a comment

Get WebGL 2.0 transform_feedback tests to pass on Android

Project Member Reported by kbr@chromium.org, Jun 29 2018

Issue description

In Issue 850257 there's an attempt to finally get the WebGL 2.0 conformance tests running on the waterfall. One large block of failures are the transform_feedback tests ported from the dEQP. It looks like there's some bug in the command buffer's setup causing an OpenGL error to be generated during BackFramebuffer::Create and that's propagating into the test. If this issue is debugged the tests will likely pass, because the dEQP's native transform feedback tests are in the must-pass list.

 

Comment 1 by kbr@chromium.org, Jun 29 2018

kainino@ points out in https://bugs.chromium.org/p/chromium/issues/detail?id=850257#c3 that a few of the transform feedback tests are suppressed in the native dEQP:

From gles3-driver-issues.txt ( b/22760309 ):
dEQP-GLES3.functional.transform_feedback.random.separate.triangles.1
dEQP-GLES3.functional.transform_feedback.random.interleaved.lines.3
dEQP-GLES3.functional.transform_feedback.random.interleaved.triangles.7
dEQP-GLES3.functional.transform_feedback.random.interleaved.triangles.8
dEQP-GLES3.functional.transform_feedback.random.interleaved.triangles.10

Blockedon: 802379
Components: Internals>GPU
Labels: -Type-Feature Type-Bug-Regression
Owner: penghuang@chromium.org
Status: Assigned (was: Available)
We've seen these fail on Pixel 2, but not on Nexus 5X:

WebglConformance_deqp_functional_gles3_transformfeedback_array_interleaved_lines
WebglConformance_deqp_functional_gles3_transformfeedback_array_interleaved_points
WebglConformance_deqp_functional_gles3_transformfeedback_array_interleaved_triangles
WebglConformance_deqp_functional_gles3_transformfeedback_array_separate_lines
WebglConformance_deqp_functional_gles3_transformfeedback_array_separate_points
WebglConformance_deqp_functional_gles3_transformfeedback_array_separate_triangles
WebglConformance_deqp_functional_gles3_transformfeedback_basic_types_interleaved_lines
WebglConformance_deqp_functional_gles3_transformfeedback_basic_types_interleaved_points
WebglConformance_deqp_functional_gles3_transformfeedback_basic_types_interleaved_triangles
WebglConformance_deqp_functional_gles3_transformfeedback_basic_types_separate_lines
WebglConformance_deqp_functional_gles3_transformfeedback_basic_types_separate_points
WebglConformance_deqp_functional_gles3_transformfeedback_basic_types_separate_triangles
WebglConformance_deqp_functional_gles3_transformfeedback_interpolation_centroid
WebglConformance_deqp_functional_gles3_transformfeedback_interpolation_flat
WebglConformance_deqp_functional_gles3_transformfeedback_interpolation_smooth
WebglConformance_deqp_functional_gles3_transformfeedback_point_size
WebglConformance_deqp_functional_gles3_transformfeedback_position
WebglConformance_deqp_functional_gles3_transformfeedback_random_interleaved_lines
WebglConformance_deqp_functional_gles3_transformfeedback_random_interleaved_points
WebglConformance_deqp_functional_gles3_transformfeedback_random_interleaved_triangles
WebglConformance_deqp_functional_gles3_transformfeedback_random_separate_lines
WebglConformance_deqp_functional_gles3_transformfeedback_random_separate_points
WebglConformance_deqp_functional_gles3_transformfeedback_random_separate_triangles

On my Pixel 1, I bisected the failure in
WebglConformance_deqp_functional_gles3_transformfeedback_array_interleaved_lines
to this commit:
https://chromium.googlesource.com/chromium/src/+/715e4f97981a55b2983077649f4915125afdd628

So assigning to penghuang for investigation.

Logs from dev tools console:
                               Init testcase: transform_feedback.array.interleaved.lines.lowp_float
js-test-pre.js:153             FAIL gl.createProgram()
js-test-pre.js:137             TestFailedException {message: "gl.createProgram()", name: "TestFailedException"}
array_interleaved_lines.html:1 [GroupMarkerNotSet( crbug.com/242999 )!:509A9CC9]GL ERROR :GL_INVALID_OPERATION : BackFramebuffer::Create: <- error from previous GL command
array_interleaved_lines.html:1 [.Offscreen-For-WebGL-0xdb2bb700]GL ERROR :GL_INVALID_OPERATION : GetIntegerv: <- error from previous GL command
array_interleaved_lines.html:1 [.Offscreen-For-WebGL-0xdb2bb700]GL ERROR :GL_INVALID_OPERATION : glTexImage2D: <- error from previous GL command
Cc: -kainino@chromium.org penghuang@chromium.org
Owner: kainino@chromium.org
I recently fixed a PresentationHelper related issue. It can be related. Could you please check if this issue still happens with below change? Thanks.

https://chromium.googlesource.com/chromium/src/+/eac113a73a2aaab75f2068b9ef36a871ade6919a

Cc: kainino@chromium.org
Owner: penghuang@chromium.org
Unfortunately it is still failing on 69.0.3480.0, which according to:
https://chromiumdash.appspot.com/commits?user=penghuang%40chromium.org&platform=Android
https://chromiumdash.appspot.com/commit/eac113a73a2aaab75f2068b9ef36a871ade6919a
looks like it should include your fix.
Cc: piman@chromium.org
I found the GPU crashed for this test. Crash log:

[FATAL:gpu_tracer.cc(336)] Check failed: GL_NO_ERROR == glGetError(). 

Stack Trace:
  RELADDR   FUNCTION                                                                                               FILE:LINE
  000db1ad  logging::LogMessage::~LogMessage()                                                                     /usr/local/google/home/penghuang/sources/chromium/src/base/logging.cc:592:29
  0010adfb  gpu::gles2::GPUTracer::ProcessTraces()                                                                 /usr/local/google/home/penghuang/sources/chromium/src/gpu/command_buffer/service/gpu_tracer.cc:336:3
  000ecdc9  gpu::gles2::GLES2DecoderImpl::PerformIdleWork()                                                        /usr/local/google/home/penghuang/sources/chromium/src/gpu/command_buffer/service/gles2_cmd_decoder.cc:16553:16
  00019485  gpu::CommandBufferStub::PerformWork()                                                                  /usr/local/google/home/penghuang/sources/chromium/src/gpu/ipc/service/command_buffer_stub.cc:363:25
  00019361  gpu::CommandBufferStub::PollWork()                                                                     /usr/local/google/home/penghuang/sources/chromium/src/gpu/ipc/service/command_buffer_stub.cc:333:3
  000c719b  base::OnceCallback<void ()>::Run() &&                                                                  /usr/local/google/home/penghuang/sources/chromium/src/base/callback.h:99:12
  000cf7a3  base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)                                   /usr/local/google/home/penghuang/sources/chromium/src/base/debug/task_annotator.cc:101:33
  000e153f  base::MessageLoop::RunTask(base::PendingTask*)                                                         /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:454:46
  000e174b  base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)                                            /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:465:5
  000e1acf  base::MessageLoop::DoDelayedWork(base::TimeTicks*)                                                     /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:620:10
  000e2fc1  base::MessagePumpDefault::Run(base::MessagePump::Delegate*)                                            /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_pump_default.cc:41:27
  000e1307  base::MessageLoop::Run(bool)                                                                           /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:406:12
  000f57f3  base::RunLoop::Run()                                                                                   /usr/local/google/home/penghuang/sources/chromium/src/base/run_loop.cc:102:14
  00855d7f  content::GpuMain(content::MainFunctionParams const&)                                                   /usr/local/google/home/penghuang/sources/chromium/src/content/gpu/gpu_main.cc:346:21
  00fe8ef5  content::ContentMainRunnerImpl::Run(bool)                                                              /usr/local/google/home/penghuang/sources/chromium/src/content/app/content_main_runner_impl.cc:951:10
  0000e93d  service_manager::Main(service_manager::MainParams const&)                                              /usr/local/google/home/penghuang/sources/chromium/src/services/service_manager/embedder/main.cc:472:29
  v------>  content::JNI_ContentMain_Start(_JNIEnv*, base::android::JavaParamRef<_jclass*> const&, unsigned char)  /usr/local/google/home/penghuang/sources/chromium/src/content/app/android/content_main.cc:53:10
  00fe85ff  Java_org_chromium_content_app_ContentMain_nativeStart                                                  /usr/local/google/home/penghuang/sources/chromium/src/out_android/Release/gen/content/public/android/content_jni_headers/content/jni/ContentMain_jni.h:48:0
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so

Cc: kbr@chromium.org
Couple of questions:
1- why do we run GPUTracer::ProcessTraces? I think we should only do that if we request the gpu.device trace category, which I don't think we should for webgl conformance tests (I don't believe they would give us useful information anyway)
2- is it possible that this is a GL error from a test (which shouldn't happen, but...), that only gets caught there? It should be straightforward to add another DCHECK right after the MakeCurrent to validate that none of the code in this function causes an error.
FYI. I added a DCHECK() just after decoder->MakeCurrent(), it can catch the error.
1) I'm not sure why we're calling GPUTracer::ProcessTraces. It's certainly unintentional. Looking at the webgl2_conformance_tests run here:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28NVIDIA%29/1730

and shard #0 in particular:

https://chromium-swarm.appspot.com/task?id=3ebd19486b3a3110&refresh=10&show_raw=1

it looks like the browser's command line arguments are:

'--disable-gpu-watchdog', '--enable-experimental-web-platform-features', '--test-type=gpu', '--disable-domain-blocking-for-3d-apis', '--disable-gpu-process-crash-limit', '--disable-blink-features=WebXR', '--js-flags=--expose-gc', '--enable-logging=stderr', '--autoplay-policy=no-user-gesture-required', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--enable-gpu-benchmarking', '--deny-permission-prompts', '--autoplay-policy=no-user-gesture-required', '--disable-background-networking', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--disable-search-geolocation-disclosure', '--proxy-server=socks://localhost:56377', '--ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I=', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--disable-component-update', '--window-size=1280,1024', '--user-data-dir=c:\\b\\s\\w\\itudsqrd\\tmpo7cxcg', 'about:blank'

There's no intent to call Telemetry's tracing_controller's StartTracing method from anywhere in the WebGL conformance tests:
https://cs.chromium.org/search/?q=StartTracing+file:third_party/catapult/telemetry/telemetry&sq=package:chromium&type=cs

Any help figuring out why this is running would be appreciated.

2) The OpenGL error started showing up (in this form, at least) when https://chromium.googlesource.com/chromium/src/+/715e4f97981a55b2983077649f4915125afdd628 was landed, so the question at the moment is: what OpenGL or MakeCurrent calls did that CL enable?

I wouldn't be surprised if there's a bad interaction between the tracing's queries and the GLSurfacePresentationHelper's queries - even though GPUTimingImpl is meant to deal with that, I don't think we've exercised this logic a lot (and might run into driver issues as well).

Finding out why tracing gets turned on would be super helpful I think, as it doesn't sound like we want it, probably has some overhead, and might be a cause of this.
I'd forgotten that Kai found in https://bugs.chromium.org/p/chromium/issues/detail?id=858879#c5 that this happens with just Chrome. The Telemetry harness is not involved.

If GPUTracer is not supposed to be running in vanilla Chrome without tracing enabled then penghuang@ can you please track down why it is?

As my test, gpu::gles2::GPUTracer::ProcessTraces() is called, but finished_traces_ is always empty. So gpu::gles2::GPUTracer::ProcessTraces()() doesn't do any useful things, but it will MakeCurrent and check the GL error.
I also tried to early return in gpu::gles2::GPUTracer::ProcessTraces(), if the finished_traces_ is empty. The crash will happen in different place(See the crash log). I also added many DCHECK(GL_NO_ERROR, glGetError()) in GlSurfacePresentationHelper, but none of them catches the GL error.


Crash log:

[FATAL:gl_context.cc(323)] Check failed: error == GL_NO_ERROR || error == GL_CONTEXT_LOST_KHR. GL error was: 1282

Stack Trace:
  RELADDR   FUNCTION                                                                                               FILE:LINE
  000db1ad  logging::LogMessage::~LogMessage()                                                                     /usr/local/google/home/penghuang/sources/chromium/src/base/logging.cc:592:29
  0006d8af  gl::GLContext::MakeVirtuallyCurrent(gl::GLContext*, gl::GLSurface*)                                    /usr/local/google/home/penghuang/sources/chromium/src/ui/gl/gl_context.cc:323:5
  000db7ef  gpu::gles2::GLES2DecoderImpl::MakeCurrent()                                                            /usr/local/google/home/penghuang/sources/chromium/src/gpu/command_buffer/service/gles2_cmd_decoder.cc:4555:18
  00017903  gpu::CommandBufferStub::MakeCurrent()                                                                  /usr/local/google/home/penghuang/sources/chromium/src/gpu/ipc/service/command_buffer_stub.cc:424:25
  00017759  gpu::CommandBufferStub::OnMessageReceived(IPC::Message const&)                                         /usr/local/google/home/penghuang/sources/chromium/src/gpu/ipc/service/command_buffer_stub.cc:263:10
  0001efb9  gpu::GpuChannel::HandleMessageHelper(IPC::Message const&)                                              /usr/local/google/home/penghuang/sources/chromium/src/gpu/ipc/service/gpu_channel.cc:538:23
  0001e023  gpu::GpuChannel::HandleMessage(IPC::Message const&)                                                    /usr/local/google/home/penghuang/sources/chromium/src/gpu/ipc/service/gpu_channel.cc:514:3
  00050391  base::OnceCallback<void ()>::Run() &&                                                                  /usr/local/google/home/penghuang/sources/chromium/src/base/callback.h:99:12
  00053a1d  gpu::Scheduler::RunNextTask()                                                                          /usr/local/google/home/penghuang/sources/chromium/src/gpu/command_buffer/service/scheduler.cc:526:24
  000c719b  base::OnceCallback<void ()>::Run() &&                                                                  /usr/local/google/home/penghuang/sources/chromium/src/base/callback.h:99:12
  000cf7a3  base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)                                   /usr/local/google/home/penghuang/sources/chromium/src/base/debug/task_annotator.cc:101:33
  000e153f  base::MessageLoop::RunTask(base::PendingTask*)                                                         /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:454:46
  000e174b  base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)                                            /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:465:5
  000e1847  base::MessageLoop::DoWork()                                                                            /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:535:16
  000e3003  base::MessagePumpDefault::Run(base::MessagePump::Delegate*)                                            /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_pump_default.cc:37:31
  000e1307  base::MessageLoop::Run(bool)                                                                           /usr/local/google/home/penghuang/sources/chromium/src/base/message_loop/message_loop.cc:406:12
  000f57f3  base::RunLoop::Run()                                                                                   /usr/local/google/home/penghuang/sources/chromium/src/base/run_loop.cc:102:14
  00855d7f  content::GpuMain(content::MainFunctionParams const&)                                                   /usr/local/google/home/penghuang/sources/chromium/src/content/gpu/gpu_main.cc:346:21
  00fe8ef5  content::ContentMainRunnerImpl::Run(bool)                                                              /usr/local/google/home/penghuang/sources/chromium/src/content/app/content_main_runner_impl.cc:951:10
  0000e93d  service_manager::Main(service_manager::MainParams const&)                                              /usr/local/google/home/penghuang/sources/chromium/src/services/service_manager/embedder/main.cc:472:29
  v------>  content::JNI_ContentMain_Start(_JNIEnv*, base::android::JavaParamRef<_jclass*> const&, unsigned char)  /usr/local/google/home/penghuang/sources/chromium/src/content/app/android/content_main.cc:53:10
  00fe85ff  Java_org_chromium_content_app_ContentMain_nativeStart                                                  /usr/local/google/home/penghuang/sources/chromium/src/out_android/Release/gen/content/public/android/content_jni_headers/content/jni/ContentMain_jni.h:48:0
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so
  00405a79  <UNKNOWN>                                                                                              /system/lib/libart.so

re:#12, yeah, we should early-out ProcessTraces. I realize that we'll call it if any of the GLES2DecoderImpl::HasMoreIdleWork conditions are true. Actually, I think we shouldn't even call decoder_->MakeCurrent as the context was already made current before GLES2DecoderImpl::PerformIdleWork (which calls this).

re:#13: passing --enable-gpu-debugging will check glGetError after every command buffer command, logging the command in case of error, which should narrow down which command causes the error to happen. We can then instrument further that particular command.

Command kBeginQueryEXT from WebGL causes the problem. But it doesn't happen, if I disable the presentation callback(GLSurface will not use GPUTimer).
So what should we do? We may disable the timer query extension for Pixel, but WebGL cannot use it too.

BTW, are WebGL and display compositor sharing the same OS GL context on android? If they share the same OS GL context, maybe some WebGL
calls will affect GLSurfacePresentationHelper.

The Debug log:
07-19 09:35:04.146  2808  2808 D cr_OfflinePageUtils: [OfflinePageUtils.java:168] showReloadSnackbar called with controller org.chromium.chrome.browser.offlinepages.OfflinePageTabObserver$3@1e616eb
07-19 09:35:05.109  7023  7105 I chromium: [INFO:library_loader_hooks.cc(36)] Chromium logging enabled: level = 0, default verbosity = 0
07-19 09:35:08.123  7037  7148 I chromium: [INFO:library_loader_hooks.cc(36)] Chromium logging enabled: level = 0, default verbosity = 0
07-19 09:35:12.617  7259  7259 I chromium: [INFO:library_loader_hooks.cc(36)] Chromium logging enabled: level = 0, default verbosity = 0
07-19 09:35:16.652  3338  3373 E chromium: [ERROR:gles2_cmd_decoder.cc(5644)] [.WebGL-0xcad3ac00] GL ERROR: GL_INVALID_OPERATION : kBeginQueryEXT
07-19 09:35:16.652  3338  3373 E chromium: [ERROR:gles2_cmd_decoder.cc(5647)] [.WebGL-0xcad3ac00]GL ERROR :GL_INVALID_OPERATION : DoCommand: GL error from driver
07-19 09:35:16.653  3338  3373 E chromium: [ERROR:gles2_cmd_decoder.cc(5644)] [.WebGL-0xcad3ac00] GL ERROR: GL_INVALID_OPERATION : kEndQueryEXT
07-19 09:35:16.653  3338  3373 E chromium: [ERROR:gles2_cmd_decoder.cc(5647)] [.WebGL-0xcad3ac00]GL ERROR :GL_INVALID_OPERATION : DoCommand: GL error from driver
07-19 09:35:16.657  2808  2808 I chromium: [INFO:CONSOLE(1190)] "performance warning: READ-usage buffer was read back without waiting on a fence. This caused a graphics pipeline stall.", source: http://khronos.org/registry/webgl/sdk/tests/deqp/functional/gles3/es3fTransformFeedbackTests.js (1190)
07-19 09:35:16.729  3338  3373 E chromium: [ERROR:gles2_cmd_decoder.cc(5644)] [.WebGL-0xcad3ac00] GL ERROR: GL_INVALID_OPERATION : kUnmapBuffer
07-19 09:35:16.729  3338  3373 E chromium: [ERROR:gles2_cmd_decoder.cc(5647)] [.WebGL-0xcad3ac00]GL ERROR :GL_INVALID_OPERATION : DoCommand: GL error from driver

If we use virtualized contexts, then we'll use the same underlying GL context. But timer queries (whether GL_TIME_ELAPSED or GL_TIMESTAMP) shouldn't interact with transform feedback (GL_TRANSFORM_FEEDBACK_PRIMITIVES_WRITTEN), so something is very odd here.
Either we're doing something wrong and reusing query ids without deleting/recreating them (but I don't see evidence of that) or maybe we're running into a driver bug.
Maybe the best way to debug this would be to log calls to glGenQueries/glDeleteQueries/glBeginQuery/glEndQuery on the service side. You can use --enable-gpu-service-logging, but that will log all GL calls (might be hard to make sense out of) and won't log actual ids in glGenQueries/glDeleteQueries
Blockedon: 856858
Labels: -Pri-2 Pri-1
penghuang@ do you have any further progress on figuring out whether Chromium's usage of queries is incorrect, or whether we seem to be running into a driver bug? From  Issue 856858  here is a tryjob of these conformance tests on the Pixel 2 and it shows that the majority of the failing tests are the transform feedback tests:

https://chromium-review.googlesource.com/1115802
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/gpu-manual-try-android-p-pixel-2-32/3

I'm raising this to P1 to reflect the urgency. We would like to start running the WebGL 2.0 conformance tests on Android ASAP and this bug is blocking that.

I found this problem is related to virtualized gl context. If I force chromium to use real gl context instead of virtualized gl context, the problem will not happen. And I found chrome will pause and resume gl queries during switching GLContextVirtuals (see[1]). And seems PauseQueries() and ResumeQueries() will recreate queries. It may have some bug which causes the problem. But I am not familiar to those code, and cannot understand the logic for pause and remove queries and can not figure how to fix it.


[1] https://cs.chromium.org/chromium/src/ui/gl/gl_context.cc?type=cs&q=GLContext&sq=package:chromium&g=0&l=344

I reproduced the OS GL problem with below code. The crash happends with the DCHECK.
But if I remove the glDeleteQueries(), the DCHECK will not crash. Seems glDeleteQueries()
doesn't clear all states associated with the a query id. When the query id is reused, it
may cause problem if the query target is different. 

I think it should be a driver bug, or I missed something and not use the query API correctly?

GLint query_id = 0;
glGenQueries(1, &query_id);
glQueryCounter(gl_query_id_, GL_TIMESTAMP);
glDeleteQueries(1, &query_id);  // If remove this delete, the DCHECK() below will not crash.
glGenQueries(1, &query_id);
glBeginQuery(GL_TIME_ELAPSED, query_id);
DCHECK(glGetError() == 0);  // Crashed here
glEndQuery(GL_TIME_ELAPSED);
glDeleteQueries(1, &query_id);

Cc: ericrk@chromium.org
Components: -Internals>GPU Internals>GPU>VendorSpecific
Labels: GPU-Qualcomm
Yes, that is a driver bug.
I'm not sure there is going to be a good way to work around it without leaking queries.
So what should we do? Should we disable glQueryCounter for qualcomm driver?
Is the issue solely with glQueryCounter? Or does it happen when you use the same query id (even after delete/gen) with a different target?

If the former, I believe we have a path to use GL_TIME_ELAPSED queries instead of GL_TIMESTAMP for timing queries, using GPUTimingClient::ForceTimeElpasedQuery.
Project Member

Comment 24 by bugdroid1@chromium.org, Jul 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c51f537164d29b80ba89ca51fd94721873bd85c

commit 6c51f537164d29b80ba89ca51fd94721873bd85c
Author: Peng Huang <penghuang@chromium.org>
Date: Wed Jul 25 20:58:03 2018

Disable timestamp queries for Qualcomm driver on Android

glDeleteQueries() doesn't reset the states for GL_TIMESTAMP
query id correctly. When the deleted query id is returned from
glGenQueries() again, it cannot be used for other query targets
successfully.

This CL also fixed a bug in GPUTimingImpl::DoTimeStampQuery().

Bug:  858879 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: Ifca73198e1a1cee6412c19df5e087a5f8d2354b4
Reviewed-on: https://chromium-review.googlesource.com/1150511
Reviewed-by: Antoine Labour <piman@chromium.org>
Commit-Queue: Peng Huang <penghuang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578057}
[modify] https://crrev.com/6c51f537164d29b80ba89ca51fd94721873bd85c/gpu/config/gpu_driver_bug_list.json
[modify] https://crrev.com/6c51f537164d29b80ba89ca51fd94721873bd85c/ui/gl/gpu_timing.cc

The CL to workaround the driver issue has been landed. Should we close this issue or assign it to someone who is responsible for fixing this driver issue? 
Status: Verified (was: Assigned)
Thank you penghuang@ for getting to the bottom of this and working around the driver bug! Confirmed that these tests are passing in Chrome Canary 70.0.3503.0.

I've filed internal bug ID 111888307 about the driver bug, so hopefully Qualcomm will triage it. The best way to ensure that this is fixed would be to add a conformance test for it in Khronos' VK-GL-CTS suite:
https://github.com/KhronosGroup/VK-GL-CTS

If you are interested in making a contribution to that suite, please email me directly and I'll help you get set up to do so.

Labels: Merge-Request-69 Merge-Request-68
The change in comment 24 also fixes a bug in gpu_timing.cc. It is needed for fixing issue 811661. So I would like to merge it to M68 and M69.
Project Member

Comment 28 by sheriffbot@chromium.org, Aug 1

Labels: -Merge-Request-68 Hotlist-Merge-Review Merge-Review-68
This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), kariahda@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 29 by sheriffbot@chromium.org, Aug 2

Labels: -Merge-Request-69 Hotlist-Merge-Approved Merge-Approved-69
Your change meets the bar and is auto-approved for M69. Please go ahead and merge the CL to branch 3497 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), kariahda@(iOS), cindyb@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 30 by sheriffbot@chromium.org, Aug 6

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 31 by sheriffbot@chromium.org, Aug 10

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 32 by bugdroid1@chromium.org, Aug 10

Labels: -merge-approved-69 merge-merged-3497
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/00467876ea8e9980bf7bc46a554db4aac223927b

commit 00467876ea8e9980bf7bc46a554db4aac223927b
Author: Peng Huang <penghuang@chromium.org>
Date: Fri Aug 10 18:39:55 2018

Disable timestamp queries for Qualcomm driver on Android

glDeleteQueries() doesn't reset the states for GL_TIMESTAMP
query id correctly. When the deleted query id is returned from
glGenQueries() again, it cannot be used for other query targets
successfully.

This CL also fixed a bug in GPUTimingImpl::DoTimeStampQuery().

TBR=penghuang@chromium.org

(cherry picked from commit 6c51f537164d29b80ba89ca51fd94721873bd85c)

Bug:  858879 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: Ifca73198e1a1cee6412c19df5e087a5f8d2354b4
Reviewed-on: https://chromium-review.googlesource.com/1150511
Reviewed-by: Antoine Labour <piman@chromium.org>
Commit-Queue: Peng Huang <penghuang@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#578057}
Reviewed-on: https://chromium-review.googlesource.com/1171303
Reviewed-by: Peng Huang <penghuang@chromium.org>
Cr-Commit-Position: refs/branch-heads/3497@{#540}
Cr-Branched-From: 271eaf50594eb818c9295dc78d364aea18c82ea8-refs/heads/master@{#576753}
[modify] https://crrev.com/00467876ea8e9980bf7bc46a554db4aac223927b/gpu/config/gpu_driver_bug_list.json
[modify] https://crrev.com/00467876ea8e9980bf7bc46a554db4aac223927b/ui/gl/gpu_timing.cc

Sign in to add a comment