New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 5 users

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2013
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug

Blocking:
issue 180605
issue 175729



Sign in to add a comment

Use client side vertex arrays for DYNAMIC or STREAM usage VBOs on SGX and Mali GPUs

Project Member Reported by bsalomon@chromium.org, Feb 25 2013

Issue description

These tile based deferred rendering GPUs/drivers have problems with VBOs that are updated frequently either via glBufferSubData or glBufferData. The updates trigger shadow copy code paths in the driver which leads to poor performance and potentially OOM errors with content that uses a lot of dynamic vertex data.

The proposed fix is to make the gpu process server internally handle VBOs marked STREAM and/or DYNAMIC as client-side vertex arrays transparently to the command buffer client. In Skia standalone testing using content that is a close mirror of the GUIMark2 Vector test client-side vertex arrays have shown 40-50% reduction in frametime and elimination of OOM errors.

Some of the other ideas that have tried but either didn't fix the OOMs or had other problems:

*Tried all 3 usage params to glBufferData. [No difference]

*Deleting old VBO ID and using a new VBO IDs rather than calling glBufferData with a recycled ID. [OOM fixed but there's a huge hitch on Mali when deleted VBOs are reclaimed by the driver]

*Using a ring with various numbers of VBOs to give the driver time to have fully consumed a VBO before it is reused. [Problem was not ameliorated]


 
Blocking: chromium:175729

Comment 2 by gman@chromium.org, Feb 25 2013

Status: Started
how soon do you need this?

Comment 3 by bsalo...@google.com, Feb 25 2013

Cc: tomhud...@chromium.org wiltzius@chromium.org
TomH and TomW probably have a better idea than I do about how this should be prioritized.
Cc: klo...@chromium.org skyos...@chromium.org
How heavyily do we use VBOs? If this is the win for CC that it was for Skia in isolation, it could be a huge help with our performance and our crash rate on those devices. Sami, is this something we could prototype to see? This strikes me as a bigger possible win than the one RB bug I have on my plate.
The compositor mostly uses static VBOs that are allocated at start-up and remain constant (I just checked with a GL trace that this is still the case). It seems like we wouldn't get a huge benefit from this there.

Looks like Skia's now using client side attribs to work around this, but in the case of canvas those attrib arrays get sent to the GPU process that turns them back into VBOs. Perhaps it would be better to have Skia use VBOs and only use client side arrays in the GPU process?

I can see this also helping WebGL apps that do a lot of VBO manipulation.

Finally, I think it would be worth talking to ARM about this. Maybe they're purposefully ignoring STREAM/DYNAMIC since those flags tend to get used when they shouldn't.

Comment 6 Deleted

Client-side arrays don't work with the GPU process. So within Chromium Skia uses VBOs always. Making the GPU process use client-side arrays to implement VBOs (on select GL drivers) is exactly what is proposed in this bug.

I don't have contacts at ARM but IMG did suggest using client-side arrays for one-time-use vertex data.

[Apologies for posting/deleting/reposting. I accidentally posted from my personal email address.]
Thanks the correction Brian. Now I see why this makes sense.

Do we know where the single-use VBOs are coming from in the renderer? Badly written canvas apps?
In the particular example that caught our attention it is canvas paths that change every frame. It's a stock-type graph rendered from randomly generated data. With path objects we will be able to do some caching of path data (for apps that don't behave like this one). That said, It seems like the single-use vertex data use case for canvas and WebGL will be common enough that it is worth this level of optimization, IMO.
Not understanding the full list of cases this will apply I don't have any real recommendation for priority. If its only path-heavy Canvas apps that's probably less of a big deal than some of the other changes in the GPU process under consideration (such as dropping contexts or virtualizing memory to try to stay more stable on desktops with poor driver memory management)... if its more general then perhaps this is worth doing sooner. Vangelis any thoughts?
Note that this is blocking https://code.google.com/p/chromium/issues/detail?id=175729, which we need to turn on --disable-canvas-aa on Nexus10.
Cc: -gman@chromium.org
Owner: gman@chromium.org
This seems pretty important to resolve and should give us a good perf boost as well. Gregg, any sense of how difficult it would be to implement it? 

Comment 13 by gman@chromium.org, Feb 27 2013

I don't expect it will take too long. I'll start on it ASAP

Comment 14 by gman@chromium.org, Mar 1 2013

Here's a CL in progress that seems to work. Can you test with this and see if it solves the issue? It's hard coded to use client side arrays if GL_STREAM_DRAW is used.

https://codereview.chromium.org/12378034

I need to write a bunch of unit tests and I'd like to refactor a few things before I check it in but I'd like to know it's going to work first.


Thanks, Gregg. I'm trying to test it. I'm having some clank build issues (unrelated to your patch) and will update this as soon as I have some info.

Comment 16 by gman@chromium.org, Mar 1 2013

I uploaded a new patch with a minor fix for unused attributes.

As a test, locally I have a patch that replaces 'usage' in GLES2Implementation::BufferData randomly with GL_STATIC_DRAW, GL_DYNAMIC_DRAW, and GL_STREAM_DRAW. It passes the OpenGL ES 2.0 conformance tests and the WebGL Conformance tests.


I'm getting incorrect rendering with the first patch. I attached the skia side change to use STREAM rather than DYNAMIC (relative to third_party/skia/src). I was loading this URL:

http://www.craftymind.com/factory/guimark2/HTML5ChartingTest.html

I'll try to latest patch.
stream.diff
1.3 KB View Download
I get the same rendering issues with the latest patch but not when switching back to DYNAMIC usage.

Comment 19 by gman@chromium.org, Mar 1 2013

I'll play with that page. Thanks!

Comment 20 by gman@chromium.org, Mar 1 2013

Hmm, I'm not getting any corruption on Linux with my random 'usage' patch on that page.

I tried making it always use GL_STREAM_DRAW and still no corruption. 

I'll try your patch
I should add that I was building clank and running on an N10.

Comment 22 by gman@chromium.org, Mar 1 2013

No corruption with your stream.diff patch either. :-(
I'll upload a patch in a min that will disable AA (which is necessary to simulate how this runs on Android with Stephen's change).
Here is the patch to disable AA. It is relative to third_party/skia/include.

One of the graphs is missing the fill when I run with all three patches but not when I switch back to dynamic (though it eventually stops working with dynamic due to the OOM problem).

Also, not sure if it matters but I've been running a Debug build. I ran with --enable-logging and watched the adb logcat. I didn't see any errors reported by the gpu service.


no_aa.diff
438 bytes View Download

Comment 25 by gman@chromium.org, Mar 1 2013

blarg, I patched in your no_aa patch but it still works fine here. I guess I need to get nexus 10. No idea how long that will take.

Comment 26 by gman@chromium.org, Mar 2 2013

Status update,

So I have it repoed here on a N10. If I hard code in GLES2Implementation to make all buffers GL_STREAM_DRAW nothing draws. Where as that works just fine on my Linux Box :-(

One difference is Android is using virtualized contexts. I'm trying to think how that would muck things up. Maybe I can turn on virtual contexts on linux and see if that repos it there as it's easier to debug there.



Comment 27 by gman@chromium.org, Mar 4 2013

ha, it was running on linux because hardware accelerated canvas is blacklisted on linux :-p  adding --ignore-gpu-blacklist repoed the bug.

I uploaded a new CL. It seems to work. Testing both Linux and N10




Comment 28 by gman@chromium.org, Mar 5 2013

Can you please post the "about:gpu" contents from a SGX and Mali devices here? Thanks
Gregg, I tried to run your patch on my N10 but I'm having some weird issue where the install script can't find the APK. But it sounds like you tested in there already.

I attached a screenshot of the N10's about:gpu (Mali). I'll do the same for the Nexus S shortly (SGX).
N10 about:gpu screenshot.png
430 KB View Download
I couldn't fit all of the info on one screen. Let me know if I cut off anything important.
GS about:gpu screenshot.png
152 KB View Download
Project Member

Comment 31 by bugdroid1@chromium.org, Mar 6 2013

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=186416

------------------------------------------------------------------------
r186416 | gman@chromium.org | 2013-03-06T13:06:50.144194Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder_unittest_1_autogen.h?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/gpu.gyp?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/query_manager_unittest.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/common/gles2_cmd_utils_implementation_autogen.h?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/context_group.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager_unittest.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager.h?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info_unittest.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info.h?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder_autogen.h?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/build_gles2_cmd_buffer.py?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager_unittest.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/test_helper.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager.h?r1=186416&r2=186415&pathrev=186416
   A http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_stream_draw_unittests.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/test_helper.h?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_test_utils.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder.cc?r1=186416&r2=186415&pathrev=186416
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_test_utils.h?r1=186416&r2=186415&pathrev=186416

Use client side arrays for GL_STREAM_DRAW attributes

Certain GPU/drivers are slow when using constantly changing
vertex buffers. They also run out of memory as the pipeline
the buffers so while a buffer is in used being drawn to they
can't delete it immediately when you upload new data to the 
buffer.

This is an attempt to work around that issue seemlessly by
using client side arrays for buffers marked as GL_STREAM_DRAW

BUG= 178093 


Review URL: https://chromiumcodereview.appspot.com/12494005
------------------------------------------------------------------------
Project Member

Comment 32 by bugdroid1@chromium.org, Mar 6 2013

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=186459

------------------------------------------------------------------------
r186459 | zmo@chromium.org | 2013-03-06T17:48:26.576202Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/test_helper.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager.h?r1=186459&r2=186458&pathrev=186459
   D http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_stream_draw_unittests.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/test_helper.h?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_test_utils.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_test_utils.h?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder_unittest_1_autogen.h?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/gpu.gyp?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/query_manager_unittest.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/common/gles2_cmd_utils_implementation_autogen.h?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/context_group.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager_unittest.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager.h?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info_unittest.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info.h?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder_autogen.h?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/build_gles2_cmd_buffer.py?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager_unittest.cc?r1=186459&r2=186458&pathrev=186459
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager.cc?r1=186459&r2=186458&pathrev=186459

Revert 186416
> Use client side arrays for GL_STREAM_DRAW attributes
> 
> Certain GPU/drivers are slow when using constantly changing
> vertex buffers. They also run out of memory as the pipeline
> the buffers so while a buffer is in used being drawn to they
> can't delete it immediately when you upload new data to the 
> buffer.
> 
> This is an attempt to work around that issue seemlessly by
> using client side arrays for buffers marked as GL_STREAM_DRAW
> 
> BUG= 178093 
> 
> 
> Review URL: https://chromiumcodereview.appspot.com/12494005

TBR=gman@chromium.org
Review URL: https://codereview.chromium.org/12544006
------------------------------------------------------------------------

Comment 33 by kbr@chromium.org, Mar 7 2013

Blocking: chromium:180605
Project Member

Comment 34 by bugdroid1@chromium.org, Mar 7 2013

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=186573

------------------------------------------------------------------------
r186573 | gman@chromium.org | 2013-03-07T01:26:08.086279Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_test_utils.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_test_utils.h?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder_unittest_1_autogen.h?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/gpu.gyp?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/query_manager_unittest.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/common/gles2_cmd_utils_implementation_autogen.h?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/context_group.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager_unittest.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder_unittest_1.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/buffer_manager.h?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info_unittest.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/gles2_cmd_decoder_autogen.h?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/feature_info.h?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/build_gles2_cmd_buffer.py?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager_unittest.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/test_helper.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/vertex_attrib_manager.h?r1=186573&r2=186572&pathrev=186573
   A http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/tests/gl_stream_draw_unittests.cc?r1=186573&r2=186572&pathrev=186573
   M http://src.chromium.org/viewvc/chrome/trunk/src/gpu/command_buffer/service/test_helper.h?r1=186573&r2=186572&pathrev=186573

Revert "Revert 186416"

This reverts commit a8fb8f44bc56943c45bd06034fc004e22ef5da85 and
fixes the bug related to the WebGL null object test.

BUG= 178093 
TBR=apatrick@chromium.org

Review URL: https://chromiumcodereview.appspot.com/12542009
------------------------------------------------------------------------
Project Member

Comment 35 by bugdroid1@chromium.org, Mar 10 2013

Labels: -Area-Internals -Feature-GPU-Canvas2D -Feature-GPU-WebGL -Feature-GPU-VendorSpecific -Feature-GPU-Internals Cr-Internals-GPU-Internals Cr-Internals-GPU-WebGL Cr-Internals-GPU-VendorSpecific Cr-Internals-GPU-Canvas2D Cr-Internals

Comment 36 by gman@chromium.org, Mar 14 2013

Status: Fixed
Project Member

Comment 37 by bugdroid1@chromium.org, Apr 10 2013

Labels: -Cr-Internals-GPU-WebGL Cr-Blink-WebGL
Has anyone taken this to ARM to show how terrible their glBufferSubData implementation is when it comes to performance? As of r3p0 on the ARM Chromebook, the performance issues with glBufferData remain in the driver.
I've opened https://b/10616078 for follow-up.
Good to know. Gives a lot of information there.
Here's the response from ARM:

"""The problem here is shared between all the different deferred renderering GPUs and it is due to the deferred nature itself: small updates on buffers that still have not been read by previous drawcalls produce a Copy-on-Write which copies the entire buffer before it is override with the new data.

There could be a possible optimization by making glBufferSubdata asynchronous, which would make it a bit faster, but probably not as fast as client side arrays.

The best option to be faster than client side arrays is to make apps use GLES3 MapBufferRange (map unsynchronized)."""
They gave me more information than "That is how deferred renders work" over in their development forum.

http://forums.arm.com/index.php?/topic/17061-glmapbufferrange-overhead/
With the latest r4p0 drivers that these Mali devices are running, the workaround for glMapBufferRange is no longer necessary.

glBufferSubData performance still isn't fixed(May not cause OOM anymore though?), but the workaround for glMapBufferRange is no longer needed on the Mali series GPUs.
Project Member

Comment 44 by bugdroid1@chromium.org, Jan 24

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f392821d892bfa393f87a102caf133c0b4ed46a2

commit f392821d892bfa393f87a102caf133c0b4ed46a2
Author: Kai Ninomiya <kainino@chromium.org>
Date: Wed Jan 24 19:56:04 2018

Add crbugs to some old GPU driver bug list entries

Add crbugs to the use_client_side_arrays_for_stream_buffers workarounds.

TBR=kbr@chromium.org

Bug:  178093 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I8516cefd80bc96e06f1787f39c405f528fd10e58
Reviewed-on: https://chromium-review.googlesource.com/882489
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kai Ninomiya <kainino@chromium.org>
Commit-Queue: Kai Ninomiya <kainino@chromium.org>
Cr-Commit-Position: refs/heads/master@{#531655}
[modify] https://crrev.com/f392821d892bfa393f87a102caf133c0b4ed46a2/gpu/config/gpu_driver_bug_list.json

Sign in to add a comment