New issue
Advanced search Search tips

Issue 907195 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug-Regression

Blocked on:
issue 916832

Blocking:
issue 776222



Sign in to add a comment

WebglConformance_conformance_uniforms_no_over_optimization_on_uniform_array_09 failing on Windows 10 Rel

Project Member Reported by rjkroege@chromium.org, Nov 20

Issue description

WebglConformance_conformance_uniforms_no_over_optimization_on_uniform_array_09 is failing on Builder Win10 FYI Release (Intel HD 630) https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20(Intel%20HD%20630)

spew:

[102/137] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_conformance_uniforms_no_over_optimization_on_uniform_array_09 failed unexpectedly 2.0290s:
  
  Traceback (most recent call last):
    _RunGpuTest at content\test\gpu\gpu_tests\gpu_integration_test.py:155
      self.RunActualGpuTest(url, *args)
    RunActualGpuTest at content\test\gpu\gpu_tests\webgl_conformance_integration_test.py:192
      getattr(self, test_name)(test_path, *args[1:])
    _RunConformanceTest at content\test\gpu\gpu_tests\webgl_conformance_integration_test.py:212
      self._CheckTestCompletion()
    _CheckTestCompletion at content\test\gpu\gpu_tests\webgl_conformance_integration_test.py:208
      self.fail(self._WebGLTestMessages(self.tab))
    fail at .swarming_module\bin\Lib\unittest\case.py:410
      raise self.failureException(msg)
  AssertionError: should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  FAIL should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  FAIL should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  
  Locals:
    msg : u'should be green\nat (0, 0) expected: 0,255,0,255 was 0,255,0,0\nFAIL should be green\nat (0, 0) expected: 0,255,0,255 was 0,255,0,0\nshould be green\nat (0, 0) expected: 0,255,0,255 was 0,255,0,0\nFAIL should be green\nat (0, 0) expected: 0,255,0,255 was 0,255,0,0\n'
  
  Found crashpad_database_util
  No minidump found via crashpad_database_util
  Restarting browser due to unexpected test failure
  Closing browser (pid=9004) ...
  Successfully shut down browser cooperatively
  Browser is closed.
  Overriding env[LC_ALL]=="English_United States.UTF-8" with default value "en_US.UTF-8"
  Chrome Env: {'TMP': 'c:\\b\\s\\w\\itxbz_wi', 'LC_NUMERIC': 'English_United States.UTF-8', 'COMPUTERNAME': 'BUILD125-A9', 'VS140COMNTOOLS': 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\Tools\\', 'USERDOMAIN': 'LABS', 'LC_CTYPE': 'English_United States.UTF-8', 'PSMODULEPATH': 'C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'GTEST_TOTAL_SHARDS': '20', 'IPG_DIR': 'C:\\Program Files\\Intel\\Power Gadget 3.5', 'PROCESSOR_REVISION': '9e09', 'HOME': 'c:\\Users\\chrome-bot', 'BOTO_CONFIG': 'c:\\Users\\chrome-bot\\.boto', 'SYSTEMROOT': 'C:\\WINDOWS', 'SWARMING_BOT_ID': 'build125-a9', 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)', 'DRIVERDATA': 'C:\\Windows\\System32\\Drivers\\DriverData', 'VIRTUAL_ENV': 'c:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\8231ed', 'LC_MESSAGES': 'en_US.UTF-8', 'SWARMING_SERVER': 'https://chromium-swarm.appspot.com', 'TEMP': 'c:\\b\\s\\w\\itxbz_wi', 'SWARMING_TASK_ID': '4148f782c1928011', 'LC_MONETARY': 'English_United States.UTF-8', 'CHROME_DEVEL_SANDBOX': '/opt/chromium/chrome_sandbox', 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files', 'PUBLIC': 'C:\\Users\\Public', 'PROCESSOR_ARCHITECTURE': 'AMD64', 'SWARMING_HEADLESS': '1', 'ALLUSERSPROFILE': 'C:\\ProgramData', 'LOCALAPPDATA': 'C:\\Users\\chrome-bot\\AppData\\Local', 'IPG_VER': '3.0', 'HOMEPATH': '\\Users\\chrome-bot', 'LUCI_CONTEXT': 'c:\\b\\s\\w\\itxbz_wi\\luci_ctx.qllyll.json', 'CIPD_CACHE_DIR': 'c:\\b\\s\\cipd_cache\\cache', 'PROGRAMW6432': 'C:\\Program Files', 'USERNAME': 'chrome-bot', 'LANG': 'en_US.UTF-8', 'LC_ALL': 'en_US.UTF-8', 'LOGONSERVER': '\\\\AD1-B', 'PROMPT': '$P$G', 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', 'PROGRAMDATA': 'C:\\ProgramData', 'PYTHONPATH': 'c:\\b\\s\\w\\ir\\third_party\\catapult\\telemetry', 'PYTHONNOUSERSITE': '1', 'ONEDRIVE': 'C:\\Users\\chrome-bot\\OneDrive', 'GTEST_SHARD_INDEX': '3', 'VPYTHON_VIRTUALENV_ROOT': 'c:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython', 'PATH': 'c:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\8231ed\\Scripts;c:\\b\\s\\w\\ir\\.swarming_module;c:\\b\\s\\w\\ir\\.swarming_module\\bin;c:\\b\\s\\cipd_cache\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit\\;C:\\Tools;C:\\CMake\\bin;C:\\Program Files\\Puppet Labs\\Puppet\\bin;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\chrome-bot\\AppData\\Local\\Microsoft\\WindowsApps;c:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\8231ed\\lib\\site-packages\\pywin32_system32;c:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\8231ed\\lib\\site-packages\\pywin32_system32;c:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\8231ed\\lib\\site-packages\\pywin32_system32;c:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\8231ed\\lib\\site-packages\\pywin32_system32', 'USERDNSDOMAIN': 'LABS.CHROMIUM.ORG', 'NO_GCE_CHECK': 'False', 'CHROME_HEADLESS': '1', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC', 'TMPDIR': 'c:\\b\\s\\w\\itxbz_wi', 'SESSIONNAME': 'Console', 'WINDIR': 'C:\\WINDOWS', 'USERDOMAIN_ROAMINGPROFILE': 'LABS', 'HOMEDRIVE': 'C:', 'CIPD_PROTOCOL': 'v2', 'PROGRAMFILES': 'C:\\Program Files', 'LC_COLLATE': 'English_United States.UTF-8', 'PROCESSOR_IDENTIFIER': 'Intel64 Family 6 Model 158 Stepping 9, GenuineIntel', 'NUMBER_OF_PROCESSORS': '8', 'APPDATA': 'C:\\Users\\chrome-bot\\AppData\\Roaming', 'PROCESSOR_LEVEL': '6', 'LC_TIME': 'English_United States.UTF-8', 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', 'OS': 'Windows_NT', 'SYSTEMDRIVE': 'C:', 'BREAKPAD_DUMP_LOCATION': 'c:\\b\\s\\w\\itxbz_wi\\tmp6jf5gl', 'USERPROFILE': 'C:\\Users\\chrome-bot'}
  Starting Chrome ['c:\\b\\s\\w\\ir\\out\\Release\\chrome.exe', '--use-gl=angle', '--disable-gpu-watchdog', '--enable-experimental-web-platform-features', '--use-angle=gl', '--disable-gpu-process-for-dx12-vulkan-info-collection', '--disable-domain-blocking-for-3d-apis', '--test-type=gpu', '--disable-gpu-process-crash-limit', '--use-cmd-decoder=passthrough', '--disable-blink-features=WebXR', '--js-flags=--expose-gc', '--enable-logging=stderr', '--autoplay-policy=no-user-gesture-required', '--disable-features=UseSurfaceLayerForVideo', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--ignore-background-tasks', '--enable-gpu-benchmarking', '--deny-permission-prompts', '--autoplay-policy=no-user-gesture-required', '--disable-background-networking', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--disable-search-geolocation-disclosure', '--proxy-server=socks://localhost:53204', '--proxy-bypass-list=<-loopback>', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--disable-component-update', '--window-size=1280,1024', '--user-data-dir=c:\\b\\s\\w\\itxbz_wi\\tmpf186cd', 'about:blank']
  DoNothingForwarder started between 127.0.0.1:53474 and 53474
  Browser started (pid=7596).
  OS: win win10
  Detailed OS version: 10.0.17134
  Browser command line: "c:\b\s\w\ir\out\Release\chrome.exe" --use-gl=angle --disable-gpu-watchdog --enable-experimental-web-platform-features --use-angle=gl --disable-gpu-process-for-dx12-vulkan-info-collection --disable-domain-blocking-for-3d-apis --test-type=gpu --disable-gpu-process-crash-limit --use-cmd-decoder=passthrough --disable-blink-features=WebXR --js-flags=--expose-gc --enable-logging=stderr --autoplay-policy=no-user-gesture-required --disable-features=UseSurfaceLayerForVideo --enable-net-benchmarking --metrics-recording-only --no-default-browser-check --no-first-run --ignore-background-tasks --enable-gpu-benchmarking --deny-permission-prompts --autoplay-policy=no-user-gesture-required --disable-background-networking --disable-component-extensions-with-background-pages --disable-default-apps --disable-search-geolocation-disclosure --proxy-server=socks://localhost:53204 --proxy-bypass-list=<-loopback> --remote-debugging-port=0 --enable-crash-reporter-for-testing --disable-component-update --window-size=1280,1024 --user-data-dir="c:\b\s\w\itxbz_wi\tmpf186cd" --flag-switches-begin --flag-switches-end --file-url-path-alias="/gen=c:\b\s\w\ir\out\Release\gen" about:blank
  GPU device 0: VENDOR = 0x8086 (Google Inc.), DEVICE = 0x5912 (ANGLE (Intel, Intel(R) HD Graphics 630, OpenGL 4.5 core))
  GPU Attributes:
    amd_switchable      : False
    can_support_threaded_texture_mailbox: False
    direct_composition  : False
    direct_rendering    : True
    dx12_feature_level  : 0
    encrypted_only      : False
    gl_extensions       : GL_ANGLE_client_arrays GL_ANGLE_depth_texture GL_ANGLE_explicit_context GL_ANGLE_explicit_context_gles1 GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_memory_size GL_ANGLE_multi_draw GL_ANGLE_program_cache_control GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_rectangle GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_color_buffer_float_rgb GL_CHROMIUM_color_buffer_float_rgba GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_multisample_compatibility GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_sRGB_write_control GL_EXT_shader_texture_lod GL_EXT_texture_compression_bptc GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_sRGB_decode GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_KHR_parallel_shader_compile GL_NV_pack_subimage GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_fbo_render_mipmap GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_surfaceless_context GL_OES_texture_border_clamp GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object OES_compressed_EAC_R11_signed_texture OES_compressed_EAC_R11_unsigned_texture OES_compressed_EAC_RG11_signed_texture OES_compressed_EAC_RG11_unsigned_texture OES_compressed_ETC2_RGB8_texture OES_compressed_ETC2_RGBA8_texture OES_compressed_ETC2_punchthroughA_RGBA8_texture OES_compressed_ETC2_punchthroughA_sRGB8_alpha_texture OES_compressed_ETC2_sRGB8_alpha8_texture OES_compressed_ETC2_sRGB8_texture 
    gl_renderer         : ANGLE (Intel, Intel(R) HD Graphics 630, OpenGL 4.5 core)
    gl_reset_notification_strategy: 33362
    gl_vendor           : Google Inc.
    gl_version          : OpenGL ES 2.0 (ANGLE 2.1.0.b81ff9e6c518)
    gl_ws_extensions    : EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_KHR_get_all_proc_addresses EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXT_pixel_format_float EGL_KHR_surfaceless_context EGL_ANGLE_display_texture_share_group EGL_ANGLE_create_context_client_arrays EGL_ANGLE_program_cache_control EGL_ANGLE_robust_resource_initialization EGL_ANGLE_create_context_extensions_enabled EGL_ANDROID_blob_cache 
    gl_ws_vendor        : 
    gl_ws_version       : 1.4 (ANGLE 2.1.0.b81ff9e6c518)
    in_process_gpu      : False
    initialization_time : 0.146397
    jpeg_decode_accelerator_supported: False
    max_framerate_denominator: 1
    max_framerate_numerator: 30
    max_msaa_samples    : 16
    max_resolution_height: 2176
    max_resolution_width: 3840
    min_resolution_height: 48
    min_resolution_width: 48
    oop_rasterization_supported: True
    optimus             : False
    passthrough_cmd_decoder: True
    pixel_shader_version: 1.00
    process_crash_count : 0
    profile             : 3
    sandboxed           : True
    software_rendering  : False
    supports_dx12       : False
    supports_overlays   : False
    supports_vulkan     : False
    vertex_shader_version: 1.00
    video_decode_accelerator_flags: 0
    vulkan_version      : 0
  Feature Status:
    2d_canvas           : enabled
    flash_3d            : enabled
    flash_stage3d       : enabled
    flash_stage3d_baseline: enabled
    gpu_compositing     : enabled
    multiple_raster_threads: enabled_on
    native_gpu_memory_buffers: disabled_software
    oop_rasterization   : disabled_off
    protected_video_decode: enabled
    rasterization       : enabled
    skia_deferred_display_list: disabled_off
    skia_renderer       : disabled_off
    surface_control     : disabled_off
    surface_synchronization: enabled_on
    video_decode        : enabled
    viz_display_compositor: enabled_on
    webgl               : enabled
    webgl2              : enabled
  Driver Bug Workarounds:
    clear_uniforms_before_first_program_use
    decode_encode_srgb_for_generatemipmap
    disable_discard_framebuffer
    disable_framebuffer_cmaa
    exit_on_context_lost
    force_cube_complete
    msaa_is_slow
    scalarize_vec_and_mat_constructor_args
    disabled_extension__gl__khr_blend_equation_advanced
    disabled_extension__gl__khr_blend_equation_advanced_coherent
  Traceback (most recent call last):
    File "c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\testing\serially_executed_browser_test_case.py", line 214, in <lambda>
      return lambda self: based_method(self, *args)
    File "c:\b\s\w\ir\content\test\gpu\gpu_tests\gpu_integration_test.py", line 155, in _RunGpuTest
      self.RunActualGpuTest(url, *args)
    File "c:\b\s\w\ir\content\test\gpu\gpu_tests\webgl_conformance_integration_test.py", line 192, in RunActualGpuTest
      getattr(self, test_name)(test_path, *args[1:])
    File "c:\b\s\w\ir\content\test\gpu\gpu_tests\webgl_conformance_integration_test.py", line 212, in _RunConformanceTest
      self._CheckTestCompletion()
    File "c:\b\s\w\ir\content\test\gpu\gpu_tests\webgl_conformance_integration_test.py", line 208, in _CheckTestCompletion
      self.fail(self._WebGLTestMessages(self.tab))
  AssertionError: should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  FAIL should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  FAIL should be green
  at (0, 0) expected: 0,255,0,255 was 0,255,0,0
  
DevTools console [network]: Failed to load resource: the server responded with a status of 404 (File not found) http://127.0.0.1:53224/favicon.ico

 
Cc: shaobo....@intel.com
First failure for both with and without Exp happened with https://chromium-review.googlesource.com/c/1235335. 

I reverted (https://chromium-review.googlesource.com/c/chromium/src/+/1344251) and will update this bug appropriately.


Cc: kainino@chromium.org
Blocking: 776222
Owner: shaobo....@intel.com
Status: Assigned (was: Available)
Thanks for the cc! With this reverted, assigning to Shaobo to investigate.
Components: -Internals>GPU Blink>WebGL
Labels: -Type-Bug -Pri-3 -Hotlist-PixelWrangler Pri-2 Type-Bug-Regression
Shaobo: Based on the run history - 4 consistent failures followed by a pass as soon as it was reverted, this seems like it was caused by your patch. You can try running this test locally to try to reproduce, by just going to:
https://www.khronos.org/registry/webgl/sdk/tests/conformance/uniforms/no-over-optimization-on-uniform-array-09.html?webglVersion=2
in your local build of chrome, with
--use-angle=gl --use-cmd-decoder=passthrough
Kai:
Got it and I'll see what happened.
Kai:
I did some tests on my local machine with HD630 graphics card and the flags you provided. But I can always pass this case. Here are some infos:
Command Line	"C:\Users\shaoboya\Desktop\609884\chrome.exe" --use-gl=angle --use-angle=gl --use-cmd-decoder=passthrough --flag-switches-begin --enable-webgl-draft-extensions --enable-webvr --enable-features=OpenVR --flag-switches-end --file-url-path-alias="/gen=C:\Users\shaoboya\Desktop\609884\gen"

And GPU info:
GPU0	VENDOR = 0x8086 [Google Inc.], DEVICE= 0x3e92 [ANGLE (Intel, Intel(R) UHD Graphics 630, OpenGL 4.5 core)] *ACTIVE*

When I run case https://www.khronos.org/registry/webgl/sdk/tests/conformance/uniforms/no-over-optimization-on-uniform-array-09.html?webglVersion=2
with my patch applied to latest master, this case always pass.




I noticed your path, 609884\chrome.exe. Is that Chrome revision 609884? That one is after the revert, which was r609720.
Oh, sorry for not understanding what you want me to do. I just update chromium to latest(which this patch is reverted) and apply the patch again to build a local chromium. 
I'll try to reset the chromium before r609720

Oh I see, I didn't realize you had reapplied the patch.

In that case, I don't know what's wrong here. You can check the "Browser command line" above for the full list of flags, but I don't know if it will help.
You can also try to reproduce the bot run using the repro info from the Swarming task page:

Reproducing the task locally

Download inputs files into directory foo:
# (if needed) git clone https://chromium.googlesource.com/infra/luci/client-py
python ./client-py/isolateserver.py download -I https://isolateserver.appspot.com --namespace default-gzip -s be7c287ecfe497e5abbef890b14025883819dae2 --target foo

Run this task locally:
# (if needed) git clone https://chromium.googlesource.com/infra/luci/client-py
python ./client-py/swarming.py reproduce -S chromium-swarm.appspot.com 4148ef281d6b9a10

Download output results into directory foo:
# (if needed) git clone https://chromium.googlesource.com/infra/luci/client-py
python ./client-py/swarming.py collect -S chromium-swarm.appspot.com --task-output-dir=foo 4148ef281d6b9a10

Thx for the info !
I'll try to reproducing the task locally!
Hi, Kai,
Finally got some progress on this. Thx for your info about swarming test, it helps a lot. And the failed cases should from tex-2d-luminance_alpha-luminance_alpha-unsigned_byte. Since this is about texImage2D and texSubImage2D, it makes sense that my patch has some faults. I'll investigated it.
Sorry for the misunderstanding of the output. The Fail case still WebglConformance_conformance_uniforms_no_over_optimization_on_uniform_array_09.html according to the final result json file. 

But it is weired that I observed other case failed on the console but not in the final result.

I'll do further investigation to reproduce this failing.(Using all flags not works for me ....)
Shaobo, thanks for looking into this. I'm trying to see if we can reproduce this in a tryjob on Gerrit - it could make it easier for you to investigate.
Experimenting on this CL:
https://chromium-review.googlesource.com/c/chromium/src/+/1352451

For the record, the original failing runs were:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28Intel%20HD%20630%29/2750
through
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28Intel%20HD%20630%29/2753
Note also that it was the webgl2_conformance_gl_passthrough_tests test suite which was failing, so the command line flags:
  --use-angle=gl --use-cmd-decoder=passthrough
will be needed to reproduce.

You will probably also want to see which tests ran just before this one; I have a feeling that it was one of those tests which set OpenGL state which caused this one to fail.

You may also want to run the webgl_conformance_integration_test.py test and use:
  --total-shards=20 --shard-index=3
to try to reproduce exactly the test order on the bots.

Shaobo: if the --total-shards=2 --shard-index=2 works for you, that would be best.

But if not, it looks like it is reproducible in a tryjob on win_angle_rel_ng:
https://chromium-review.googlesource.com/c/chromium/src/+/1352451
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win_angle_rel_ng/4321

So you can use that to try to debug. Maybe by adding debug printing or something.

(I think you will need to create your own CL to do this. The old patch needed a rebase, so you can just copy mine:

git fetch https://chromium.googlesource.com/chromium/src refs/changes/51/1352451/2
git checkout -b your_branch_name FETCH_HEAD
git cl upload

should re-upload it as a separate patch.)
Ken and Kai,
Thx for your helps!
Kai and Ken,
Just update the latest status, by adding another feature --read-abbreviated-json-results-from=content/test/data/gpu/webgl2_conformance_tests_output.json with --total-shards=20 and --shards-index=3, I can finally reproduce this failure in local(Big thx for your helps again).

And by try --test-fitler, it proves that the failure is indeed related with the order of test cases.

I think that's a good start, I can finally debug into the chromium and see what happened.
Update the latest status, I've narrowed down the suit cases and can reproduce this issue every time.
Sorry for slow progress. Occupied by other tasks and plan to solve this next week. Sorry again.
Hi, Ken and Kai,
Things became interesting here.
Per my investigation, the failure is caused by started browser with --use-gl=angle --use-angle=gl --use-cmd-decoder=passthrough and run webgl/sdk/tests/conformance/ogles/GL/biuDepthRange/biuDepthRange_001_to_002.html?webglVersion=2&quiet=0 test first , and then run webgl/sdk/tests/conformance/uniforms/no-over-optimization-on-uniform-array-09.html?webglVersion=2&quiet=0

But what'more, this failure could be reproduced on latest canary(73.0.3643.0) latest stable(71.0.3578.98 ) on Intel Platform(HD630) too !(without this reverted patch)

My patch seems reorder the test cases order and trigger this failure(See the attachment, result-without-this-patch.txt and result-with-this-patch.txt are all shard 3 tests).

So it seems that the root cause is not related with the reverted patch. Maybe you can help to double check it.(And NVidia and AMD don't have this problem on stable, seems like a Intel driver issue)

BR,
Yan, Shaobo
result-without-patch.txt
24.4 KB View Download
result-with-patch.txt
26.9 KB View Download
BTW, the other finding is that, if run webgl/sdk/tests/conformance/ogles/GL/biuDepthRange/biuDepthRange_001_to_002.html?webglVersion=2&quiet=0 twice, the failure won't happened on Intel Platform(seems more like a driver bug)
Shaobo: please keep investigating. I don't think your CL https://chromium-review.googlesource.com/1235335 should have reordered the test cases; that only happens during a WebGL conformance roll.

We can't reland your patch with these test failures, and it would be really unfortunate to just suppress them without knowing what's going on. If you could help debug them - or talk with Intel's OpenGL driver team and try to get the underlying bug fixed - we would greatly appreciate it.

Hi, Ken,
Sure, I'm happy to investigate the root cause and try to inform our OpenGL driver team to fix it with this chance.
Just want you know that it is a existence issue.

BR,
Yan, Shaobo
And for curious, I did some more investigation why this CL  https://chromium-review.googlesource.com/1235335 cause test cases reordered. It turns out to be that the modification of file https://chromium-review.googlesource.com/c/chromium/src/+/1235335/19/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py make it.
If I revert this change, the test cases order is the same as "result-without-patch.txt"
result-without-python-modification.txt
24.4 KB View Download
Ah, I see - your CL adds a new extension test, which implicitly changes the ordering of everything else. Sorry about my confusion there.

Please do investigate a little more why conformance/ogles/GL/biuDepthRange/biuDepthRange_001_to_002.html is causing the subsequent test to fail. Thanks.

Sure and happy to continue investigation and fix this bug !
Hi, Ken,
Double check with Intel Latest Driver 25.20.100.6444 and the issue has been fixed already. https://downloadcenter.intel.com/download/28139/Intel-Graphics-Driver-for-Windows-10?product=80939

BR,
Yan, Shaobo
Blockedon: 916832
Thanks Shaobo. Have started the process of upgrading our fleet to 25.20.100.6444 in Issue 916832.

Do you know what the driver bug was?

Sorry, don't know it. Just update my driver to latest and find the issue has been fixed.

I can try to find it if you feel necessary.

BR,
Yan, Shaobo
OK, not necessary.

Cc: yang...@intel.com

Sign in to add a comment