New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 642984 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
OOO until 2019-01-24
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 0
Type: Bug



Sign in to add a comment

context_lost tests started flaking recently on multiple bots

Project Member Reported by kbr@chromium.org, Sep 1 2016

Issue description

The context_lost tests started flaking recently on multiple bots, including:
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29

The tests that fail are those where the GPU process exits, including:
GpuCrash.GPUProcessCrashesExactlyOnce
ContextLost.WebGLContextLostFromGPUProcessExit

It looks like the renderer process is crashing, and that's why DevTools is failing to connect. Here's a log excerpt:

[ RUN      ] GpuCrash.GPUProcessCrashesExactlyOnce
(INFO) 2016-08-31 13:51:33,195 desktop_browser_backend.GetBrowserStartupArgs:250  Requested remote debugging port: 0
(INFO) 2016-08-31 13:51:33,195 desktop_browser_backend.Start:285  Starting Chrome ['/b/s/w/ircgjeBq/out/Debug/Chromium.app/Contents/MacOS/Chromium', '--js-flags=--expose-gc', '--enable-logging=stderr', '--disable-domain-blocking-for-3d-apis', '--disable-gpu-process-crash-limit', '--enable-gpu-benchmarking', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--enable-gpu-benchmarking', '--disable-background-networking', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--window-size=1280,1024', '--user-data-dir=/b/s/w/it_SMOA_/tmpFUTbnU', 'about:blank']
2016-08-31 13:51:33.690 Chromium[33681:304852] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa745139820>
2016-08-31 13:51:33.690 Chromium[33681:304852] Call stack:
(
    "+callStackSymbols disabled for performance reasons"
)
(INFO) 2016-08-31 13:51:33,714 desktop_browser_backend.HasBrowserFinishedLaunching:237  Discovered ephemeral port 63675
(INFO) 2016-08-31 13:51:34,311 browser._LogBrowserInfo:113  OS: mac yosemite
(INFO) 2016-08-31 13:51:34,331 browser._LogBrowserInfo:117  Model: Macmini
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:120  GPU device 0: VENDOR = 0x8086 (Intel), DEVICE = 0xa2e
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:122  GPU Attributes:
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124    adapter_luid        : 0.0
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124    amd_switchable      : False
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124    basic_info_state    : 1
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124    context_info_state  : 1
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124    direct_rendering    : True
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124    driver_date         : 
(INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124    driver_vendor       : 
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    driver_version      : 10.6.33
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    encrypted_only      : False
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_extensions       : GL_ARB_blend_func_extended GL_ARB_draw_buffers_blend GL_ARB_draw_indirect GL_ARB_ES2_compatibility GL_ARB_explicit_attrib_location GL_ARB_gpu_shader_fp64 GL_ARB_gpu_shader5 GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_occlusion_query2 GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_separate_shader_objects GL_ARB_shader_bit_encoding GL_ARB_shader_subroutine GL_ARB_shading_language_include GL_ARB_tessellation_shader GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_cube_map_array GL_ARB_texture_gather GL_ARB_texture_query_lod GL_ARB_texture_rgb10_a2ui GL_ARB_texture_storage GL_ARB_texture_swizzle GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback3 GL_ARB_vertex_attrib_64bit GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_EXT_debug_label GL_EXT_debug_marker GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_texture_compression_s3tc GL_EXT_texture_filter_anisotropic GL_EXT_texture_sRGB_decode GL_APPLE_client_storage GL_APPLE_container_object_shareable GL_APPLE_flush_render GL_APPLE_object_purgeable GL_APPLE_rgb_422 GL_APPLE_row_bytes GL_APPLE_texture_range GL_ATI_texture_mirror_once GL_NV_texture_barrier
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_renderer         : Intel Iris OpenGL Engine
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_reset_notification_strategy: 0
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_vendor           : Intel Inc.
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_version          : 4.1 INTEL-10.6.33
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_ws_extensions    : 
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_ws_vendor        : 
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    gl_ws_version       : 
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    in_process_gpu      : False
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    initialization_time : 0.023583
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    jpeg_decode_accelerator_supported: False
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    lenovo_dcute        : False
(INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124    max_framerate_denominator: 1
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    max_framerate_numerator: 30
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    max_msaa_samples    : 8
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    max_resolution_height: 2160
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    max_resolution_width: 4096
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    min_resolution_height: 16
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    min_resolution_width: 16
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    optimus             : False
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    pixel_shader_version: 4.10
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    process_crash_count : 0
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    profile             : 0
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    sandboxed           : True
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    software_rendering  : False
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    vertex_shader_version: 4.10
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124    video_decode_accelerator_flags: 0
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:126  Feature Status:
(INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:128    2d_canvas           : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    flash_3d            : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    flash_stage3d       : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    flash_stage3d_baseline: enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    gpu_compositing     : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    multiple_raster_threads: enabled_on
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    native_gpu_memory_buffers: enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    rasterization       : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    video_decode        : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    video_encode        : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    vpx_decode          : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128    webgl               : enabled
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:130  Driver Bug Workarounds:
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:132    disable_framebuffer_cmaa
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:132    disable_multimonitor_multisampling
(INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:132    disable_webgl_rgb_multisampling_usage
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    get_frag_data_info_bug
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    msaa_is_slow
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    pack_parameters_workaround_with_pack_buffer
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    rebind_transform_feedback_before_resume
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    regenerate_struct_names
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    reset_base_mipmap_level_before_texstorage
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    scalarize_vec_and_mat_constructor_args
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    set_zero_level_before_generating_mipmap
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    unfold_short_circuit_as_ternary_operation
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    unpack_alignment_workaround_with_unpack_buffer
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    use_intermediary_for_copy_texture_image
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    use_shadowed_tex_level_params
(INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132    validate_multisample_buffer_allocation
(INFO) 2016-08-31 13:51:34,359 cache_temperature.EnsurePageCacheTemperature:55  PageCacheTemperature: any
[33681:103939:0831/135134:WARNING:simple_synchronous_entry.cc(1054)] Could not open platform files for entry.
[33681:1299:0831/135134:INFO:CONSOLE(33)] "Harness injected.", source:  (33)
Received signal 11 SEGV_MAPERR 000000000000
 [0x000115a7774e]
 [0x000115a777b5]
 [0x000115a776b7]
 [0x7fff8bae1f1a]
 [0x7fff5d901b08]
 [0x00011a9db50d]
 [0x00011a9db489]
 [0x00011a9db41d]
 [0x00011a9c940d]
 [0x00011a9c71a7]
 [0x00011aafaaf5]
 [0x00011a9c9fa5]
 [0x00012523fe78]
 [0x000125247f03]
 [0x000125247db6]
 [0x000125247d59]
 [0x00012524785c]
 [0x000115a369ce]
 [0x000115a79284]
 [0x000115b4fcbd]
 [0x000115b502a4]
 [0x000115b50d1d]
 [0x000115b61c08]
 [0x000115b61b8c]
 [0x000115b09c9a]
 [0x000115b61105]
 [0x7fff92967a01]
 [0x7fff92959b8d]
 [0x7fff929591bf]
 [0x7fff92958bd8]
 [0x000115b6226c]
 [0x000115b618ea]
 [0x000115b4f4ca]
 [0x000115c18015]
 [0x00011a9e4d99]
 [0x00011e2685a7]
 [0x00011e26a496]
 [0x00011e267e0d]
 [0x0001023366d3]
 [0x0001022f9d7c]
 [0x0001022f9a64]
 [0x000000000015]
[end of stack trace]
[33691:1299:0831/135135:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[33691:1299:0831/135135:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer.
[33681:1299:0831/135135:ERROR:gpu_process_transport_factory.cc(850)] Lost UI shared context.
[33681:1299:0831/135135:INFO:CONSOLE(0)] "WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost", source: http://127.0.0.1:63683/gpu_process_crash.html (0)
Received signal 11 SEGV_MAPERR 000000000000
 [0x0001187d174e]
 [0x0001187d17b5]
 [0x0001187d16b7]
 [0x7fff8bae1f1a]
 [0x7fff5aba7ae8]
 [0x00011d73550d]
 [0x00011d735489]
 [0x00011d73541d]
 [0x00011d72340d]
 [0x00011d7211a7]
 [0x00011d854af5]
 [0x00011d723fa5]
 [0x000127f99e78]
 [0x000127fa1f03]
 [0x000127fa1db6]
 [0x000127fa1d59]
 [0x000127fa185c]
 [0x0001187909ce]
 [0x0001187d3284]
 [0x0001188a9cbd]
 [0x0001188aa2a4]
 [0x0001188aad1d]
 [0x0001188bbc08]
 [0x0001188bbb8c]
 [0x000118863c9a]
 [0x0001188bb105]
 [0x7fff92967a01]
 [0x7fff92959b8d]
 [0x7fff929591bf]
 [0x7fff92958bd8]
 [0x0001188bc26c]
 [0x0001188bb8ea]
 [0x0001188a94ca]
 [0x000118972015]
 [0x00011d73ed99]
 [0x000120fc25a7]
 [0x000120fc4496]
 [0x000120fc1e0d]
 [0x0001050906d3]
 [0x000105053d7c]
 [0x000105053a64]
 [0x000000000015]
[end of stack trace]
[33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer.
[33681:1299:0831/135141:INFO:CONSOLE(0)] "WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost", source: http://127.0.0.1:63683/gpu_process_crash.html (0)
[33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer.
[33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer.
[33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer.
(INFO) 2016-08-31 13:51:41,688 browser.DumpStateUponFailure:346  *************** BROWSER STANDARD OUTPUT ***************
Can't get standard output with --show-stdout
(INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:348  
(INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:351  *********** END OF BROWSER STANDARD OUTPUT ************
(INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:353  ********************* BROWSER LOG *********************
(INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:355  No log file
(INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:358  ***************** END OF BROWSER LOG ******************
(WARNING) 2016-08-31 13:51:41,689 shared_page_state.DumpStateUponFailure:145  Taking screenshots upon failures disabled.
Waiting for page to finish.
Traceback (most recent call last):
  File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 86, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/s/w/ircgjeBq/content/test/gpu/gpu_tests/gpu_test_base.py", line 111, in RunStory
    RunStoryWithRetries(GpuSharedPageState, self, results)
  File "/b/s/w/ircgjeBq/content/test/gpu/gpu_tests/gpu_test_base.py", line 72, in RunStoryWithRetries
    super(cls, shared_page_state).RunStory(results)
  File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 298, in RunStory
    self._current_page, self._current_tab, results)
  File "/b/s/w/ircgjeBq/content/test/gpu/gpu_tests/context_lost.py", line 112, in ValidateAndMeasurePage
    tab.Activate()
  File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/browser/tab.py", line 86, in Activate
    self._tab_list_backend.ActivateTab(self.id)
  File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/backends/chrome/tab_list_backend.py", line 79, in ActivateTab
    timeout)
  File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 270, in ActivateTab
    timeout=timeout)
  File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 88, in Request
    response = self._conn.getresponse()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1132, in getresponse
    response.begin()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 417, in _read_status
    raise BadStatusLine(line)
DevToolsClientConnectionError: ''

[  FAILED  ] GpuCrash.GPUProcessCrashesExactlyOnce (8503 ms)


I suspect a flaky failure was introduced and this was forced through the CQ via its retry mechanism. The first failing builds on these machines:

https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29/builds/15051
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14474

Unfortunately there is no symbolized stack trace for the renderer process -- I'm not sure why.

I'd suspect either a change in the graphics or DevTools stack. Here are a couple of suspect CLs:

https://chromium.googlesource.com/chromium/src/+/9848a61393772cc8a9aa8349c68f7d436c743369
https://chromium.googlesource.com/chromium/src/+/f629859fdf0b7588a6dc53c8653c67b6766a64da

This is affecting the CQ:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/

Must be addressed immediately via a revert.


 

Comment 1 by kbr@chromium.org, Sep 1 2016

I reverted https://chromium.googlesource.com/chromium/src/+/9848a61393772cc8a9aa8349c68f7d436c743369 in:
https://codereview.chromium.org/2304483002/

Let's see whether that clears up the bots. If not, let's reland that and find the next possible culprit.

Cc: weiliangc@chromium.org
+ pixel wrangler
Cc: fgor...@chromium.org kbr@chromium.org zmo@chromium.org
 Issue 643055  has been merged into this issue.
Labels: Hotlist-PixelWrangler
All bots have cycled back green since revert.

Comment 6 by kbr@chromium.org, Sep 1 2016

Cc: danakj@chromium.org fsam...@chromium.org
Owner: kbr@chromium.org
Status: Fixed (was: Assigned)
That was a pretty lucky guess and I'm still not 100% convinced that the patch was at fault since the last failing build on one of the bots was this one:

https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14483

but there were these three builds where those tests didn't flake:

https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14484
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14485
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14486

before the revert landed:

https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14487

On this other bot the cause and effect are clearer. Last failing build:
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29/builds/15064

first good build:
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29/builds/15065

The first good build contains the revert.

This is fixed. Let's be careful about re-landing that patch since it clearly passes the CQ but breaks the debug Mac bots.

Comment 7 by enne@chromium.org, Sep 1 2016

Thanks for the revert.  I'll take an extremely careful look at that and fix the underlying issue before relanding.

Sign in to add a comment