context_lost tests started flaking recently on multiple bots |
|||||
Issue descriptionThe context_lost tests started flaking recently on multiple bots, including: https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29 https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29 The tests that fail are those where the GPU process exits, including: GpuCrash.GPUProcessCrashesExactlyOnce ContextLost.WebGLContextLostFromGPUProcessExit It looks like the renderer process is crashing, and that's why DevTools is failing to connect. Here's a log excerpt: [ RUN ] GpuCrash.GPUProcessCrashesExactlyOnce (INFO) 2016-08-31 13:51:33,195 desktop_browser_backend.GetBrowserStartupArgs:250 Requested remote debugging port: 0 (INFO) 2016-08-31 13:51:33,195 desktop_browser_backend.Start:285 Starting Chrome ['/b/s/w/ircgjeBq/out/Debug/Chromium.app/Contents/MacOS/Chromium', '--js-flags=--expose-gc', '--enable-logging=stderr', '--disable-domain-blocking-for-3d-apis', '--disable-gpu-process-crash-limit', '--enable-gpu-benchmarking', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--enable-gpu-benchmarking', '--disable-background-networking', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--window-size=1280,1024', '--user-data-dir=/b/s/w/it_SMOA_/tmpFUTbnU', 'about:blank'] 2016-08-31 13:51:33.690 Chromium[33681:304852] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x7fa745139820> 2016-08-31 13:51:33.690 Chromium[33681:304852] Call stack: ( "+callStackSymbols disabled for performance reasons" ) (INFO) 2016-08-31 13:51:33,714 desktop_browser_backend.HasBrowserFinishedLaunching:237 Discovered ephemeral port 63675 (INFO) 2016-08-31 13:51:34,311 browser._LogBrowserInfo:113 OS: mac yosemite (INFO) 2016-08-31 13:51:34,331 browser._LogBrowserInfo:117 Model: Macmini (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:120 GPU device 0: VENDOR = 0x8086 (Intel), DEVICE = 0xa2e (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:122 GPU Attributes: (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124 adapter_luid : 0.0 (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124 amd_switchable : False (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124 basic_info_state : 1 (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124 context_info_state : 1 (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124 direct_rendering : True (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124 driver_date : (INFO) 2016-08-31 13:51:34,332 browser._LogBrowserInfo:124 driver_vendor : (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 driver_version : 10.6.33 (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 encrypted_only : False (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_extensions : GL_ARB_blend_func_extended GL_ARB_draw_buffers_blend GL_ARB_draw_indirect GL_ARB_ES2_compatibility GL_ARB_explicit_attrib_location GL_ARB_gpu_shader_fp64 GL_ARB_gpu_shader5 GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_occlusion_query2 GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_separate_shader_objects GL_ARB_shader_bit_encoding GL_ARB_shader_subroutine GL_ARB_shading_language_include GL_ARB_tessellation_shader GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_cube_map_array GL_ARB_texture_gather GL_ARB_texture_query_lod GL_ARB_texture_rgb10_a2ui GL_ARB_texture_storage GL_ARB_texture_swizzle GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback3 GL_ARB_vertex_attrib_64bit GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_EXT_debug_label GL_EXT_debug_marker GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_texture_compression_s3tc GL_EXT_texture_filter_anisotropic GL_EXT_texture_sRGB_decode GL_APPLE_client_storage GL_APPLE_container_object_shareable GL_APPLE_flush_render GL_APPLE_object_purgeable GL_APPLE_rgb_422 GL_APPLE_row_bytes GL_APPLE_texture_range GL_ATI_texture_mirror_once GL_NV_texture_barrier (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_renderer : Intel Iris OpenGL Engine (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_reset_notification_strategy: 0 (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_vendor : Intel Inc. (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_version : 4.1 INTEL-10.6.33 (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_ws_extensions : (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_ws_vendor : (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 gl_ws_version : (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 in_process_gpu : False (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 initialization_time : 0.023583 (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 jpeg_decode_accelerator_supported: False (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 lenovo_dcute : False (INFO) 2016-08-31 13:51:34,333 browser._LogBrowserInfo:124 max_framerate_denominator: 1 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 max_framerate_numerator: 30 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 max_msaa_samples : 8 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 max_resolution_height: 2160 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 max_resolution_width: 4096 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 min_resolution_height: 16 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 min_resolution_width: 16 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 optimus : False (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 pixel_shader_version: 4.10 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 process_crash_count : 0 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 profile : 0 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 sandboxed : True (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 software_rendering : False (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 vertex_shader_version: 4.10 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:124 video_decode_accelerator_flags: 0 (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:126 Feature Status: (INFO) 2016-08-31 13:51:34,334 browser._LogBrowserInfo:128 2d_canvas : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 flash_3d : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 flash_stage3d : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 flash_stage3d_baseline: enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 gpu_compositing : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 multiple_raster_threads: enabled_on (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 native_gpu_memory_buffers: enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 rasterization : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 video_decode : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 video_encode : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 vpx_decode : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:128 webgl : enabled (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:130 Driver Bug Workarounds: (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:132 disable_framebuffer_cmaa (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:132 disable_multimonitor_multisampling (INFO) 2016-08-31 13:51:34,335 browser._LogBrowserInfo:132 disable_webgl_rgb_multisampling_usage (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 get_frag_data_info_bug (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 msaa_is_slow (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 pack_parameters_workaround_with_pack_buffer (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 rebind_transform_feedback_before_resume (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 regenerate_struct_names (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 reset_base_mipmap_level_before_texstorage (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 scalarize_vec_and_mat_constructor_args (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 set_zero_level_before_generating_mipmap (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 unfold_short_circuit_as_ternary_operation (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 unpack_alignment_workaround_with_unpack_buffer (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 use_intermediary_for_copy_texture_image (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 use_shadowed_tex_level_params (INFO) 2016-08-31 13:51:34,336 browser._LogBrowserInfo:132 validate_multisample_buffer_allocation (INFO) 2016-08-31 13:51:34,359 cache_temperature.EnsurePageCacheTemperature:55 PageCacheTemperature: any [33681:103939:0831/135134:WARNING:simple_synchronous_entry.cc(1054)] Could not open platform files for entry. [33681:1299:0831/135134:INFO:CONSOLE(33)] "Harness injected.", source: (33) Received signal 11 SEGV_MAPERR 000000000000 [0x000115a7774e] [0x000115a777b5] [0x000115a776b7] [0x7fff8bae1f1a] [0x7fff5d901b08] [0x00011a9db50d] [0x00011a9db489] [0x00011a9db41d] [0x00011a9c940d] [0x00011a9c71a7] [0x00011aafaaf5] [0x00011a9c9fa5] [0x00012523fe78] [0x000125247f03] [0x000125247db6] [0x000125247d59] [0x00012524785c] [0x000115a369ce] [0x000115a79284] [0x000115b4fcbd] [0x000115b502a4] [0x000115b50d1d] [0x000115b61c08] [0x000115b61b8c] [0x000115b09c9a] [0x000115b61105] [0x7fff92967a01] [0x7fff92959b8d] [0x7fff929591bf] [0x7fff92958bd8] [0x000115b6226c] [0x000115b618ea] [0x000115b4f4ca] [0x000115c18015] [0x00011a9e4d99] [0x00011e2685a7] [0x00011e26a496] [0x00011e267e0d] [0x0001023366d3] [0x0001022f9d7c] [0x0001022f9a64] [0x000000000015] [end of stack trace] [33691:1299:0831/135135:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer. [33691:1299:0831/135135:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer. [33681:1299:0831/135135:ERROR:gpu_process_transport_factory.cc(850)] Lost UI shared context. [33681:1299:0831/135135:INFO:CONSOLE(0)] "WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost", source: http://127.0.0.1:63683/gpu_process_crash.html (0) Received signal 11 SEGV_MAPERR 000000000000 [0x0001187d174e] [0x0001187d17b5] [0x0001187d16b7] [0x7fff8bae1f1a] [0x7fff5aba7ae8] [0x00011d73550d] [0x00011d735489] [0x00011d73541d] [0x00011d72340d] [0x00011d7211a7] [0x00011d854af5] [0x00011d723fa5] [0x000127f99e78] [0x000127fa1f03] [0x000127fa1db6] [0x000127fa1d59] [0x000127fa185c] [0x0001187909ce] [0x0001187d3284] [0x0001188a9cbd] [0x0001188aa2a4] [0x0001188aad1d] [0x0001188bbc08] [0x0001188bbb8c] [0x000118863c9a] [0x0001188bb105] [0x7fff92967a01] [0x7fff92959b8d] [0x7fff929591bf] [0x7fff92958bd8] [0x0001188bc26c] [0x0001188bb8ea] [0x0001188a94ca] [0x000118972015] [0x00011d73ed99] [0x000120fc25a7] [0x000120fc4496] [0x000120fc1e0d] [0x0001050906d3] [0x000105053d7c] [0x000105053a64] [0x000000000015] [end of stack trace] [33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer. [33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer. [33681:1299:0831/135141:INFO:CONSOLE(0)] "WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost", source: http://127.0.0.1:63683/gpu_process_crash.html (0) [33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer. [33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer. [33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer. [33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer. [33695:1299:0831/135141:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer. [33695:1299:0831/135141:ERROR:context_provider_command_buffer.cc(163)] GpuChannelHost failed to create command buffer. (INFO) 2016-08-31 13:51:41,688 browser.DumpStateUponFailure:346 *************** BROWSER STANDARD OUTPUT *************** Can't get standard output with --show-stdout (INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:348 (INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:351 *********** END OF BROWSER STANDARD OUTPUT ************ (INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:353 ********************* BROWSER LOG ********************* (INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:355 No log file (INFO) 2016-08-31 13:51:41,689 browser.DumpStateUponFailure:358 ***************** END OF BROWSER LOG ****************** (WARNING) 2016-08-31 13:51:41,689 shared_page_state.DumpStateUponFailure:145 Taking screenshots upon failures disabled. Waiting for page to finish. Traceback (most recent call last): File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 86, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/s/w/ircgjeBq/content/test/gpu/gpu_tests/gpu_test_base.py", line 111, in RunStory RunStoryWithRetries(GpuSharedPageState, self, results) File "/b/s/w/ircgjeBq/content/test/gpu/gpu_tests/gpu_test_base.py", line 72, in RunStoryWithRetries super(cls, shared_page_state).RunStory(results) File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 298, in RunStory self._current_page, self._current_tab, results) File "/b/s/w/ircgjeBq/content/test/gpu/gpu_tests/context_lost.py", line 112, in ValidateAndMeasurePage tab.Activate() File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/browser/tab.py", line 86, in Activate self._tab_list_backend.ActivateTab(self.id) File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/backends/chrome/tab_list_backend.py", line 79, in ActivateTab timeout) File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 270, in ActivateTab timeout=timeout) File "/b/s/w/ircgjeBq/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 88, in Request response = self._conn.getresponse() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1132, in getresponse response.begin() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 453, in begin version, status, reason = self._read_status() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 417, in _read_status raise BadStatusLine(line) DevToolsClientConnectionError: '' [ FAILED ] GpuCrash.GPUProcessCrashesExactlyOnce (8503 ms) I suspect a flaky failure was introduced and this was forced through the CQ via its retry mechanism. The first failing builds on these machines: https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29/builds/15051 https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14474 Unfortunately there is no symbolized stack trace for the renderer process -- I'm not sure why. I'd suspect either a change in the graphics or DevTools stack. Here are a couple of suspect CLs: https://chromium.googlesource.com/chromium/src/+/9848a61393772cc8a9aa8349c68f7d436c743369 https://chromium.googlesource.com/chromium/src/+/f629859fdf0b7588a6dc53c8653c67b6766a64da This is affecting the CQ: https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/ Must be addressed immediately via a revert.
,
Sep 1 2016
+ pixel wrangler
,
Sep 1 2016
Issue 643055 has been merged into this issue.
,
Sep 1 2016
,
Sep 1 2016
All bots have cycled back green since revert.
,
Sep 1 2016
That was a pretty lucky guess and I'm still not 100% convinced that the patch was at fault since the last failing build on one of the bots was this one: https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14483 but there were these three builds where those tests didn't flake: https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14484 https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14485 https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14486 before the revert landed: https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/14487 On this other bot the cause and effect are clearer. Last failing build: https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29/builds/15064 first good build: https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Debug%20%28Intel%29/builds/15065 The first good build contains the revert. This is fixed. Let's be careful about re-landing that patch since it clearly passes the CQ but breaks the debug Mac bots.
,
Sep 1 2016
Thanks for the revert. I'll take an extremely careful look at that and fix the underlying issue before relanding. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by kbr@chromium.org
, Sep 1 2016