> @hans: was this test one of the newly enabled tests?

No, I haven't touched content_shell_crash_tests, and it seems the bot was running them successfully before build 80169.

Interestingly, telemetry_perf_unittests also fails in a stack trace related test, starting at the same time. And those run on swarming, so the failure shouldn't be due to anything specific on the machine..

https://chromium-review.googlesource.com/c/chromium/src/+/1066065 is part of that build. Maybe it's related? If cdb.exe can't be found, I think that would explain the test failures even if it's not obvious from the logs..

+scottmg and brucedawson who know about the CDB copying

Hmm, looking at the Extract Build step of the build in #0, cdb\cdb.exe is there all right.

Diffing the Extract Build steps of builds 80168 and 80169 shows that the latter, failing, build has an extra file: 

$ diff -u /tmp/80168.txt /tmp/80169.txt 
--- /tmp/80168.txt      2018-05-24 16:41:47.101850875 +0200
+++ /tmp/80169.txt      2018-05-24 16:41:09.446246598 +0200
@@ -24,6 +24,7 @@
 Extracting  full-build-win32\cdb\api-ms-win-core-sysinfo-l1-1-0.dll
 Extracting  full-build-win32\cdb\api-ms-win-core-timezone-l1-1-0.dll
 Extracting  full-build-win32\cdb\api-ms-win-core-util-l1-1-0.dll
+Extracting  full-build-win32\cdb\API-MS-Win-core-xstate-l2-1-0.dll
 Extracting  full-build-win32\cdb\api-ms-win-crt-conio-l1-1-0.dll
 Extracting  full-build-win32\cdb\api-ms-win-crt-convert-l1-1-0.dll
 Extracting  full-build-win32\cdb\api-ms-win-crt-environment-l1-1-0.dll

Where did that come from? copy_cdb_to_output.py uses a glob to copy dlls:

  # Note that the outputs from the "copy_cdb_to_output" target need to
  # be kept in sync with this list.
  _CopyImpl('cdb.exe', output_dir, src_dir)
  _CopyImpl('dbgeng.dll', output_dir, src_dir)
  _CopyImpl('dbghelp.dll', output_dir, src_dir)
  _CopyImpl('dbgmodel.dll', output_dir, src_dir)
  _CopyImpl('ext.dll', dst_winext_dir, src_winext_dir)
  _CopyImpl('uext.dll', dst_winext_dir, src_winext_dir)
  _CopyImpl('exts.dll', dst_winxp_dir, src_winxp_dir)
  _CopyImpl('ntsdexts.dll', dst_winxp_dir, src_winxp_dir)
  _CopyImpl('api-ms-win-eventing-provider-l1-1-0.dll', output_dir, src_dir)
  for dll_path in glob.glob(os.path.join(src_crt_dir, 'api-ms-win-*.dll')):
    _CopyImpl(os.path.split(dll_path)[1], output_dir, src_crt_dir)


So the glob could have picked up the new file if the SDK path changed somehow, and maybe the new file breaks things..

I actually don't think this is the same as  issue 844424  (its bots are now green).

Here's the regression range for https://ci.chromium.org/buildbot/chromium.win/Win7%20Tests%20%281%29/?limit=200 :

https://chromium.googlesource.com/chromium/src/+log/bee4a5a94f7ea8942a6b51200be99076d4db57eb..b7a3fd5d28f223b18d5d8660aa57ce190678b921?pretty=fuller&n=10000

I suspect r561063 which touched toolchain related build files (revert in progress).

telemetry_perf_unittests is broken in same range on https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win%207%20Tests%20x64%20%281%29?limit=200

Range : https://chromium.googlesource.com/chromium/src/+log/6bc1c032f114d9bb69e6561e1d3b40cfd5efade7..383f09a52d885a8916cc057df1fc5c36f20769db?pretty=fuller&n=10000

suspect same culprit r561063

Log :


[1/1] core.stacktrace_unittest.TabStackTraceTest.testValidDump failed unexpectedly 4.3510s:
  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
  Tsproxy commandline: ['e:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\ba88f3\\Scripts\\python.exe', 'e:\\b\\s\\w\\ir\\third_party\\catapult\\telemetry\\third_party\\tsproxy\\tsproxy.py', '--port=0', '--desthost=127.0.0.1']
  TsProxy port: 65109
  DoNothingForwarder started between 127.0.0.1:65109 and 65109
  Starting Chrome ['e:\\b\\s\\w\\ir\\out\\Release_x64\\chrome.exe', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--enable-gpu-benchmarking', '--deny-permission-prompts', '--autoplay-policy=no-user-gesture-required', '--disable-background-networking', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--disable-search-geolocation-disclosure', '--proxy-server=socks://localhost:65109', '--ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I=', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--disable-component-update', '--window-size=1280,1024', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', 'about:blank']
  DoNothingForwarder started between 127.0.0.1:65110 and 65110
  Got devtools config: ws://127.0.0.1:65110/devtools/browser/66938124-6677-40b4-ad83-3c4741af175d
  Browser started (pid=11252).
  OS: win win7
  Detailed OS version: 6.1.7601
  Browser command line: "e:\b\s\w\ir\out\Release_x64\chrome.exe" --enable-net-benchmarking --metrics-recording-only --no-default-browser-check --no-first-run --enable-gpu-benchmarking --deny-permission-prompts --autoplay-policy=no-user-gesture-required --disable-background-networking --disable-component-extensions-with-background-pages --disable-default-apps --disable-search-geolocation-disclosure --proxy-server=socks://localhost:65109 --ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I= --remote-debugging-port=0 --enable-crash-reporter-for-testing --disable-component-update --window-size=1280,1024 --user-data-dir="e:\b\s\w\itvhk9mb\tmpnwn6pn" --flag-switches-begin --flag-switches-end about:blank
  GPU device 0: VENDOR = 0x15ad (Google Inc.), DEVICE = 0x405 (Google SwiftShader)
  GPU Attributes:
    amd_switchable      : False
    can_support_threaded_texture_mailbox: False
    d3d_feature_level   : 0
    direct_composition  : False
    direct_rendering    : True
    driver_date         : 12-2-2014
    driver_vendor       : VMware, Inc.
    driver_version      : 8.14.1.6
    gl_extensions       : GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth24 GL_OES_depth32 GL_OES_depth_texture GL_OES_depth_texture_cube_map GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_sync GL_OES_element_index_uint GL_OES_framebuffer_object GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_surfaceless_context GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_texture_3D GL_OES_vertex_array_object GL_OES_vertex_half_float GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_draw_buffers GL_EXT_instanced_arrays GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_ARB_texture_rectangle GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_APPLE_texture_format_BGRA8888 GL_CHROMIUM_color_buffer_float_rgba GL_CHROMIUM_texture_filtering_hint GL_NV_fence GL_NV_framebuffer_blit GL_NV_read_depth GL_EXT_color_buffer_float 
    gl_renderer         : Google SwiftShader
    gl_reset_notification_strategy: 0
    gl_vendor           : Google Inc.
    gl_version          : OpenGL ES 2.0 SwiftShader 4.0.0.5
    gl_ws_extensions    : EGL_KHR_create_context EGL_KHR_get_all_proc_addresses EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_fence_sync EGL_KHR_image_base EGL_KHR_surfaceless_context EGL_ANGLE_iosurface_client_buffer EGL_ANDROID_framebuffer_target EGL_ANDROID_recordable
    gl_ws_vendor        : Google Inc.
    gl_ws_version       : 1.4 SwiftShader 4.0.0.5
    in_process_gpu      : False
    initialization_time : 0.022232
    jpeg_decode_accelerator_supported: False
    max_msaa_samples    : 4
    optimus             : False
    passthrough_cmd_decoder: False
    pixel_shader_version: 1.00
    process_crash_count : 0
    sandboxed           : True
    software_rendering  : True
    supports_dx12       : False
    supports_overlays   : False
    supports_vulkan     : False
    vertex_shader_version: 1.00
    video_decode_accelerator_flags: 0
    vulkan_version      : 0
  Feature Status:
    2d_canvas           : unavailable_software
    flash_3d            : disabled_software
    flash_stage3d       : disabled_software
    flash_stage3d_baseline: disabled_software
    gpu_compositing     : disabled_software
    multiple_raster_threads: enabled_on
    native_gpu_memory_buffers: disabled_software
    protected_video_decode: disabled_off
    rasterization       : disabled_software
    skia_deferred_display_list: disabled_off
    skia_renderer       : disabled_off
    surface_synchronization: enabled_on
    video_decode        : disabled_software
    viz_display_compositor: disabled_off
    webgl               : unavailable_software
    webgl2              : unavailable_software
  Running sub processes (7 processes): python.exe (4332) - ['e:\\b\\s\\w\\ir\\.swarming_module_cache\\vpython\\ba88f3\\Scripts\\python.exe', 'e:\\b\\s\\w\\ir\\third_party\\catapult\\telemetry\\third_party\\tsproxy\\tsproxy.py', '--port=0', '--desthost=127.0.0.1']
  chrome.exe (11252) - ['e:\\b\\s\\w\\ir\\out\\Release_x64\\chrome.exe', '--enable-net-benchmarking', '--metrics-recording-only', '--no-default-browser-check', '--no-first-run', '--enable-gpu-benchmarking', '--deny-permission-prompts', '--autoplay-policy=no-user-gesture-required', '--disable-background-networking', '--disable-component-extensions-with-background-pages', '--disable-default-apps', '--disable-search-geolocation-disclosure', '--proxy-server=socks://localhost:65109', '--ignore-certificate-errors-spki-list=PhrPvGIaAMmd29hj8BCZOq096yj7uMpRNHpn5PDxI6I=', '--remote-debugging-port=0', '--enable-crash-reporter-for-testing', '--disable-component-update', '--window-size=1280,1024', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', 'about:blank']
  chrome.exe (4740) - ['e:\\b\\s\\w\\ir\\out\\Release_x64\\chrome.exe', '--type=renderer', '--autoplay-policy=no-user-gesture-required', '--enable-gpu-benchmarking', '--field-trial-handle=1300,6658577638399846038,488823873222225499,131072', '--service-pipe-token=AD906E63ED7B1F6D4CB06717A0F8E54A', '--lang=en-US', '--noerrdialogs', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '--enable-offline-auto-reload', '--enable-offline-auto-reload-visible-only', '--blink-settings=cssExternalScannerNoPreload=false,cssExternalScannerPreload=true', '--enable-net-benchmarking', '--device-scale-factor=1', '--num-raster-threads=4', '--enable-main-frame-before-activation', '--service-request-channel-token=AD906E63ED7B1F6D4CB06717A0F8E54A', '--renderer-client-id=4', '--mojo-platform-channel-handle=2252', '/prefetch:1']
  chrome.exe (2972) - ['e:\\b\\s\\w\\ir\\out\\Release_x64\\chrome.exe', '--type=gpu-process', '--field-trial-handle=1300,6658577638399846038,488823873222225499,131072', '--noerrdialogs', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '--start-stack-profiler', '--gpu-preferences=KAAAAAAAAACAAwBgAQAAAAAAAAAAAGAAEAAAAAAAAAAAAAAAAAAAACgAAAAEAAAAIAAAAAAAAAAoAAAAAAAAADAAAAAAAAAAOAAAAAAAAAAQAAAAAAAAAAAAAAAKAAAAEAAAAAAAAAAAAAAACwAAABAAAAAAAAAAAQAAAAoAAAAQAAAAAAAAAAEAAAALAAAA', '--use-gl=swiftshader-webgl', '--noerrdialogs', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '--start-stack-profiler', '--service-request-channel-token=31D27CDD5A5FE284108CF0B86CF81681', '--mojo-platform-channel-handle=1476', '--ignored= --type=renderer ', '/prefetch:2']
  chrome.exe (9636) - ['e:\\b\\s\\w\\ir\\out\\Release_x64\\chrome.exe', '--type=renderer', '--autoplay-policy=no-user-gesture-required', '--enable-gpu-benchmarking', '--field-trial-handle=1300,6658577638399846038,488823873222225499,131072', '--service-pipe-token=2390C81EEDDC4B4521BB91B549F9CD5E', '--lang=en-US', '--noerrdialogs', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '--enable-offline-auto-reload', '--enable-offline-auto-reload-visible-only', '--blink-settings=cssExternalScannerNoPreload=false,cssExternalScannerPreload=true', '--enable-net-benchmarking', '--device-scale-factor=1', '--num-raster-threads=4', '--enable-main-frame-before-activation', '--service-request-channel-token=2390C81EEDDC4B4521BB91B549F9CD5E', '--renderer-client-id=3', '--mojo-platform-channel-handle=2272', '/prefetch:1']
  chrome.exe (3608) - ['e:\\b\\s\\w\\ir\\out\\Release_x64\\chrome.exe', '--type=crashpad-handler', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '/prefetch:7', '--monitor-self', '--monitor-self-argument=--type=crashpad-handler', '--monitor-self-argument=--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '--monitor-self-argument=/prefetch:7', '--monitor-self-annotation=ptype=crashpad-handler', '--database=e:\\b\\s\\w\\itvhk9mb\\tmpqso_ui', '--metrics-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '--annotation=plat=Win64', '--annotation=prod=Chromium', '--annotation=ver=68.0.3439.0-devel', '--initial-client-data=0x80,0x84,0x88,0x7c,0x8c,0x7fef8305510,0x7fef8305520,0x7fef8305530']
  chrome.exe (8248) - ['e:\\b\\s\\w\\ir\\out\\Release_x64\\chrome.exe', '--type=crashpad-handler', '--user-data-dir=e:\\b\\s\\w\\itvhk9mb\\tmpnwn6pn', '/prefetch:7', '--no-periodic-tasks', '--monitor-self-annotation=ptype=crashpad-handler', '--database=e:\\b\\s\\w\\itvhk9mb\\tmpqso_ui', '--annotation=plat=Win64', '--annotation=prod=Chromium', '--annotation=ver=68.0.3439.0-devel', '--initial-client-data=0x98,0x9c,0xa4,0xa0,0xa8,0x13fe77aa0,0x13fe77ab0,0x13fe77ac0']
  Found crashpad_database_util
  Minidump found: e:\b\s\w\itvhk9mb\tmpqso_ui\reports\ca7d980e-3e57-4f29-a65f-7dead78642a0.dmp
  Uploading e:\b\s\w\itvhk9mb\tmpqso_ui\reports\ca7d980e-3e57-4f29-a65f-7dead78642a0.dmp to gs://chrome-telemetry-output/minidump-2018-05-23_08-48-57-870560.dmp
  Problem when trying to gather stack trace:
  Traceback (most recent call last):
    File "e:\b\s\w\ir\third_party\catapult\telemetry\telemetry\core\exceptions.py", line 72, in __init__
      self._is_valid_dump, trace_output = app.GetStackTrace()
    File "e:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\browser\browser.py", line 237, in GetStackTrace
      return self._browser_backend.GetStackTrace()
    File "e:\b\s\w\ir\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "e:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\backends\chrome\desktop_browser_backend.py", line 493, in GetStackTrace
      return self._InternalSymbolizeMinidump(most_recent_dump)
    File "e:\b\s\w\ir\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "e:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\backends\chrome\desktop_browser_backend.py", line 527, in _InternalSymbolizeMinidump
      stack = self._GetStackFromMinidump(minidump_path)
    File "e:\b\s\w\ir\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 52, in traced_function
      return func(*args, **kwargs)
    File "e:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\backends\chrome\desktop_browser_backend.py", line 437, in _GetStackFromMinidump
      '-z', minidump])
    File "e:\b\s\w\ir\.swarming_module\bin\Lib\subprocess.py", line 219, in check_output
      raise CalledProcessError(retcode, cmd, output=output)
  CalledProcessError: Command '['e:\\b\\s\\w\\ir\\out\\Release_x64\\cdb\\cdb.exe', '-y', 'e:\\b\\s\\w\\ir\\out\\Release_x64', '-c', '.ecxr;.lastevent;kb30;~*kb30;q', '-z', 'e:\\b\\s\\w\\itvhk9mb\\tmpqso_ui\\reports\\ca7d980e-3e57-4f29-a65f-7dead78642a0.dmp']' returned non-zero exit status -1073741512
  *************** BROWSER STANDARD OUTPUT ***************
  
  *********** END OF BROWSER STANDARD OUTPUT ************
  ********************* BROWSER LOG *********************
  No log file
  ***************** END OF BROWSER LOG ******************
  Closing browser (pid=11252) ...
  Successfully shut down browser cooperatively
  Browser is closed.
  Traceback (most recent call last):
    File "e:\b\s\w\ir\third_party\catapult\telemetry\telemetry\testing\browser_test_case.py", line 39, in WrappedMethod
      method(self)
    File "e:\b\s\w\ir\tools\perf\core\stacktrace_unittest.py", line 22, in testValidDump
      self.assertTrue(c.exception.is_valid_dump)
  AssertionError: False is not true

P2 (revert in progress)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/864136ef9686128f4afbfe6146cd3e2d08182d52

commit 864136ef9686128f4afbfe6146cd3e2d08182d52
Author: Gabriel Charette <gab@chromium.org>
Date: Thu May 24 20:26:28 2018

Revert "Fixed SDK lookup for non C:\ Windows installation."

This reverts commit 57fd44c74e13be05a3091976f9bcee39ee8f703e.

Reason for revert: suspect for messing up symbols on Win7 bot
                   @  crbug.com/846313 

Original change's description:
> Fixed SDK lookup for non C:\ Windows installation.
>
> Bug: None
> Change-Id: Ia0aa186d1d39b1beac8ce0152683f774ad5d2eaf
> Reviewed-on: https://chromium-review.googlesource.com/1066065
> Reviewed-by: Jochen Eisinger <jochen@chromium.org>
> Commit-Queue: Jochen Eisinger <jochen@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#561063}

TBR=jochen@chromium.org,yura.yaroshevich@gmail.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  846313 
Change-Id: Id93a965aea5555961c539d47e05e79410894eff8
Reviewed-on: https://chromium-review.googlesource.com/1072307
Commit-Queue: Gabriel Charette <gab@chromium.org>
Reviewed-by: Gabriel Charette <gab@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561609}
[modify] https://crrev.com/864136ef9686128f4afbfe6146cd3e2d08182d52/AUTHORS
[modify] https://crrev.com/864136ef9686128f4afbfe6146cd3e2d08182d52/build/vs_toolchain.py
[modify] https://crrev.com/864136ef9686128f4afbfe6146cd3e2d08182d52/build/win/copy_cdb_to_output.py

This bot may be wedged :/. The revert is in https://ci.chromium.org/buildbot/chromium.win/Win7%20Tests%20%281%29/80206

and it still fails.

Additional test environment:
    CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox
    CHROME_HEADLESS=1
    LANG=en_US.UTF-8
Command: e:\b\s\w\ir\.swarming_module_cache\vpython\ba88f3\Scripts\python.exe ..\..\testing\scripts\content_shell_crash_test.py --isolated-script-test-output=e:\b\s\w\ioq1xqg4\output.json --isolated-script-test-perf-output=e:\b\s\w\ioq1xqg4\perftest-output.json

Additional test environment:
    CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox
    CHROME_HEADLESS=1
    LANG=en_US.UTF-8
Command: e:\b\s\w\ir\.swarming_module_cache\vpython\ba88f3\Scripts\python.exe e:\b\s\w\ir\content\shell\tools\breakpad_integration_test.py --verbose --build-dir . --binary .\content_shell.exe --json e:\b\s\w\ituwafa8\tmpmsqrsw

#READY

DevTools listening on ws://127.0.0.1:52837/devtools/browser/ed597873-9630-4515-b275-5d2ae60d1950
[8048:4532:0524/143235.729:ERROR:render_frame_impl.cc(886)] Intentionally crashing (with null pointer dereference) because user navigated to chrome://crash/
#CRASHED - renderer
#CRASHED - renderer
# Running test without trap handler.
# Run content_shell and make it crash.
.\content_shell.exe --run-web-tests chrome://crash --enable-crash-reporter --crash-dumps-dir=e:\b\s\w\ituwafa8\tmpvwzjrq
# Retrieve crash dump.
# Symbolize crash dump.
.\cdb\cdb.exe -y . -c .lines;.excr;k30;q -z e:\b\s\w\ituwafa8\tmpvwzjrq\reports\2cadffd3-acc0-4a65-9337-3fc3f1eba17f.dmp

FAIL: Could not find reference to CrashIntentionally in stack.




a passing run looks like

Additional test environment:
    CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox
    CHROME_HEADLESS=1
    LANG=en_US.UTF-8
Command: e:\b\s\w\ir\.swarming_module_cache\vpython\ba88f3\Scripts\python.exe ..\..\testing\scripts\content_shell_crash_test.py --isolated-script-test-output=e:\b\s\w\iojfsseb\output.json --isolated-script-test-perf-output=e:\b\s\w\iojfsseb\perftest-output.json

Additional test environment:
    CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox
    CHROME_HEADLESS=1
    LANG=en_US.UTF-8
Command: e:\b\s\w\ir\.swarming_module_cache\vpython\ba88f3\Scripts\python.exe e:\b\s\w\ir\content\shell\tools\breakpad_integration_test.py --verbose --build-dir . --binary .\content_shell.exe --json e:\b\s\w\itdidlyz\tmpszqr3w

#READY

DevTools listening on ws://127.0.0.1:49264/devtools/browser/ebf26482-a7b9-4bc7-93ce-a5e9ec5b0029
[748:2056:0523/080702.472:ERROR:render_frame_impl.cc(886)] Intentionally crashing (with null pointer dereference) because user navigated to chrome://crash/
#CRASHED - renderer
#CRASHED - renderer
#READY

DevTools listening on ws://127.0.0.1:49265/devtools/browser/f5b7a2ef-1adf-441f-94d4-8ad4c9af922b
[2952:3580:0523/080705.110:ERROR:render_frame_impl.cc(886)] Intentionally crashing (with null pointer dereference) because user navigated to chrome://crash/
#CRASHED - renderer
#CRASHED - renderer
# Running test without trap handler.
# Run content_shell and make it crash.
.\content_shell.exe --run-web-tests chrome://crash --enable-crash-reporter --crash-dumps-dir=e:\b\s\w\itdidlyz\tmpmsibax
# Retrieve crash dump.
# Symbolize crash dump.
.\cdb\cdb.exe -y . -c .lines;.excr;k30;q -z e:\b\s\w\itdidlyz\tmpmsibax\reports\a46e1460-7f6b-430e-962d-81b8b4eca835.dmp
# Running test with trap handler.
# Run content_shell and make it crash.
.\content_shell.exe --run-web-tests chrome://crash --enable-crash-reporter --crash-dumps-dir=e:\b\s\w\itdidlyz\tmpmsibax --enable-features=WebAssemblyTrapHandler
# Retrieve crash dump.
# Symbolize crash dump.
.\cdb\cdb.exe -y . -c .lines;.excr;k30;q -z e:\b\s\w\itdidlyz\tmpmsibax\reports\58ed2d7e-fdc7-4b68-99eb-a5a75b07b86a.dmp
PASS: Breakpad integration test ran successfully.

From comment #c6 I see that CDB is exited with error code -1073741512

CalledProcessError: Command '['e:\\b\\s\\w\\ir\\out\\Release_x64\\cdb\\cdb.exe', '-y', 'e:\\b\\s\\w\\ir\\out\\Release_x64', '-c', '.ecxr;.lastevent;kb30;~*kb30;q', '-z', 'e:\\b\\s\\w\\itvhk9mb\\tmpqso_ui\\reports\\ca7d980e-3e57-4f29-a65f-7dead78642a0.dmp']' returned non-zero exit status -1073741512

It looks like exit code is 0xC0000138, which is STATUS_ORDINAL_NOT_FOUND (https://msdn.microsoft.com/en-us/library/cc704588.aspx). I haven't seen this error before, but quick googling of 0xC0000138 suggested several articles with information about further diagnosing the error:
1. https://blogs.msdn.microsoft.com/vishalsi/2007/02/26/when-you-get-a-module-load-error/
2. https://jeffpar.github.io/kbarchive/kb/262/Q262602/
3. https://stackoverflow.com/questions/10349155/debugging-process-load-exception-0xc0000138-ordinal-not-found
4. https://stackoverflow.com/questions/12905300/qt-unexpected-gdb-exit

According to articles I've found it is necessary to play with Dependency Walker (https://en.wikipedia.org/wiki/Dependency_Walker) on failing build machine to diagnose the root cause of failure. So, maybe using additional tooling manually on failing build machine can help finding root cause of test crash?

Bumping to P1, as this is blocking the CQ for CLs with retries disabled; see https://chromium-review.googlesource.com/c/chromium/src/+/1071568.

I've just nuked out/Release/cdb/ on the builder to see if that unwedges the original tester. (Should be picked up in this build: https://ci.chromium.org/buildbot/chromium.win/Win%20Builder/55059).

I suspect the reason the test fails is that API-MS-Win-core-xstate-l2-1-0.dll isn't part of the isolate, so never gets sent to the swarming server. It should be listed as an output copy_cdb_to_output target for that to happen.

It's still not clear why crrev.com/561063 caused a new file to get copied. For our builds it should be a no-op, really. I'll investigate some more.

cdb.exe is broken in vs toolchain 5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c (r560002):

cd C:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c\win_sdk\Debuggers\x64
cdb.exe
---------------------------
cdb.exe - Ordinal Not Found
---------------------------
The ordinal 1002 could not be located in the dynamic link library C:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c\win_sdk\Debuggers\x64\dbgeng.dll. 
---------------------------
OK   
---------------------------

but works in 1180cb75833ea365097e279efb2d5d7a42dee4b0.

Digging deeper, the problem is that the package script pulls dbghelp.dll from the VS install rather than from the actual debugging tools. dbghelp.dll 10.0.17625.1000 (the one from VS) is not compatible with the 10.0.17134.12 debugging tools (cdb.exe and friends).

This hack of grabbing dbghelp.dll from VS was added in https://chromium.googlesource.com/chromium/tools/depot_tools/+/af0fd4eb99437a7a914d26d2fca409fc3547e674. Since this is actively harmful, something new needs to be done. In the short-term (until Bruce or someone else with the knowledge is able to take action), we should disable this test suite.

In the interest of spreading the love, here's how I reached this conclusion with a fairly short time investment:

- why on earth can't i land https://chromium-review.googlesource.com/c/chromium/src/+/1071568? win7_chromium_rel_ng is red on every CQ run, but the change is a noop there.
- content_shell_crash_test is failing.
- why isn't it for anyone else?
- oh, my cl doesn't do retries
- so it must be failing for everyone.
- crbug search -- oh, there's a bug for this
- but the gravity (blocking the CQ) isn't known
- and i bet there's noone in EMEA working on it. okay. let's see...
- let's try the "Run this task locally:" instructions on the swarming results page to see if it repros locally.... it does
- it looks like it's doing something with cdb. let's run just that command and see what happens... oh, "The ordinal 1002 could not be located in the dynamic link library ...dbgeng.dll."
- where does that come from? ah there's a gn command to copy it into the build output from the vs toolchain
- can i run cdb.exe from the toolchain dir? nope, same error.
- can i run cdb.exe from my local install of the debugging tools? yup.
- is dbgeng.dll the same between the two? yup, no difference. hmm.
- what's the difference, then? diff the dirs and note that dbghelp.dll is different.
- dig around to find vs_toolchain.py and package_from_installed.py
- look at that, there's special handling for dbghelp.dll in package_from_installed.py
- copy the real dbghelp.dll into the vs toolchain dir
- does cbd.exe work in there now? yup.
- problem identified!

> I've just nuked out/Release/cdb/ on the builder to see if that unwedges the original tester. (Should be picked up in this build: https://ci.chromium.org/buildbot/chromium.win/Win%20Builder/55059).

That didn't help the tests: https://ci.chromium.org/buildbot/chromium.win/Win7%20Tests%20%281%29/80223
Sounds like that the extra .dll was a wild goose chase and grt has a better idea of what's going on.

I'm still wondering what changed though. The bot was fine before https://ci.chromium.org/buildbot/chromium.win/Win7%20Tests%20%281%29/80169 and started failing consistently after. But that doesn't coincide with rolling the VS package or anything like that. Greg, do you have any idea why this started failing now?

Oh, I see. It's a build system dependency problem.

dbghelp.dll broke when the VS version was updated in crrev.com/560002 as grt pointed out.

But look at the tryjob from that update: https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/171124 the copy_cdb_to_output step never ran! So builds were still happily using the old cdb and dlls.

(Any builders doing clobber builds should have been broken immediately, but maybe there aren't any that also run these tests?)

The copy_cdb_to_output action obviously depends on copy_cdb_to_output.py though, so when crrev.com/561063 touched it a few days later, even if it was a no-op change, it caused the new files to get copied and now the builds broke and reverting doesn't help.

Ideally the copy_cdb_to_output step should depend on the files it's copying, but that might not be practical so it should probably at least depend on build/vs_toolchain.py



Let's disable the tests until we have a fix, reinstate crrev.com/561063 since it was innocent, and make copy_cdb_to_output depend on vs_toolchain.py

 Issue 846289  has been merged into this issue.

Taking a stab at disabling the tests: https://chromium-review.googlesource.com/#/c/chromium/src/+/1072648

Adding the dependency: https://chromium-review.googlesource.com/#/c/chromium/src/+/1073451
Re-landing crrev.com/561063: https://chromium-review.googlesource.com/c/chromium/src/+/1073314


We still need to get cdb fixed. It seems Bruce is currently out, and I'm starting the weekend soon.

gab, scottmg: Could you look into this or find someone who can?

Wrapping up two days of sheriffing:

telemetry_perf_unittests (re. #6) was fixed in [1] which includes the revert of r561063, content_shell_crash_test is still failing however...

I see others above have identified potential source of issues for content_shell_crash_test, I'll let bruce handle this from here per #15.

[1] https://chromium.googlesource.com/chromium/src/+log/f85585b37189469ba6c9b53dfea57d88f1a9fe0f..ce7674644d13e37b274716876a3867e1b1dd8b8d?pretty=fuller&n=10000

[2] https://chromium.googlesource.com/chromium/src/+log/bee4a5a94f7ea8942a6b51200be99076d4db57eb..b7a3fd5d28f223b18d5d8660aa57ce190678b921?pretty=fuller&n=10000

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cd6b04c60efd0a56ad80297d752af53a61113674

commit cd6b04c60efd0a56ad80297d752af53a61113674
Author: Hans Wennborg <hans@chromium.org>
Date: Fri May 25 14:34:09 2018

Disable content_shell_crash_test on Windows

cdb is currently broken. Make the test pass by skipping the cdb invocation
until it's fixed.

TBR=mkwst

Bug:  846313 
Change-Id: I2db6460b72d4cf5e81b0c9e590ab10ea5b9241a0
Reviewed-on: https://chromium-review.googlesource.com/1072648
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Greg Thompson <grt@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561869}
[modify] https://crrev.com/cd6b04c60efd0a56ad80297d752af53a61113674/content/shell/tools/breakpad_integration_test.py
[modify] https://crrev.com/cd6b04c60efd0a56ad80297d752af53a61113674/tools/perf/core/stacktrace_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2e7257da41fbc81029fae5f306ae8123ec975f0c

commit 2e7257da41fbc81029fae5f306ae8123ec975f0c
Author: Hans Wennborg <hans@chromium.org>
Date: Fri May 25 14:41:24 2018

Reland "Fixed SDK lookup for non C:\ Windows installation."

This reverts commit 864136ef9686128f4afbfe6146cd3e2d08182d52.

Reason for revert:
While it seems this did trigger the cdb failures in  crbug.com/846313 ,
it was not the root cause and reverting it didn't actually help.

Original change's description:
> Revert "Fixed SDK lookup for non C:\ Windows installation."
> 
> This reverts commit 57fd44c74e13be05a3091976f9bcee39ee8f703e.
> 
> Reason for revert: suspect for messing up symbols on Win7 bot
>                    @  crbug.com/846313 
> 
> Original change's description:
> > Fixed SDK lookup for non C:\ Windows installation.
> >
> > Bug: None
> > Change-Id: Ia0aa186d1d39b1beac8ce0152683f774ad5d2eaf
> > Reviewed-on: https://chromium-review.googlesource.com/1066065
> > Reviewed-by: Jochen Eisinger <jochen@chromium.org>
> > Commit-Queue: Jochen Eisinger <jochen@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#561063}
> 
> TBR=jochen@chromium.org,yura.yaroshevich@gmail.com
> 
> # Not skipping CQ checks because original CL landed > 1 day ago.
> 
> Bug:  846313 
> Change-Id: Id93a965aea5555961c539d47e05e79410894eff8
> Reviewed-on: https://chromium-review.googlesource.com/1072307
> Commit-Queue: Gabriel Charette <gab@chromium.org>
> Reviewed-by: Gabriel Charette <gab@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#561609}

TBR=gab@chromium.org,jochen@chromium.org,yura.yaroshevich@gmail.com

Bug:  846313 
Change-Id: I9955a47bbe8a81a0dcf7befebfc422560f8a1cc4
Reviewed-on: https://chromium-review.googlesource.com/1073314
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561873}
[modify] https://crrev.com/2e7257da41fbc81029fae5f306ae8123ec975f0c/AUTHORS
[modify] https://crrev.com/2e7257da41fbc81029fae5f306ae8123ec975f0c/build/vs_toolchain.py
[modify] https://crrev.com/2e7257da41fbc81029fae5f306ae8123ec975f0c/build/win/copy_cdb_to_output.py

Re. reland in #21, see #19, I do believe this was the culprit for telemetry_perf_unittests failures on [1]. We'll see.

[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win%207%20Tests%20x64%20%281%29?limit=200

I supressed the failed Telemetry test yesterday. Lemme try to reenable it

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/69141da060f335106bfc7313fa1fde65cc72da91

commit 69141da060f335106bfc7313fa1fde65cc72da91
Author: nednguyen <nednguyen@google.com>
Date: Fri May 25 15:19:09 2018

Revert "Disable core.stacktrace_unittest.TabStackTraceTest.testValidDump on win"

This reverts commit 216e6d16194a576c5fdaf126d2984e2d60ed32a7.

Reason for revert: culrpit CL was reverted 

BUG: chromium:846313 

Original change's description:
> Disable core.stacktrace_unittest.TabStackTraceTest.testValidDump on win
> 
> TBR=kbr@chromium.org
> NOTRY=true  # net_unittests flake
> 
> Bug:  846289 
> Change-Id: I7fa42786ebdb448fd424e37a7c777e60df6b3c36
> Reviewed-on: https://chromium-review.googlesource.com/1071868
> Commit-Queue: Ned Nguyen <nednguyen@google.com>
> Reviewed-by: Ned Nguyen <nednguyen@google.com>
> Cr-Commit-Position: refs/heads/master@{#561594}

TBR=nednguyen@google.com

Change-Id: Ibf328d3ea480b094e47c5037a7498f198ef94e62
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  846289 
Reviewed-on: https://chromium-review.googlesource.com/1073091
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#561881}
[modify] https://crrev.com/69141da060f335106bfc7313fa1fde65cc72da91/tools/perf/core/stacktrace_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bec05e14e8ac760ca6182f8cad11e1e8507758f4

commit bec05e14e8ac760ca6182f8cad11e1e8507758f4
Author: Hans Wennborg <hans@chromium.org>
Date: Fri May 25 15:52:19 2018

Make copy_cdb_to_output depend on vs_toolchain.py

Ideally, copy_cdb_to_output should depend on the files it's copying. But
since those live outside Chromium that's not really practical. Depending
on vs_toolchain.py is probably good enough; the important part is this
will force copy_cdb_to_output to run whenever the VS version changes.

Bug:  846313 
Change-Id: Ie0596fb820ba71c40b65325d663f95c2bffe30a4
Reviewed-on: https://chromium-review.googlesource.com/1073451
Commit-Queue: Scott Graham <scottmg@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561893}
[modify] https://crrev.com/bec05e14e8ac760ca6182f8cad11e1e8507758f4/build/win/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8efd16ff6e3f26db9ac52e7d8bf8d65b723c668e

commit 8efd16ff6e3f26db9ac52e7d8bf8d65b723c668e
Author: Luna Lu <loonybear@chromium.org>
Date: Fri May 25 17:40:50 2018

Reland "Disable core.stacktrace_unittest.TabStackTraceTest.testValidDump on win"

This reverts commit 69141da060f335106bfc7313fa1fde65cc72da91.

Reason for revert: core.stacktrace_unittest.TabStackTraceTest.testValidDump is failing on Win 7, disabling this test for now. 

Original change's description:
> Revert "Disable core.stacktrace_unittest.TabStackTraceTest.testValidDump on win"
> 
> This reverts commit 216e6d16194a576c5fdaf126d2984e2d60ed32a7.
> 
> Reason for revert: culrpit CL was reverted 
> 
> BUG: chromium:846313 
> 
> Original change's description:
> > Disable core.stacktrace_unittest.TabStackTraceTest.testValidDump on win
> > 
> > TBR=kbr@chromium.org
> > NOTRY=true  # net_unittests flake
> > 
> > Bug:  846289 
> > Change-Id: I7fa42786ebdb448fd424e37a7c777e60df6b3c36
> > Reviewed-on: https://chromium-review.googlesource.com/1071868
> > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > Cr-Commit-Position: refs/heads/master@{#561594}
> 
> TBR=nednguyen@google.com
> 
> Change-Id: Ibf328d3ea480b094e47c5037a7498f198ef94e62
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug:  846289 
> Reviewed-on: https://chromium-review.googlesource.com/1073091
> Commit-Queue: Ned Nguyen <nednguyen@google.com>
> Reviewed-by: Ned Nguyen <nednguyen@google.com>
> Cr-Commit-Position: refs/heads/master@{#561881}

TBR=nednguyen@google.com

Change-Id: I039a5ffa03afbd50d79c57529d1b561def58ccdf
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  846289 
Reviewed-on: https://chromium-review.googlesource.com/1073492
Reviewed-by: Luna Lu <loonybear@chromium.org>
Commit-Queue: Luna Lu <loonybear@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561933}
[modify] https://crrev.com/8efd16ff6e3f26db9ac52e7d8bf8d65b723c668e/tools/perf/core/stacktrace_unittest.py

Sorry for the imprecision of the copy_cdb_to_output target.

In hindsight, would it have been possible to revert the VS roll rather than disable the tests? It seems to me that would have been the appropriate course of action since the tests were good but the build system around the cdb roll was what was broken.

It looks like the telemetry test is still failing (i.e., it started failing again).

See also bugs  844424  and  bug 846729  for related issues.

 Issue 846729  has been merged into this issue.

taking out of sheriff queue whilst this isn't immediately impacting things

Sorry, I had to run to the airport last Friday and didn't have a computer to follow up on this.

Re #19
> telemetry_perf_unittests (re. #6) was fixed in [1] which includes the revert of r561063, content_shell_crash_test is still failing however...
> [1] https://chromium.googlesource.com/chromium/src/+log/f85585b37189469ba6c9b53dfea57d88f1a9fe0f..ce7674644d13e37b274716876a3867e1b1dd8b8d?pretty=fuller&n=10000

It's not fixed. [1] included Ned's https://chromium-review.googlesource.com/1071868 which disabled the test.


Re #22
> Re. reland in #21, see #19, I do believe this was the culprit for telemetry_perf_unittests

No, the "Fixed SDK lookup" change triggered the errors by making the bots pick up the broken cdb brought in by the VS roll. Reverting this didn't help.


Re #23
> I supressed the failed Telemetry test yesterday. Lemme try to reenable it

Nooo, cdb is still broken, the disabling should stay in for now.


Re #27
> In hindsight, would it have been possible to revert the VS roll rather than disable the tests? It seems to me that would have been the appropriate course of action since the tests were good but the build system around the cdb roll was what was broken.

Yes, that would have been possible, but potentially a little scary since it's been in a while. I think disabling the tests to unblock the CQ until a proper fix can be done makes sense, but perhaps reverting the roll would have been better, I'm not sure.



It looks like Bruce is still out-of-office. Should we try to revert the VS roll? Does anyone feel qualified to try and roll in a good one?

I'm back at work and will investigate this

If I run cdb.exe from the latest toolchain package (on Windows 10) I get this error:

---------------------------
cdb.exe - Ordinal Not Found
---------------------------
The ordinal 1003 could not be located in the dynamic link library c:\src\chromium\src\third_party\depot_tools\win_toolchain\vs_files\5454e45bf3764c03d3fc1024b3bf5bc41e3ab62c\win_sdk\Debuggers\x86\dbgeng.dll. 
---------------------------
OK   
---------------------------

Presumably this is the bug.

This is with the 10.0.17134.0 version of cdb.exe. It runs fine from the 10.0.17134.0 SDK install directory. Comparison of the debuggers directories shows that dbghelp.dll is the only file that is different. It looks like there was an initially broken version of the debuggers package. The version of dbghelp.dll that was signed on 3/15/2018 was incompatible with dbgeng.dll.

Every other DLL and EXE (with the exception of pdbcopy.exe, last updated 12/15/2015) was signed on 4/20/2018, so it looks like the build process failed for dbghelp.dll, and was then fixed.

I added a --repackage option to package_from_installed.py so it is easy for me to hot-patch the depot tools package with updated copies (x86 and x64) of dbghelp.dll. I'll create a CL to do this.

Ah ha! I found the real root cause:

https://cs.chromium.org/chromium/tools/depot_tools/win_toolchain/package_from_installed.py?q=package_from_inst&sq=package:chromium&g=0&l=193

We used to have to copy the VS version of dbghelp.dll to the Debuggers package in order to get /debug:fastlink support. This was always a bit dodgy and we finally got burned by it. This hack isn't need anymore because we don't use /debug:fastlink anymore (lld FTW) and because the Debuggers version is presumed to be updated by now.

I'll remove that hacked piece of code and repackage.

So, two part fix coming in:

crrev.com/c/1086990 - this removes the dbghelp.dll hack from the packaging script
crrev.com/c/1086992 - this updates the toolchain package. I just ran the updated script on the VM I used for the previous packaging and now cdb.exe runs correctly.

I also tested that the new cdb.exe runs on Windows 7 (well, Server 2008 R2) and it does.

Can you reenable the Telemetry test to see if it's fixed as well?

> Can you reenable the Telemetry test to see if it's fixed as well?

I have a CL for re-enabling both tests here, once crrev.com/c/1086992 lands: https://chromium-review.googlesource.com/c/chromium/src/+/1087001

Okay, catching up on all the details of the bug. I see that grt@ found the same root cause that I did, so we're in agreement there :-)

If I understand correctly then change crrev.com/c/1073451 fixed the dependency bug so that the cdb copy step will happen whenever the toolchain changes - that will prevent this type of bug from recurring.

I've updated crrev.com/c/1086992 to remove the changes that disabled the test and we'll see what the try jobs say. If the test runs and passes then that implies that the cdb copy step ran, meaning that the dependency fix is working. I'll probably remove the test re-enabling from the CL at that point, to avoid having a CL that tries to do too much.

Thank you for your patience and apologies for the breakage and the delays.

Re-enabling the tests in the CL to test them didn't work because the test didn't run. But, I manually reproed the failure on my machine and then confirmed that the fix worked. So, it's ready to land I believe, followed by re-enabling the test.

I'm flying tomorrow so another option would be to wait until Thursday, depending on priority/risk-assesment/etc.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ac88874511d02aaf42ebbefaf078d8c8f8c2af8a

commit ac88874511d02aaf42ebbefaf078d8c8f8c2af8a
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Wed Jun 06 23:52:33 2018

Update toolchain package to fix dbghelp.dll bug

This change updates the VS 2017 package to fix a bad version of
dbghelp.dll in the Debuggers package, caused by a now-obsolete
workaround in the packaging script, removed in crrev.com/c/1086990.

The package still uses VS 2017 Update 7.1 and the 10.0.17134.12 SDK.

Packaging was done on a Windows Server 2016 VM, cleanly created for this
purpose.

Compiler was packaged up by downloading VS 2017 Update 7.1, from
https://www.visualstudio.com/vs/, and then passing these parameters to
the installer:

    --add Microsoft.VisualStudio.Workload.NativeDesktop
    --add Microsoft.VisualStudio.Component.VC.ATLMFC --includeRecommended
    --passive

Then Add or Remove Programs was used to modify the 10.0.17134.0 SDK to add
the Debuggers package.

Then the packaging script was run like this:

  python depot_tools\win_toolchain\package_from_installed.py 2017 -w 10.0.17134.0

The results were compared to make sure that there were no unintended changes. One
quirk is that the new package is missing the arm/arm64 directories in Debuggers,
which is correct, whereas the previous package was *not* missing them, for some
reason. Other than that there are no unintended changes.

Bug:  773476 ,  846313 
Change-Id: I43db2ea95999fb7b2aeb02ba078e70298b62ffad
Reviewed-on: https://chromium-review.googlesource.com/1086992
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#565106}
[modify] https://crrev.com/ac88874511d02aaf42ebbefaf078d8c8f8c2af8a/build/vs_toolchain.py

This should be ready to have the tests re-enabled now - reassigning for that step.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e867a749a65388c708daa830ee5231d4740d3098

commit e867a749a65388c708daa830ee5231d4740d3098
Author: Hans Wennborg <hans@chromium.org>
Date: Thu Jun 07 10:06:30 2018

Re-enable content_shell_crash_test and telemetry_unittests testValidDump

They should pass after the latest toolchain package update, which brought
in a fixed dbghelp.dll. (crrev.com/565106).

(This effectively reverts crrev.com/561869 and crrev.com/561594, so
TBR'ing the reviewers of those.)

TBR=mkwst,kbr

Bug:  846313 
Change-Id: Ifd5f3a84465f3ef2a1ea0a291005b1305f7adf71
Reviewed-on: https://chromium-review.googlesource.com/1087001
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#565216}
[modify] https://crrev.com/e867a749a65388c708daa830ee5231d4740d3098/content/shell/tools/breakpad_integration_test.py
[modify] https://crrev.com/e867a749a65388c708daa830ee5231d4740d3098/tools/perf/core/stacktrace_unittest.py

Thank you for turning these tests back on!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/3fb5aa7bc5ad229e7dc86e0c8360157efd74cdc1

commit 3fb5aa7bc5ad229e7dc86e0c8360157efd74cdc1
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Fri Jun 08 18:02:13 2018

Remove dbghelp.dll hack from packaging script

The script that packages the Visual Studio toolchain had special code to
handle grabbing dbghelp.dll from the VS install and copying it to the
Debuggers directories. This was necessary around VS 2017 RTM to handle
/debug:fastlink but it now causes failures due to mismatched DLLs
leading to a missing ordinal. The hack shouldn't be needed anymore
because we no longer depend on /debug:fastlink (lld!) and because the
Debuggers package is assumed to have been updated to handle VS 2017 by
now.

Bug:  846313 
Change-Id: I2b5fd87aaa785ce24a0905d70e7e586ff4838b1c
Reviewed-on: https://chromium-review.googlesource.com/1086990
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>

[modify] https://crrev.com/3fb5aa7bc5ad229e7dc86e0c8360157efd74cdc1/win_toolchain/package_from_installed.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/85bd22097fe4baec7ff3aafaff72f94a944116d8

commit 85bd22097fe4baec7ff3aafaff72f94a944116d8
Author: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Jun 09 02:16:43 2018

Roll src/third_party/depot_tools e05f18d..3fb5aa7 (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/e05f18d..3fb5aa7


git log e05f18d..3fb5aa7 --date=short --no-merges --format='%ad %ae %s'
2018-06-08 brucedawson@chromium.org Remove dbghelp.dll hack from packaging script


Created with:
  gclient setdep -r src/third_party/depot_tools@3fb5aa7

The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:846313 
TBR=agable@chromium.org

Change-Id: I1670bc12e0e0ccd343f9d2a483f2126fe5c52586
Reviewed-on: https://chromium-review.googlesource.com/1093594
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#565822}
[modify] https://crrev.com/85bd22097fe4baec7ff3aafaff72f94a944116d8/DEPS