Seems to be the same issue as the failure on Win 10 High-DPI Perf (3).  This one has been failing for a while with this same stack trace, see build status page: 

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%2010%20High-DPI%20Perf%20%283%29

Most recent log: 
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%2010%20High-DPI%20Perf%20%283%29/builds/290/steps/system_health.memory_desktop/logs/stdio

 Issue 654211  has been merged into this issue.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8998440381492084624

This has been consistently failing since build 2758, so I started a return code bisect on that range.

===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@425222  0          N/A      20  good
chromium@425228  1          N/A      20  bad
chromium@425234  1          N/A      20  bad
chromium@425245  1          N/A      20  bad
chromium@425267  1          N/A      20  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 656040

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests system_health.memory_desktop
Test Metric: memory:chrome:all_processes:dump_count/load_games
Relative Change: Zero to non-zero
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/554
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8998440381492084624


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5813202225463296

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

It looks like this failed because they did a metric bisect instead of a RC bisect. Starting a RC bisect. 

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997626105918259616

=== Auto-CCing suspected CL author pennymac@chromium.org ===

Hi pennymac@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [Windows MSVC CFG] Add support for CFG, and enable on chrome.exe & chrome_elf.dll.
Author  : pennymac
Commit description:
  
CFG supported on x86 and x64 >= Windows 8.1 Update (KB3000850).

Initial support for Control Flow Guard.  "Enable" on chrome.exe to take
advantage of MS system32 dll CFG.  Also compile and link chrome_elf with CFG.

NOTE: As ASLR is disabled in debug builds, so too is CFG.

Using the dumpbin tool from the Visual Studio command prompt, run "dumpbin /headers /loadconfig" against chrome.exe or chrome_elf.dll.  1) "DLL characteristics" now includes "Control Flow Guard", and 2) if code was compiled with cfg there will be "load config" for "Guard CF" function table and flags.

BUG= 584575 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win10_chromium_x64_rel_ng

Review-Url: https://codereview.chromium.org/2412983006
Cr-Commit-Position: refs/heads/master@{#425226}
Commit  : d46842301ebc1f41a92448026bf95e3194538213
Date    : Fri Oct 14 01:16:57 2016


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N  Good?
chromium@425221  0          N/A      2  good
chromium@425224  0          N/A      2  good
chromium@425225  0          N/A      2  good
chromium@425226  1          N/A      2  bad    <--
chromium@425227  1          N/A      2  bad
chromium@425233  1          N/A      2  bad
chromium@425245  1          N/A      2  bad
chromium@425268  1          N/A      2  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 656040

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests system_health.memory_desktop
Test Metric: memory:chrome:all_processes:dump_count/load_games
Relative Change: Zero to non-zero
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/570
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997626105918259616


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5849977245401088

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Ping. Any update?

Hi there,

Now keep in mind I don't have any experience dealing with the perf infrastructure.

I'm afraid I'm going to need some input from somebody who knows about the perf bot harness.  It seems to me there's a problem getting a memory dump?

https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_Zenbook_Perf__3_%2F2758%2F%2B%2Frecipes%2Fsteps%2Fsystem_health.memory_desktop%2F0%2Fstdout#

snippet from above log:

{
C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\image_processing\_bitmap.py:152: UserWarning: Using pure python png decoder, which could be very slow. To speed up, consider installing numpy & cv2 (OpenCV).
  'Using pure python png decoder, which could be very slow. To speed up, '
Traceback (most recent call last):
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\story_runner.py", line 86, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\page\shared_page_state.py", line 310, in RunStory
    self._current_page.Run(self)
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\page\__init__.py", line 109, in Run
    self.RunPageInteractions(action_runner)
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\tools\perf\page_sets\system_health\system_health_story.py", line 113, in RunPageInteractions
    self._Measure(action_runner)
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\tools\perf\page_sets\system_health\system_health_story.py", line 96, in _Measure
    action_runner.MeasureMemory(deterministic_mode=True)
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\actions\action_runner.py", line 138, in MeasureMemory
    assert dump_id, 'Unable to obtain memory dump'
AssertionError: Unable to obtain memory dump
INFO:root:Chrome version: 2891
ERROR:root:Error when deleting c:\users\chrome~1.lab\appdata\local\temp\tmpfi5gdn, will try again at exit.
Traceback (most recent call last):
  File "C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 275, in CollectAgentTraceData
    os.remove(self._trace_log)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\chrome~1.lab\\appdata\\local\\temp\\tmpfi5gdn'
[  FAILED  ] blank:about:blank (11332 ms)
INFO:root:Successfully shut down browser cooperatively
INFO:root:Try printing formatted exception: None None None
Traceback (most recent call last):
  RunBenchmark at C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\story_runner.py:341
    benchmark.ShouldTearDownStateAfterEachStorySetRun())
  Run at C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\story_runner.py:227
    _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
  _RunStoryAndProcessErrorIfNeeded at C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\story_runner.py:86
    state.RunStory(results)
  traced_function at C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py:75
    return func(*args, **kwargs)
  RunStory at C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\page\shared_page_state.py:310
    self._current_page.Run(self)
  Run at C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\page\__init__.py:109
    self.RunPageInteractions(action_runner)
  RunPageInteractions at C:\b\c\b\Win_Zenbook_Perf__3_\src\tools\perf\page_sets\system_health\system_health_story.py:113
    self._Measure(action_runner)
  _Measure at C:\b\c\b\Win_Zenbook_Perf__3_\src\tools\perf\page_sets\system_health\system_health_story.py:96
    action_runner.MeasureMemory(deterministic_mode=True)
  MeasureMemory at C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\catapult\telemetry\telemetry\internal\actions\action_runner.py:138
    assert dump_id, 'Unable to obtain memory dump'
AssertionError: Unable to obtain memory dump
Locals:
  deterministic_mode : True
  dump_id            : None
  platform           : <telemetry.core.platform.Platform object at 0x03411890>
INFO:py_utils.cloud_storage:Uploading c:\users\chrome~1.lab\appdata\local\temp\tmpju2o9l.png to gs://chrome-telemetry-output/profiler-file-id_0-2016-10-14_04-40-2468942.png
View generated profiler files online at https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_0-2016-10-14_04-40-2468942.png for page blank:about:blank
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ]  blank:about:blank
1 FAILED TEST
}

And then there are failures to clean up files as well, as there are handles open to them:

{
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\chrome~1.lab\\appdata\\local\\temp\\tmpfi5gdn'
}

And there doesn't seem to be any exception information ("INFO:root:Try printing formatted exception: None None None") or even test information 
("[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ]  blank:about:blank
1 FAILED TEST")



Also, the perf waterfall seems to show most of the windows perf bots as green, so does this mean that the reported failure only happened once, or only happens on one specific bot (Zen 3)?  https://build.chromium.org/p/chromium.perf/console

If someone who understands the perf testing harness could be CC'd to this ticket, that would be great.  My CL only turns on Windows CFG support for MS system binaries that get loaded into the chrome.exe process space, and for the chrome_elf startup DLL.  Is there any chance the perf infra is trying to hook a DLL in chrome.exe?

Thanks for any help!

Adding primiano@ and perezju@
They focus on android, but hopefully one of them knows someone on windows that may be able to help figure out why memory dumps aren't happening.

yup I have no idea either, but it looks like from this history of action_runner.py that perezju@chromium.org will know about this or at least explain what is wrong or how to repro this locally.

That traceback shows that a memory dump was requested, but for some reason Chrome wasn't able to provide a dump id in return.

Primiano, any idea why this could be failing on this Win Zenbook?

The only thing that comes to my mind is: the cmdline is not passed properly -> memory-infra starts doing periodic dumps -> in some cases the dumps triggered by telemetry collide with the periodic dumps -> NACK.

Can somebody hand me a telemetry trace from that zenbook? Traces contain the cmdline so should be relatively easy to validate this theory.

Looks like telemetry does not upload any traces because of the failures.

The log does show the chrome command line, though, which reads (I've cut some long lines):

C:\b\c\b\Win_Zenbook_Perf__3_\src\out\Release_x64\chrome.exe
--force-fieldtrials=AutofillClassifier/Enabled/AutofillCreditCardSigninProm[...]
--force-fieldtrial-params=AutofillCreditCardSigninPromo.EnabledFive:impress[...]
--enable-features=AutofillCreditCardSigninPromo,AutofillProfileCleanup,Auto[...]
--disable-features=DocumentWriteEvaluator
--enable-memory-benchmarking
--enable-net-benchmarking
--metrics-recording-only
--no-default-browser-check
--no-first-run
--enable-gpu-benchmarking
--disable-background-networking
--proxy-server=socks://localhost:63400
--ignore-certificate-errors
--disable-component-extensions-with-background-pages
--disable-default-apps
--remote-debugging-port=0
--enable-crash-reporter-for-testing
--window-size=1280,1024
--ppapi-flash-path=C:\b\c\b\Win_Zenbook_Perf__3_\src\third_party\adobe\flas[...]
--ppapi-flash-version=99.9.999.999
--user-data-dir=c:\users\chrome~1.lab\appdata\local\temp\tmp2s9uv5
--trace-config-file=c:\users\chrome~1.lab\appdata\local\temp\tmptug2wj\chro[...]
about:blank


[1]: Search for "Starting Chrome" on https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_Zenbook_Perf__3_%2F2848%2F%2B%2Frecipes%2Fsteps%2Fsystem_health.memory_desktop%2F0%2Fstdout

Hmm I was looking for --enable-memory-benchmarking . My theory in #14 is a miss :(

Should we disable the test? It's failing pretty consistently.

Hmm, the exist status bisect points pretty clearly at pennymac's CL; but I'm not sure how this "CFG" change could interact with the memory dumping process.

Re: disabling: The benchmark is failing on the first story (blank:about:blank), and then bails out completely. So this would mean completely disabling system_health.memory_desktop on all of Windows? (Or could be something more specific?) Are we OK with that for now?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/57f0956c8472b744529cee20e6c64bd39fd84ec9

commit 57f0956c8472b744529cee20e6c64bd39fd84ec9
Author: zhenw <zhenw@chromium.org>
Date: Sat Nov 12 04:55:16 2016

Disable system_health.memory_desktop on Win and Mac perf bots

TBR=nednguyen@google.com
BUG= 656040 , 663025 

Review-Url: https://codereview.chromium.org/2497713002
Cr-Commit-Position: refs/heads/master@{#431773}

[modify] https://crrev.com/57f0956c8472b744529cee20e6c64bd39fd84ec9/tools/perf/benchmarks/system_health.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fd3aaa596bf9434716cfe9c6c84ea70d5eae3547

commit fd3aaa596bf9434716cfe9c6c84ea70d5eae3547
Author: nednguyen <nednguyen@google.com>
Date: Sat Nov 12 12:39:11 2016

Revert of Disable system_health.memory_desktop on Win and Mac perf bots (patchset #1 id:1 of https://codereview.chromium.org/2497713002/ )

Reason for revert:
Please don't disable the whole system health benchmark. Disable only the failing stories instead.

Original issue's description:
> Disable system_health.memory_desktop on Win and Mac perf bots
>
> TBR=nednguyen@google.com
> BUG= 656040 , 663025 
>
> Committed: https://crrev.com/57f0956c8472b744529cee20e6c64bd39fd84ec9
> Cr-Commit-Position: refs/heads/master@{#431773}

TBR=zhenw@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 656040 , 663025 

Review-Url: https://codereview.chromium.org/2492323003
Cr-Commit-Position: refs/heads/master@{#431783}

[modify] https://crrev.com/fd3aaa596bf9434716cfe9c6c84ea70d5eae3547/tools/perf/benchmarks/system_health.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bb3d401234313d8b4d83215cab64325dd9679815

commit bb3d401234313d8b4d83215cab64325dd9679815
Author: zhenw <zhenw@chromium.org>
Date: Mon Nov 14 19:15:39 2016

Disable blank:about:blank story on Win perf bot

BUG= 656040 

Review-Url: https://codereview.chromium.org/2503553002
Cr-Commit-Position: refs/heads/master@{#431892}

[modify] https://crrev.com/bb3d401234313d8b4d83215cab64325dd9679815/tools/perf/page_sets/system_health/blank_stories.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3c6aeea81b9c3cf57bcf300d63f6b8adf3dfa0c2

commit 3c6aeea81b9c3cf57bcf300d63f6b8adf3dfa0c2
Author: perezju <perezju@chromium.org>
Date: Wed Dec 21 18:29:00 2016

[system health] Re-enable about:blank and browse:news:cnn on win

These stories should have been fixed by crrev.com/434537

BUG=665465, 656040 

Review-Url: https://codereview.chromium.org/2592933003
Cr-Commit-Position: refs/heads/master@{#440160}

[modify] https://crrev.com/3c6aeea81b9c3cf57bcf300d63f6b8adf3dfa0c2/tools/perf/page_sets/system_health/blank_stories.py
[modify] https://crrev.com/3c6aeea81b9c3cf57bcf300d63f6b8adf3dfa0c2/tools/perf/page_sets/system_health/browsing_stories.py

And about:blank has been running without errors now on all win bots \o/

win-10-high-dpi                    ------------------
win-10                             ----------------
win-7-ati-gpu                      ------------------
win-7-intel-gpu                    -----------------
win-7-nvidia-gpu                   -----------------
win-7                              ----------------
win-7-x64                          ----------------
win-8                              -----------------
win-zenbook                        ----------------