Assigning to me to remove the bots from the waterfall for now.

This is outside my domain of expertise. Those DLLs are associated with Chrome for VR (cc'ing bsheedy@) but it looks to me like the machine might be running out of memory or something? CC'ing thakis.

What's "the new win-dbg bot"?

An "Access is denied" during linking sounds very similar to what we debugged in https://groups.google.com/a/google.com/forum/?utm_medium=email&utm_source=footer#!msg/chrome-windows/Vx1LuqTbCbU/UQSEu_TwDAAJ -- I thought we had fixed the root causes, but maybe this new bot runs into something similar.

If it is actually a VR-related issue, billorr@ is the most knowledgeable about the Windows side.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/f0f6c1453cf89ffc503d6c4916b80c43ca1867bc

commit f0f6c1453cf89ffc503d6c4916b80c43ca1867bc
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Aug 30 00:21:19 2018

Remove win debug builders from the chromium tree

These builders are flaky still.

R=jbudorick@chromium.org
TBR=jbudorick

Bug:878915
Change-Id: Ib707ac4f14e00e9d49b8bfbf9ea2ec98e4dae0e8
Reviewed-on: https://chromium-review.googlesource.com/1195914
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/f0f6c1453cf89ffc503d6c4916b80c43ca1867bc/scripts/slave/gatekeeper.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/025bf21795504cc38e2e820b7f3a5dbda924ed6b

commit 025bf21795504cc38e2e820b7f3a5dbda924ed6b
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Aug 30 01:48:49 2018

Remove Windows debug bots from consoles

They've been a bit flaky, and I don't want to bother sheriffs with them
yet.

Bug: 878915
Change-Id: Ia37db9742e3bf7e58a2efba8ab4032f7b121620e
Reviewed-on: https://chromium-review.googlesource.com/1195926
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#587409}
[modify] https://crrev.com/025bf21795504cc38e2e820b7f3a5dbda924ed6b/infra/config/global/luci-milo.cfg

win32-rel has just failed, too. Same issue, maybe?

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/win32-rel/1755

[6305/75212] LINK test_opus_api.exe test_opus_api.exe.pdb
FAILED: test_opus_api.exe test_opus_api.exe.pdb 
ninja -t msvc -e environment.x86 -- ../../third_party/llvm-build/Release+Asserts/bin/lld-link.exe /nologo /OUT:./test_opus_api.exe /PDB:./test_opus_api.exe.pdb @./test_opus_api.exe.rsp
lld-link.exe: error: ExecuteAndWait failed: C:\b\swarming\w\ir\cache\builder\src\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\bin\10.0.17134.0\x64\mt.exe /manifest C:\b\swarming\w\ir\tmp\t\lld-defaultxml-6e8f62.manifest /manifest ../../build/win/as_invoker.manifest /manifest ../../build/win/common_controls.manifest /manifest ../../build/win/compatibility.manifest /nologo /out:C:\b\swarming\w\ir\tmp\t\lld-user-4d9d47.manifest

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/453c0e679d6a648f9311fe42077c7aa3352bb990

commit 453c0e679d6a648f9311fe42077c7aa3352bb990
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Aug 30 14:57:39 2018

Revert "Remove Windows debug bots from consoles"

This reverts commit 025bf21795504cc38e2e820b7f3a5dbda924ed6b.

Reason for revert: Broke changes in //testing/buildbot. Will reland with a fix to that. Sheriffs should still not be alerted to failures on these bots in sheriff-o-matic even when this is reverted.

Original change's description:
> Remove Windows debug bots from consoles
> 
> They've been a bit flaky, and I don't want to bother sheriffs with them
> yet.
> 
> Bug: 878915
> Change-Id: Ia37db9742e3bf7e58a2efba8ab4032f7b121620e
> Reviewed-on: https://chromium-review.googlesource.com/1195926
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#587409}

TBR=hinoka@chromium.org,nodir@chromium.org,martiniss@chromium.org,jbudorick@chromium.org

Change-Id: Ib07cce63148a2f9f8de20bff0d8bf50d8c0acc2d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 878915
Reviewed-on: https://chromium-review.googlesource.com/1196983
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#587576}
[modify] https://crrev.com/453c0e679d6a648f9311fe42077c7aa3352bb990/infra/config/global/luci-milo.cfg

Re: #7; that looks a bit different, but might be the same? Let me look a bit.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/769b25119d57e1132d066081c9f37d6e6c08d004

commit 769b25119d57e1132d066081c9f37d6e6c08d004
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Aug 30 18:52:06 2018

Reland "Remove Windows debug bots from consoles"

This is a reland of 025bf21795504cc38e2e820b7f3a5dbda924ed6b

Original change's description:
> Remove Windows debug bots from consoles
> 
> They've been a bit flaky, and I don't want to bother sheriffs with them
> yet.
> 
> Bug: 878915
> Change-Id: Ia37db9742e3bf7e58a2efba8ab4032f7b121620e
> Reviewed-on: https://chromium-review.googlesource.com/1195926
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#587409}

Bug: 878915
Change-Id: I65ea5aa215a235cdfc2b93c4804f69a93a953e95
Reviewed-on: https://chromium-review.googlesource.com/1196919
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#587679}
[modify] https://crrev.com/769b25119d57e1132d066081c9f37d6e6c08d004/infra/config/global/luci-milo.cfg
[modify] https://crrev.com/769b25119d57e1132d066081c9f37d6e6c08d004/testing/buildbot/generate_buildbot_json.py

Ok, bot shouldn't close the tree anymore. Speculatively assigning to billorr@, per #4. This isn't that high priority; it's not flaking that often.

Re #3: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/win32-dbg is a new bot I spun up recently for bug 867725, to add debug coverage to the main chromium waterfall. 

Also, http://shortn/_W9NNByQ4MT is a graph of usage stats for machine. There's a spike in memory usage at some point, not sure if it means anything.

The quoted output in #1 mentions vr dlls, but I think that is just random chance.  No built vr dlls should cause createprocess to fail.

The second failure from #1 has this:
[58576/59775] LIB obj/chrome/browser/extensions/extensions.lib
[58577/59775] STAMP obj/chrome/browser/web_applications/bookmark_apps/bookmark_apps.stamp
[58578/59775] STAMP obj/chrome/browser/web_applications/web_applications.stamp
[58579/59775] STAMP obj/chrome/browser/apps/platform_apps/platform_apps.stamp
ninja: fatal: CreateProcess: Access is denied.
step returned non-zero exit code: 1

It may be interesting to know what commandline had CreateProcess fail with Access denied.  Is that available information?   I could imagine adding extra logging here on failure: https://github.com/ninja-build/ninja/blob/ca041d88f4d610332aa48c801342edfafb622ccb/src/subprocess-win32.cc#L128.  Is it possible to run bots with a forked version of ninja, or can we push a change and run with updated ninja?

Is any kind of antivirus running?  That is the typical place I've seen errors like this with access denied.

Not easily.

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/win32-dbg/120 has:

[59904/59916] ACTION //chrome/installer/mini_installer:previous_version_mini_installer(//build/toolchain/win:win_clang_x64)
FAILED: previous_version_mini_installer.exe 
C:/b/swarming/w/ir/cache/vpython/80ee9f/Scripts/python.exe ../../chrome/installer/mini_installer/generate_previous_version_mini_installer.py --out previous_version_mini_installer.exe
Traceback (most recent call last):
  File "../../chrome/installer/mini_installer/generate_previous_version_mini_installer.py", line 27, in <module>
    sys.exit(main())
  File "../../chrome/installer/mini_installer/generate_previous_version_mini_installer.py", line 22, in main
    '--out=' + args.out,
  File "C:\b\swarming\w\ir\cipd_bin_packages\bin\Lib\subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "C:\b\swarming\w\ir\cipd_bin_packages\bin\Lib\subprocess.py", line 390, in __init__
    errread, errwrite)
  File "C:\b\swarming\w\ir\cipd_bin_packages\bin\Lib\subprocess.py", line 640, in _execute_child
    startupinfo)
WindowsError: [Error 5] Access is denied


which looks similar, but since it happened in a subprocess we have some more information. That code is here: https://cs.chromium.org/chromium/src/chrome/installer/mini_installer/generate_previous_version_mini_installer.py?q=generate_previous_version_mini_installer&sq=package:chromium&dr&l=22

It tries to run alternate_version_generator.exe from the build dir, and that isn't working for some reason. The executable did get built further up ([36943/59916] LINK alternate_version_generator.exe alternate_version_generator.exe.pdb).

So either by the time the build tries to run it, it got deleted by something, or the bot can't run it for some reason.

I agree this does look more like a bot thing and less like a vr thing.

My guess is still someone has a handle to the built file open - windows search indexer, antivirus, maybe the compiler/linker tools.

Seeing this for freshly built executables makes sense since they are most likely to be read/accessed by something shortly after they finished building.

Did this start recently?  Are there any configuration changes to the machines hitting it?

+Bruce since I seem to recall he had some similar issues with running recently built files at some point.  (though I didn't find it after a quick search so I may be misremembering).

I've only heard of this on the win32-dbg bot, which is new. So I'd ask whoever added the bot if that bot is configured differently than all the other bots.

I added this bot this year. The only difference I know of is that it's just doing a debug build instead of a release build. It should be basically identical to https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/win32-rel. The bots seem to be identical; the only difference I see is that one is a Haswell CPU and one is an Ivy Bridge CPU.

Race conditions in the build are one thing that can cause this - two steps trying to generate one file, or one step trying to read the output of a build step that hasn't completed. However that doesn't explain why these failures would only happen on these builders.

The CreateProcess error is interesting. That *could* mean that we tried to run something before we finished building it, but since the build step for that binary had clearly completed it can't be that. It can't be that the .exe was deleted because then there would be a different error code. So, it's looking like interference from a virus scanner. I don't know how the bots are configured, but make sure that the build directories are excluded from scanning, which will only slow them down anyway.

I RDP-ed to the bot, and looked around for an anti virus or something. I didn't see anything super suspicious. I saw Cortana (SearchUI.exe) running in the background, although I think it's suspended, so probably not doing anything, and I saw a Windows Defender UI process thing running, although I didn't see anything like it in the task bar. OneDrive.exe is also running.

I looked at the bot powering win32-rel, and it all seemed the same; the processes I'd seen on the problematic bot were there as well. So, not really sure what's going on :(

The bot hasn't been flaky in the past few days though, as far as I can tell, so it might be safe to re-enable?

Happy to help anyone if they want to RDP to the bot and inspect it yourself.

Marking available - I've been unable to make progress.  Looks like this reproed about a week ago in build 376.

My theory is still indexing or antivirus, but we need more information to know for sure why we're getting access denied.

I did some digging into our windows bootstrap scripts, and it looks like we disable Windows Defender: http://shortn/_SCSFv56Nsy shows this. I'm not as familiar with how our bots are set up. cc-ing people who worked on this file.

Labs, can you help diagnose? We're having problems with a bot sometimes failing to compile due to some sort of access denied errors. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/win-dbg/376 is a semi recent example of this. It's not super frequent, but does happen occasionally.

The bot hostname is swarm2920-c4. I've tried RDP-ing into it to check on windows defender settings, but haven't been able to. Although I think we disable it, so maybe that's not the issue?