May not be infra related... Here's the error in the log:

[0419/081944:test_installer.py(112)] Finished action install_chrome_user
[0419/081944:test_installer.py(143)] Verifying state chrome_user_installed_not_inuse
FAIL
[0419/081945:ERROR:uninstall.cc(548)] Failed to delete path (1st try): C:\Users\chrome-bot\AppData\Local\Chromium\Application\chrome.exe
[0419/081945:ERROR:uninstall.cc(564)] Failed to delete path (2nd try): C:\Users\chrome-bot\AppData\Local\Chromium\Application\chrome.exe
Traceback (most recent call last):
  File "uninstall_chrome.py", line 68, in <module>
    sys.exit(main())
  File "uninstall_chrome.py", line 63, in main
    'status %d.' % exit_status)
Exception: Could not uninstall Chrome. The installer exited with status 20.

Could we have a sheriff to take a look?

cc'ing dpranke@

There were 12 changes between build #535 and #536.

https://chromium.googlesource.com/chromium/src/+log/5b65a57f38f0260e21b8ec190d201ae2504d73e5%5E..79c0bfc015fc21ed804ce2064ac0603ca50ed79c?pretty=fuller

The only one that's related to Windows is https://codereview.chromium.org/1888983002

Could you take a quick look and assign back to me if your CL isn't the cause?

Darn.

Yep, that's my change.

I'm not sure what's going on here. I've seen this test pass in some places and fail others.

I was pretty sure that I saw at least one build pass with this, but maybe some state carried over or something.

Bruce, can you take another look?

whoops, that's Win10 ...

and, yes, the first build w/ my change in it fails, as do all subsequent ones.

The test is passing on Win7, though, which is the bot I checked:

https://build.chromium.org/p/chromium.win/builders/Win%207%20Tests%20x64%20%281%29/builds/13004

and it passes on the Win10 trybot:

https://build.chromium.org/p/tryserver.chromium.win/builders/win10_chromium_x64_rel_ng/builds/39

So, I think that adds to my theory that there's something off on the waterfall bot. 

Bruce, you suggested that this might have to do with tests running elevated or not; is there something we can check on the bot to see if it has the right privs or not?

I'll take a look. If elevation or some other environment issue was the problem then I would have expected it to be failing prior to the gn switch. I might need to create gn and gyp versions of mini_installer and see how they differ.

are there other gn builders e.g. on win7 that these tests are passing for?

Will

I'm downloading the build .zip files so that I can compare them. I'll look and see why the gn version is 600 MB larger while I'm at it.

File list in mini_installer.exe from the 535 and 536 (gyp/gn) packages from https://build.chromium.org/p/chromium.win/builders/Win10%20Tests%20x64?numbuilds=100 are almost identical. 52.0.2712.0\Locales contains fake-bidi.pak on the 536 (gn) build. That is the only observed difference in what files are present.

The DLLs in 536 are generally larger, sometimes a lot larger (91.8 versus 59.5 million bytes for chrome.dll) and even the Locales\*.pak files are bigger in 536.

The 535 mini_installer contains more version manifests, presumably just from those rarely being deleted from the build directories.

The file paths in mini_installer are identical.

@wfh - yes, the test is passing on the Win7 bot (as I linked to in comment #6).

I can repro the problem locally on my Windows 10 machine, so I should be able to figure it out. When I run this command:

    mini_installer.exe --chrome --multi-install --verbose-logging --do-not-launch-chrome

the 536 mini_installer launches Chrome. I'm no expert but I'm guessing that this is not expected, given the "--do-not-launch-chrome" parameter. I'll trace through and figure out what's going wrong.

The 535 mini_installer does not launch Chrome.

The diagnosis was complicated by the fact that when the 536 mini_installer launches Chrome the newly created Chrome processes sometimes don't open visible windows.

My guess is manifest troubles causing base/win/windows_version.cc to think Chrome is running on Win8 or lower, leading to an attempt to pin Chrome's shortcut to the taskbar. This launches Chrome on Win10 instead. Ooops.

 Issue 604681  has been merged into this issue.

I like the theory in comment #14 because it explains why gn and Windows 10 are both required in order to trigger the bug. And indeed setup.exe is missing its manifest entirely. That should be an easy fix.

A bit more context, for posterity - debugging of setup.exe shows that CreateProcessW is being called to launch Chrome on this non-obvious call stack. This makes it non-obvious what code is causing shell32 to launch Chrome. This method of launching Chrome probably also explains why Chrome sometimes doesn't launch - there is probably a race condition about whether this thread will have time to launch Chrome before the setup.exe process dies.

 	kernel32.dll!CreateProcessWStub ()	Unknown
 	windows.storage.dll!CInvokeCreateProcessVerb::_CallCreateProcess(void)	Unknown
 	windows.storage.dll!CInvokeCreateProcessVerb::_PrepareAndCallCreateProcess(void)	Unknown
 	windows.storage.dll!CInvokeCreateProcessVerb::_TryCreateProcess(void)	Unknown
 	windows.storage.dll!CInvokeCreateProcessVerb::_DoApplication(void)	Unknown
 	windows.storage.dll!CInvokeCreateProcessVerb::Execute(void)	Unknown
 	windows.storage.dll!CBindAndInvokeStaticVerb::_DoCommand(struct IExecuteCommand *,struct IShellItemArray *,bool)	Unknown
 	windows.storage.dll!CBindAndInvokeStaticVerb::_TryCreateProcessDdeHandler(unsigned long)	Unknown
 	windows.storage.dll!CBindAndInvokeStaticVerb::Execute(void)	Unknown
 	windows.storage.dll!CRegDataDrivenCommand::_TryInvokeAssociation(struct _CMINVOKECOMMANDINFOEX const *,struct IShellItemArray *)	Unknown
 	windows.storage.dll!CRegDataDrivenCommand::_Invoke(struct _CMINVOKECOMMANDINFOEX const *,struct IShellItemArray *,struct IBindCtx *)	Unknown
 	shell32.dll!CRegistryVerbsContextMenu::_Execute(struct _CMINVOKECOMMANDINFOEX *,unsigned int)	Unknown
 	shell32.dll!CRegistryVerbsContextMenu::InvokeCommand(struct _CMINVOKECOMMANDINFO *)	Unknown
 	shell32.dll!HDXA_LetHandlerProcessCommandEx ()	Unknown
 	shell32.dll!CDefFolderMenu::InvokeCommand(struct _CMINVOKECOMMANDINFO *)	Unknown
 	windows.storage.dll!CShellLink::_InvokeDirect(struct _CMINVOKECOMMANDINFOEX *)	Unknown
 	windows.storage.dll!CShellLink::_ResolveAndInvoke(struct _CMINVOKECOMMANDINFO *,int)	Unknown
 	windows.storage.dll!CShellLink::InvokeCommand(struct _CMINVOKECOMMANDINFO *)	Unknown
 	shell32.dll!HDXA_LetHandlerProcessCommandEx ()	Unknown
 	shell32.dll!CDefFolderMenu::InvokeCommand(struct _CMINVOKECOMMANDINFO *)	Unknown
 	shell32.dll!CShellExecute::_InvokeInProcExec(struct IContextMenu *)	Unknown
 	shell32.dll!CShellExecute::_InvokeCtxMenu(void)	Unknown
 	shell32.dll!CShellExecute::_DoExecute(void)	Unknown
 	shell32.dll!AppResolverTelemetry::TAR_RankChangedForItem::~TAR_RankChangedForItem(void)	Unknown
>	SHCore.dll!CSimpleHashTable<unsigned long,class Microsoft::WRL::ComPtr<class CStreamWriterTimeoutManager::CTimerIdAndWriters>,class CDefaultHashPolicy<unsigned long>,class CDefaultKeyCompare<unsigned long>,class CDefaultResizePolicy,class CDefaultRehashPolicy>::RemoveAll(void)	Unknown
 	kernel32.dll!BaseThreadInitThunk ()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/92d3af5151b4845066895b9448b42d33c4030438

commit 92d3af5151b4845066895b9448b42d33c4030438
Author: brucedawson <brucedawson@chromium.org>
Date: Wed Apr 20 21:29:04 2016

Add default EXE manifest to setup.exe

The gn built mini_installer was failing on Windows 10. This was
because the mini_installer was inadvertently launching Chrome when it
was not supposed to. This in turn was because setup.exe was missing its
manifest and therefore thought it was running on an earlier version of
Windows and therefore tried to pin Chrome's shortcut which on Windows 10
launches Chrome through a separate thread that runs inscrutable shell32
goop.

Messy problem, trivial fix.

BUG= 604752 

Review URL: https://codereview.chromium.org/1900343005

Cr-Commit-Position: refs/heads/master@{#388579}

[modify] https://crrev.com/92d3af5151b4845066895b9448b42d33c4030438/chrome/installer/setup/BUILD.gn

Yay - it worked! Build 588 is green.

now this is green it would be nice to move the Win10 FYI bot over to gn - the bot is here: https://build.chromium.org/p/chromium.fyi/builders/Chromium%20Win%2010 do you know the correct incantations to do this?

I do, and I can handle that for you as part of flipping all of the other bots over over the next few days.