Seems to be failing since https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win64%20trunk/builds/19036

Oh no, they are failing since the beginning of time https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win64%20trunk/builds/16240

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5f384a7a92629b07253067a57ac3ffd5ca1ec150

commit 5f384a7a92629b07253067a57ac3ffd5ca1ec150
Author: jdoerrie <jdoerrie@chromium.org>
Date: Wed Jul 19 09:10:07 2017

Disable Failing SitePerProcessBrowserTests on Windows

This change disables failing SitePerProcessBrowserTests on Windows.
http://uberchromegw/i/official.desktop.continuous/builders/win64%20trunk/builds/19098/steps/content_browsertests

TBR=creis@chromium.org

Bug:  746055 
Change-Id: I94f89a739084af1e39e89efebbd74d318d428414
Reviewed-on: https://chromium-review.googlesource.com/575060
Reviewed-by: Jan Wilken Dörrie <jdoerrie@chromium.org>
Commit-Queue: Jan Wilken Dörrie <jdoerrie@chromium.org>
Cr-Commit-Position: refs/heads/master@{#487798}
[modify] https://crrev.com/5f384a7a92629b07253067a57ac3ffd5ca1ec150/content/browser/site_per_process_browsertest.cc

Failing Tests are disabled on Windows, thus removing Sheriff-label.

I'm assigning this to alexmos@ as the initial author of the AllowFullscreen test in r395496 since I am unable to find a recent culprit. Please feel free to re-route this further.

Disabling all these tests for Windows is very concerning, as they provide coverage for some core functionality for site isolation.  However, I'm not so sure that the problem is with the tests.  Looking at the logs, it seems many more tests were timing out and succeeded only after a few restarts, taking a *very* long time to complete, in many cases just under the 30s threshold for timeout.  For example:

[2146/2188] SitePerProcessBrowserTest.NavigateRemoteFrameToBlankAndDataURLs (29657 ms)
Still waiting for the following processes to finish:
	"C:\b\c\b\win64_trunk\src\out\Release_x64\content_browsertests.exe" --data-path="C:\Users\CHROME~1\AppData\Local\Temp\scoped_dir5696_32708\d5696_26131" --gtest_also_run_disabled_tests --gtest_filter=SitePerProcessBrowserTest.NavigateRemoteFrameToKilledProcess --single_process --test-launcher-bot-mode --test-launcher-jobs=4 --test-launcher-summary-output="c:\users\chrome~1\appdata\local\temp\tmpljmnpt.json" --use-fake-device-for-media-stream
[2147/2188] SitePerProcessBrowserTest.NavigateRemoteFrameToKilledProcess (18507 ms)
Still waiting for the following processes to finish: ...
[2148/2188] SitePerProcessBrowserTest.NavigateRemoteFrameToKilledProcessWithSubtree (20924 ms)
Still waiting for the following processes to finish: ...
[2149/2188] SitePerProcessBrowserTest.NavigateSiblingsToSameProcess (16311 ms)
Still waiting for the following processes to finish: ...
[2150/2188] SitePerProcessBrowserTest.NavigateSubframeToAboutBlankInSessionHistory (16269 ms)
Still waiting for the following processes to finish: ...
[2151/2188] SitePerProcessBrowserTest.NavigateSubframeToDataUrlInSessionHistory (16314 ms)
[2152/2188] SitePerProcessBrowserTest.NavigatingSubframePreservesOpenerInParent (14197 ms)
Still waiting for the following processes to finish: ...
[2153/2188] SitePerProcessBrowserTest.OpenPopupWithRemoteParent (16520 ms)
Still waiting for the following processes to finish: ...
[2154/2188] SitePerProcessBrowserTest.OriginReplication (27437 ms)

It doesn't seem normal that any of these tests would be taking so long.  So could there be something wrong with that machine, or something configured on it that makes these tests so slow?

Based on this, could it be that the tests disabled in #3 were just taking long enough to always run out of time on the bot?  All the failures seem to be timeouts, not crashes.  On the other hand, I'd expect to see some flakiness, but this set is failing pretty consistently across the different builds.  

In any case, we should investigate ASAP.

FWIW, we've also observed unexplained timeouts in issue 729662.

Reassigning to today's trooper to look at whether this might be an infra issue with the bot.  Also adding a couple of other infra people.  Does anyone know why content_browsertests running on this bot seem to take 100x the time they take on other bots?  For example: random test comparison for this bot to  https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win64%20beta/:

win64-trunk: (this bot)
[1808/2033] CrossSiteTransferTest/CrossSiteTransferTest.ReplaceEntryCrossProcessTwice/0 (22480 ms)
[1809/2033] CrossSiteTransferTest/CrossSiteTransferTest.ReplaceEntryCrossProcessTwice/1 (22544 ms)

win64-beta:
[1765/1975] CrossSiteTransferTest/CrossSiteTransferTest.ReplaceEntryCrossProcessTwice/0 (248 ms)
[1767/1975] CrossSiteTransferTest/CrossSiteTransferTest.ReplaceEntryCrossProcessTwice/1 (246 ms)

Anything we can do to debug and understand the slowness, at least on content_browsertests?  As I mentioned in #5, I suspect this is what's causing the current failures.

These failures were specific to machine win-25-m0 https://uberchromegw.corp.google.com/i/official.desktop.continuous/buildslaves/win-25-m0?numbuilds=200
it didn't have a green build since at least May

i took this machine out of rotation by creating
C:\b\build\slave\shutdown.slave

Labs, can we recreate/reprovision this machine?

win-25-m0 was reimaged last week, and looks to have picked up a build this morning:

https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win64%20beta/builds/683

Thanks!  It seems win-25-m0 was failing differently though, and win64trunk still fails the same way on content_browsertests: https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win64%20trunk

Currently it looks like it's got 
win-41-m0 through win-49-m0 assigned to it, and I can't see the last time any of those machines had a good build.  Not sure if reprovisioning these as well might help.

#8 pointed at win-25-m0 being an issue, which backs win64 beta and win64 stable. Those bots are win-{25..39}-m0

Now in #11 we're talking about bots that back the win64 trunk target, which are also being discussed in issue 650175

Seems like we're duplicating work between two tickets?

Thanks for pointing out issue 650175.  It seems to me that this issue is a dupe of that one then, as it was filed for SitePerProcessBrowserTests on win64 trunk builder, not win64 beta or stable.  I'm not actually sure where win-25-m0 came from in #8.

It seems to me that fixing issue 650175 will also fix the SitePerProcessBrowserTests tracked here.  Hence, I'll go ahead and revert the CL that disabled them, and merge this bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/777a637f9b90885d3e61758bce97b56dbaa19dbd

commit 777a637f9b90885d3e61758bce97b56dbaa19dbd
Author: Alex Moshchuk <alexmos@chromium.org>
Date: Tue Jul 25 21:59:43 2017

Revert "Disable Failing SitePerProcessBrowserTests on Windows"

This reverts commit 5f384a7a92629b07253067a57ac3ffd5ca1ec150.

Reason for revert: These tests were failing on win64 trunk builder together with many other tests.  These failures are tracked in https://crbug.com/650175 and are likely caused by config issues with that bot that cause tests to run much slower than they should.  There's likely nothing wrong with these particular tests, so let's re-enable them.

Original change's description:
> Disable Failing SitePerProcessBrowserTests on Windows
> 
> This change disables failing SitePerProcessBrowserTests on Windows.
> http://uberchromegw/i/official.desktop.continuous/builders/win64%20trunk/builds/19098/steps/content_browsertests
> 
> TBR=creis@chromium.org
> 
> Bug:  746055 
> Change-Id: I94f89a739084af1e39e89efebbd74d318d428414
> Reviewed-on: https://chromium-review.googlesource.com/575060
> Reviewed-by: Jan Wilken Dörrie <jdoerrie@chromium.org>
> Commit-Queue: Jan Wilken Dörrie <jdoerrie@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#487798}

TBR=creis@chromium.org,jdoerrie@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  746055 
Change-Id: Ie857ab080467dcf55691c4cb53e71b130d474a31
Reviewed-on: https://chromium-review.googlesource.com/584651
Reviewed-by: Alex Moshchuk <alexmos@chromium.org>
Commit-Queue: Alex Moshchuk <alexmos@chromium.org>
Cr-Commit-Position: refs/heads/master@{#489456}
[modify] https://crrev.com/777a637f9b90885d3e61758bce97b56dbaa19dbd/content/browser/site_per_process_browsertest.cc