Actually, this started flaking sporadically yesterday. So not fdoray's CL.

kkhorimoto@ any chance your crw_web_controller change could be racing with shutdown here?

https://codereview.chromium.org/2820013003

No, CRWWebController is only used on the iOS bots.

this also causes crashing failures on the mac bots without ASan.

https://build.chromium.org/p/chromium.mac/builders/Mac10.9%20Tests%20%28dbg%29

https://build.chromium.org/p/chromium.mac/builders/Mac10.9%20Tests%20%28dbg%29/builds/39551/steps/browser_tests

Stacktrace: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.mac%2FMac10.9_Tests__dbg_%2F39551%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Flogs%2FBrowserCloseManagerBrowserTest__x2f_BrowserCloseManagerBrowserTest.AddBeforeUnloadDuringClosing__x2f_0%2F0

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f18929c777dd3d711bebdb7927a32d3133f4e066

commit f18929c777dd3d711bebdb7927a32d3133f4e066
Author: rogerm <rogerm@chromium.org>
Date: Wed Apr 19 19:59:33 2017

Disable BrowserCloseManagerBrowserTest.AddBeforeUnloadDuringClosing on Mac.

BUG=713201
TBR=mkolom@yandex-team.ru, creis@chromium.org, sky@chromium.org, dgozman@chromium.org

Review-Url: https://codereview.chromium.org/2829773002
Cr-Commit-Position: refs/heads/master@{#465719}

[modify] https://crrev.com/f18929c777dd3d711bebdb7927a32d3133f4e066/chrome/browser/lifetime/browser_close_manager_browsertest.cc

mkolom@, this is the test you added in r454192.  Would you be able to help look at why it might have started failing?

Assigning to sky@ to help triage or otherwise investigate, since I think he understood the CL more than I did.

Any chance it's https://codereview.chromium.org/2554123002/?

This is according to Findit's analysis, https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVy6AELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKxAWNocm9taXVtLm1hYy9NYWMxMC45IFRlc3RzIChkYmcpLzM5NTUwL2Jyb3dzZXJfdGVzdHMvUW5KdmQzTmxja05zYjNObFRXRnVZV2RsY2tKeWIzZHpaWEpVWlhOMEwwSnliM2R6WlhKRGJHOXpaVTFoYm1GblpYSkNjbTkzYzJWeVZHVnpkQzVCWkdSQ1pXWnZjbVZWYm14dllXUkVkWEpwYm1kRGJHOXphVzVuTHpBPQwLEhNNYXN0ZXJGbGFrZUFuYWx5c2lzGAEM

Can someone please help confirm?

Sorry, I'm busy at the moment and won't be able to get to this until Friday (at the soonest). I'm also sheriff on Friday...

That's a very cool tool!

That CL does have this warning in the description:
"SHERIFFS: Don't hesitate to roll this back if it correlates well with
some kind of instability. Sampling has been known to have odd effects
in the past and this rewrites a large part of it."

However, wittman@ mentioned on the CL that it should have only affected 64-bit Windows.  (Are we confident about that?)

I don't see anything else obvious in the build it supposedly started flaking, but maybe the analysis could be wrong:
https://luci-milo.appspot.com/buildbot/chromium.mac/Mac10.9%20Tests%20(dbg)/39545

Note that the failures themselves seem to be crashes in Mac code as the test is cleaning up, which would also seem unlikely from the suspected CL:

[ RUN      ] BrowserCloseManagerBrowserTest/BrowserCloseManagerBrowserTest.AddBeforeUnloadDuringClosing/0
[34062:53019:0419/103413.750860:WARNING:mac_util.mm(222)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.MJD8Qg/db1O8Oq/Default/History': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn?t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50)
[34062:53019:0419/103413.785326:WARNING:mac_util.mm(222)] Failed to set backup exclusion for file '/private/var/folders/9x/6c6sv3cj4j53wzpzthbp4ksm0000gm/T/.org.chromium.Chromium.MJD8Qg/db1O8Oq/Default/Favicons': Error Domain=NSOSStatusErrorDomain Code=-50 "The operation couldn?t be completed. (OSStatus error -50.)" (paramErr: error in user parameter list) (-50)
[34070:771:0419/103413.933584:WARNING:vt_video_decode_accelerator_mac.cc(178)] Failed to create VTDecompressionSession: Error Domain=NSOSStatusErrorDomain Code=-8973 "The operation couldn?t be completed. (OSStatus error -8973.)" (codecOpenErr) (-8973)
[34070:771:0419/103413.934986:WARNING:vt_video_decode_accelerator_mac.cc(200)] Failed to create hardware VideoToolbox session
[34070:771:0419/103414.021239:ERROR:vt_video_encode_accelerator_mac.cc(495)]  VTCompressionSessionCreate failed: -12908
[34062:30211:0419/103414.481367:WARNING:simple_synchronous_entry.cc(1054)] Could not open platform files for entry.
[34062:55307:0419/103414.574419:WARNING:embedded_test_server.cc(219)] Request not handled. Returning 404: /favicon.ico
BrowserTestBase received signal: Segmentation fault: 11. Backtrace:
0   libbase.dylib                       0x0000000120643cee base::debug::StackTrace::StackTrace(unsigned long) + 174
1   libbase.dylib                       0x0000000120643d8d base::debug::StackTrace::StackTrace(unsigned long) + 29
2   libbase.dylib                       0x00000001206421ec base::debug::StackTrace::StackTrace() + 28
3   browser_tests                       0x000000010c352fe0 content::(anonymous namespace)::DumpStackTraceSignalHandler(int) + 560
4   libsystem_platform.dylib            0x00007fff8adb25aa _sigtramp + 26
5   ???                                 0x00007ff0c35fc080 0x0 + 140672046710912
6   libui_base.dylib                    0x0000000128c19873 l10n_util::GetStringUTF16(int) + 51
7   libui_base.dylib                    0x0000000128c3067c l10n_util::GetNSStringWithFixup(int) + 44
8   browser_tests                       0x00000001108a1408 -[ReloadButton updateTag:] + 200
9   browser_tests                       0x00000001108a16cf -[ReloadButton setIsLoading:force:] + 143
10  browser_tests                       0x00000001108a11ae -[ReloadButton viewWillMoveToWindow:] + 62
11  AppKit                              0x00007fff8a23e70a -[NSView _setWindow:] + 205
12  AppKit                              0x00007fff8a250de5 -[NSControl _setWindow:] + 154
13  AppKit                              0x00007fff8a250c9e -[NSButton _setWindow:] + 48
14  CoreFoundation                      0x00007fff86df72f9 __53-[__NSArrayM enumerateObjectsWithOptions:usingBlock:]_block_invoke + 137
15  CoreFoundation                      0x00007fff86df6a9f -[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 319
16  AppKit                              0x00007fff8a982c72 __21-[NSView _setWindow:]_block_invoke712 + 231
17  AppKit                              0x00007fff8a23f15f -[NSView _setWindow:] + 2850
18  CoreFoundation                      0x00007fff86df72f9 __53-[__NSArrayM enumerateObjectsWithOptions:usingBlock:]_block_invoke + 137
19  CoreFoundation                      0x00007fff86df6a9f -[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 319
20  AppKit                              0x00007fff8a982c72 __21-[NSView _setWindow:]_block_invoke712 + 231
21  AppKit                              0x00007fff8a23f15f -[NSView _setWindow:] + 2850
22  CoreFoundation                      0x00007fff86df72f9 __53-[__NSArrayM enumerateObjectsWithOptions:usingBlock:]_block_invoke + 137
23  CoreFoundation                      0x00007fff86df6a9f -[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 319
24  AppKit                              0x00007fff8a982c72 __21-[NSView _setWindow:]_block_invoke712 + 231
25  AppKit                              0x00007fff8a23f15f -[NSView _setWindow:] + 2850
26  CoreFoundation                      0x00007fff86df72f9 __53-[__NSArrayM enumerateObjectsWithOptions:usingBlock:]_block_invoke + 137
27  CoreFoundation                      0x00007fff86df6a9f -[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 319
28  AppKit                              0x00007fff8a982c72 __21-[NSView _setWindow:]_block_invoke712 + 231
29  AppKit                              0x00007fff8a23f15f -[NSView _setWindow:] + 2850
30  AppKit                              0x00007fff8a2480ed -[NSThemeFrame _setWindow:] + 111
31  AppKit                              0x00007fff8a495ed9 -[NSWindow dealloc] + 1163
32  browser_tests                       0x000000011073a31d -[FramedBrowserWindow dealloc] + 125
33  AppKit                              0x00007fff8a2475ea -[NSWindow release] + 193
34  AppKit                              0x00007fff8a4d906b -[_NSWindowTransformAnimation dealloc] + 115
35  libsystem_blocks.dylib              0x00007fff8dbf16f0 _Block_release + 196
36  CoreFoundation                      0x00007fff86de5aed __CFRunLoopDoBlocks + 349
37  CoreFoundation                      0x00007fff86de53de __CFRunLoopRun + 814
38  CoreFoundation                      0x00007fff86de4e75 CFRunLoopRunSpecific + 309
39  libbase.dylib                       0x000000012074d31c base::MessagePumpCFRunLoop::DoRun(base::MessagePump::Delegate*) + 76
40  libbase.dylib                       0x000000012074c9fa base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*) + 122
41  libbase.dylib                       0x0000000120738f52 base::MessageLoop::RunHandler() + 610
42  libbase.dylib                       0x000000012081dfab base::RunLoop::Run() + 267
43  browser_tests                       0x000000010c57b55f net::test_server::EmbeddedTestServer::PostTaskToIOThreadAndWait(base::Callback<void (), (base::internal::CopyMode)1, (base::internal::RepeatMode)1> const&) + 863
44  browser_tests                       0x000000010c5775a7 net::test_server::EmbeddedTestServer::ShutdownAndWaitUntilComplete() + 295
45  browser_tests                       0x000000010c576dd3 net::test_server::EmbeddedTestServer::~EmbeddedTestServer() + 243
46  browser_tests                       0x000000010c577685 net::test_server::EmbeddedTestServer::~EmbeddedTestServer() + 21
47  browser_tests                       0x000000010c35161e content::BrowserTestBase::~BrowserTestBase() + 478
48  browser_tests                       0x000000010a3fa039 InProcessBrowserTest::~InProcessBrowserTest() + 249
49  browser_tests                       0x0000000107309322 BrowserCloseManagerBrowserTest::~BrowserCloseManagerBrowserTest() + 98
50  browser_tests                       0x00000001073096c5 BrowserCloseManagerBrowserTest_AddBeforeUnloadDuringClosing_Test::~BrowserCloseManagerBrowserTest_AddBeforeUnloadDuringClosing_Test() + 21
51  browser_tests                       0x00000001072ff225 BrowserCloseManagerBrowserTest_AddBeforeUnloadDuringClosing_Test::~BrowserCloseManagerBrowserTest_AddBeforeUnloadDuringClosing_Test() + 21
52  browser_tests                       0x00000001072ff249 BrowserCloseManagerBrowserTest_AddBeforeUnloadDuringClosing_Test::~BrowserCloseManagerBrowserTest_AddBeforeUnloadDuringClosing_Test() + 25
53  browser_tests                       0x000000010c63c76b testing::Test::DeleteSelf_() + 43
54  browser_tests                       0x000000010c65db3e void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 126
55  browser_tests                       0x000000010c63b502 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 114
56  browser_tests                       0x000000010c63c5c5 testing::TestInfo::Run() + 293
57  browser_tests                       0x000000010c63d8ec testing::TestCase::Run() + 236
58  browser_tests                       0x000000010c64b4ba testing::internal::UnitTestImpl::RunAllTests() + 922
59  browser_tests                       0x000000010c6609ee bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) + 126
60  browser_tests                       0x000000010c64b0a2 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) + 114
61  browser_tests                       0x000000010c64af79 testing::UnitTest::Run() + 393

> However, wittman@ mentioned on the CL that it should have only affected 64-bit Windows.  (Are we confident about that?)

Yes. 99% of this code in that CL only runs on 64-bit Windows. The remaining 1% runs just long enough at startup to determine that the other platforms are not supported. I can't see any mechanism by which this could cause an apparent destruction ordering issue in Mac UI code.

Ok, thanks-- probably not StackSampler.  (Just cc'ing Avi in case either the Mac crash or StackSampler stuff rings a bell.)

Thanks guys for the input, the culprit for this test flaking is a tough case! Findit team and I will continue to improve the flake analyzer's algorithms to provide culprits with more accuracy.

@sky can you ptal at this disabled test? Can we just delete it at this point?