New issue
Advanced search Search tips

Issue 841157 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2018
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----



Sign in to add a comment

unit_tests failing on chromium.win/Win10 Tests x64

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, May 9 2018

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of vitaliii@google.com

unit_tests failing on chromium.win/Win10 Tests x64

Builders failed on: 
- Win10 Tests x64: 
  https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64


 
Cc: -vitaliii@google.com
Owner: vitaliii@chromium.org
Status: Started (was: Available)
The first failing build is 
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64/23277
Presumably relevant bit of log

==============
SUCCESS: all tests passed.
Tests took 485 seconds.
Failed to delete C:\b\s\w\ir (20 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 2 seconds.
Failed to delete C:\b\s\w\ir (20 files remaining).
  Maybe the test has a subprocess outliving it.
  Sleeping 4 seconds.
Failed to delete C:\b\s\w\ir. The following files remain:
- \\?\C:\b\s\w\ir
- \\?\C:\b\s\w\ir\out
- \\?\C:\b\s\w\ir\out\Release_x64
- \\?\C:\b\s\w\ir\out\Release_x64\chrome.dll
- \\?\C:\b\s\w\ir\out\Release_x64\chrome.exe
- \\?\C:\b\s\w\ir\out\Release_x64\chrome_100_percent.pak
- \\?\C:\b\s\w\ir\out\Release_x64\chrome_200_percent.pak
- \\?\C:\b\s\w\ir\out\Release_x64\chrome_child.dll
- \\?\C:\b\s\w\ir\out\Release_x64\chrome_elf.dll
- \\?\C:\b\s\w\ir\out\Release_x64\d3dcompiler_47.dll
- \\?\C:\b\s\w\ir\out\Release_x64\icudtl.dat
- \\?\C:\b\s\w\ir\out\Release_x64\initialexe
- \\?\C:\b\s\w\ir\out\Release_x64\initialexe\chrome.exe
- \\?\C:\b\s\w\ir\out\Release_x64\libEGL.dll
- \\?\C:\b\s\w\ir\out\Release_x64\libGLESv2.dll
- \\?\C:\b\s\w\ir\out\Release_x64\locales
- \\?\C:\b\s\w\ir\out\Release_x64\locales\en-US.pak
- \\?\C:\b\s\w\ir\out\Release_x64\natives_blob.bin
- \\?\C:\b\s\w\ir\out\Release_x64\resources.pak
- \\?\C:\b\s\w\ir\out\Release_x64\v8_context_snapshot.bin
Enumerating processes:
- pid 688; Handles: 137; Exe: C:\b\s\w\ir\out\Release_x64\chrome.exe; Cmd: C:\b\s\w\ir\out\Release_x64\chrome.exe --type=crashpad-handler "--user-data-dir=C:\Users\chrome-bot\AppData\Local\Chromium\User Data" /prefetch:7 --monitor-self-annotation=ptype=crashpad-handler "--database=C:\Users\chrome-bot\AppData\Local\Chromium\User Data\Crashpad" "--metrics-dir=C:\Users\chrome-bot\AppData\Local\Chromium\User Data" --annotation=plat=Win64 --annotation=prod=Chromium --annotation=ver=68.0.3425.0-devel --initial-client-data=0x190,0x194,0x198,0x18c,0x19c,0x7ffd137b6250,0x7ffd137b6260,0x7ffd137b6270
- pid 1944; Handles: 420; Exe: C:\b\s\w\ir\out\Release_x64\chrome.exe; Cmd: "C:\b\s\w\ir\out\Release_x64\chrome.exe" --type=gpu-process --field-trial-handle=1632,8640685463012438446,17892973344504919221,131072 --noerrdialogs --start-stack-profiler --gpu-preferences=KAAAAAAAAACAAwBgAQAAAAAAAAAAAGAAEAAAAAAAAAAAAAAAAAAAACgAAAAEAAAAIAAAAAAAAAAoAAAAAAAAADAAAAAAAAAAOAAAAAAAAAAQAAAAAAAAAAAAAAAKAAAAEAAAAAAAAAAAAAAACwAAABAAAAAAAAAAAQAAAAoAAAAQAAAAAAAAAAEAAAALAAAA --noerrdialogs --start-stack-profiler --service-request-channel-token=545469409D897F62EB81F47A43B5465B --mojo-platform-channel-handle=1652 --ignored=" --type=renderer " /prefetch:2
- pid 7748; Handles: 244; Exe: C:\b\s\w\ir\out\Release_x64\chrome.exe; Cmd: "C:\b\s\w\ir\out\Release_x64\chrome.exe" --type=renderer --field-trial-handle=1632,8640685463012438446,17892973344504919221,131072 --service-pipe-token=782ADE8CD2A0094D6A4DFB8B6F6538CC --lang=en-US --noerrdialogs --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=cssExternalScannerNoPreload=false,cssExternalScannerPreload=true --device-scale-factor=1 --num-raster-threads=4 --enable-main-frame-before-activation --service-request-channel-token=782ADE8CD2A0094D6A4DFB8B6F6538CC --renderer-client-id=4 --mojo-platform-channel-handle=2428 /prefetch:1
- pid 7924; Handles: 969; Exe: C:\b\s\w\ir\out\Release_x64\chrome.exe; Cmd: "C:\b\s\w\ir\out\Release_x64\chrome.exe"  --notification-launch-id=0|0|Default|0|https://example.com/|notification_id
Terminating 4 processes:
- 688 killed
- 1944 killed
- 7748 killed
- 7924 killed
*** Swarming tried multiple times to delete the run directory and failed ***
*** Hard failing the task ***
Swarming detected that your testing script ran an executable, which may have
started a child executable, and the main script returned early, leaving the
children executables playing around unguided.
You don't want to leave children processes outliving the task on the Swarming
bot, do you? The Swarming bot doesn't.
How to fix?
- For any process that starts children processes, make sure all children
  processes terminated properly before each parent process exits. This is
  especially important in very deep process trees.
  - This must be done properly both in normal successful task and in case of
    task failure. Cleanup is very important.
- The Swarming bot sends a SIGTERM in case of timeout.
  - You have 30.0 seconds to comply after the signal was sent to the process
    before the process is forcibly killed.
- To achieve not leaking children processes in case of signals on timeout, you
  MUST handle signals in each executable / python script and propagate them to
  children processes.
  - When your test script (python or binary) receives a signal like SIGTERM or
    CTRL_BREAK_EVENT on Windows), send it to all children processes and wait for
    them to terminate before quitting.
See
https://github.com/luci/luci-py/blob/master/appengine/swarming/doc/Bot.md#graceful-termination-aka-the-sigterm-and-sigkill-dance
for more information.
*** May the SIGKILL force be with you ***
+--------------------------------------------------------------------------------+
| End of shard 0                                                                 |
|  Pending: 0.4s  Duration: 487.0s  Bot: win10-0af2fd95-us-west1-b-959b  Exit: 1 |
+--------------------------------------------------------------------------------+
==============
All of them seem to fail in the same way.
There are also tests which fail first time, but pass on the retry.

e.g.

==========
[ RUN      ] ChromeDownloadManagerDelegateTest.LastSavePath
unknown file: error: 
Unexpected mock function call - returning directly.
    Function call: RequestConfirmation(000001B0395CEC40, @000001B03718B3E0 C:\Users\chrome-bot\Downloads\foo.txt, 4-byte object <02-00 00-00>, @000000A417FFE370 8-byte object <40-B8 5C-39 B0-01 00-00>)
Google Mock tried the following 1 expectation, but it didn't match:
../../chrome/browser/download/chrome_download_manager_delegate_unittest.cc(432): EXPECT_CALL(*delegate(), RequestConfirmation(save_as_download.get(), expected_prompt_path, _, _))...
  Expected arg #1: is equal to C:\Users\CHROME~1\AppData\Local\Temp\scoped_dir8996_14319\foo.txt
           Actual: C:\Users\chrome-bot\Downloads\foo.txt
         Expected: to be called once
           Actual: never called - unsatisfied and active
Stack trace:
Backtrace:
	testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop [0x00007FF7692DAA87+87]
	testing::internal::AssertHelper::operator= [0x00007FF7692DA60E+78]
	testing::internal::GoogleTestFailureReporter::ReportFailure [0x00007FF7692CBEED+109]
	testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith [0x00007FF7692CE84F+1023]
	testing::internal::FunctionMockerBase<void __cdecl(base::OnceCallback<void __cdecl(std::basic_string<char,std::char_traits<char>,std::allocator<char> > const & __ptr64)> & __ptr64)>::InvokeWith [0x00007FF768867DBA+10]
	testing::internal::FunctionMockerBase<enum safe_browsing::AsyncMatch __cdecl(GURL const & __ptr64,safe_browsing::SafeBrowsingDatabaseManager::Client * __ptr64)>::InvokeWith [0x00007FF7682F1AF2+786]
	DownloadTargetDeterminer::DoRequestConfirmation [0x00007FF76B37C639+217]
	DownloadTargetDeterminer::DoLoop [0x00007FF76B37BE23+83]
	base::internal::Invoker<base::internal::BindState<base::RepeatingCallback<void __cdecl(int,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const & __ptr64)>,int,std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,vo [0x00007FF76C2DAC29+57]
	base::debug::TaskAnnotator::RunTask [0x00007FF76B1BEC25+309]
	base::MessageLoop::RunTask [0x00007FF76B1589AC+572]
	base::MessageLoop::DoWork [0x00007FF76B158F08+408]
	base::MessagePumpForUI::DoRunLoop [0x00007FF76B1A71F9+169]
	base::MessagePumpWin::Run [0x00007FF76B1A6D58+104]
	base::RunLoop::Run [0x00007FF76B13BAD1+49]
	testing::Matcher<base::FilePath const & __ptr64>::Matcher<base::FilePath const & __ptr64> [0x00007FF7682EC800+608]
	ChromeDownloadManagerDelegateTest_LastSavePath_Test::TestBody [0x00007FF7682EAC8F+2095]
[8839/8842] ChromeDownloadManagerDelegateTest.LastSavePath (TIMED OUT)
======

This seems to happen pretty consistently.
======
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for SyncInternalsMessageHandlerTest.WriteUserEventBlank
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for SyncInternalsMessageHandlerTest.WriteUserEventZero
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for WebDialogWebContentsDelegateTest.DoNothingMethodsTest
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for WebDialogWebContentsDelegateTest.OpenURLFromTabTest
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for WebDialogWebContentsDelegateTest.AddNewContentsForegroundTabTest
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for WebDialogWebContentsDelegateTest.DetachTest
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for WindowSizerTestCommon.PersistedWindowOffscreenWithNonAggressiveRepositioning
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for WindowSizerTestCommon.AdjustFitSize
[3340:7664:0508/153715.297:630171:ERROR:unit_test_launcher.cc(338)] no test result for WindowSizerTest.DefaultSizeCase
[3801/8842] SyncInternalsMessageHandlerTest.WriteUserEventBadParse (18 ms)
[ RUN      ] SyncInternalsMessageHandlerTest.WriteUserEventBlank
[2560:6604:0508/153714.172:629046:FATAL:sequenced_task_runner_handle.cc(33)] Check failed: ThreadTaskRunnerHandle::IsSet(). Error: This caller requires a sequenced context (i.e. the current task needs to run from a SequencedTaskRunner).
Backtrace:
	base::debug::StackTrace::StackTrace [0x00007FF6B180F424+36]
	logging::LogMessage::~LogMessage [0x00007FF6B17915A5+101]
	base::SequencedTaskRunnerHandle::Get [0x00007FF6B17B46CB+219]
	mojo::internal::GetTaskRunnerToUseFromUserProvidedTaskRunner [0x00007FF6B11B7CC0+48]
	mojo::internal::InterfacePtrStateBase::Bind [0x00007FF6B11AE458+104]
	mojo::internal::InterfacePtrState<bluetooth::mojom::Adapter>::Bind [0x00007FF6B44A2405+69]
	mojo::InterfacePtr<bluetooth::mojom::Adapter>::Bind [0x00007FF6B2395CA9+89]
	mojo::MakeRequest<bluetooth::mojom::Adapter> [0x00007FF6B0F5F97E+110]
	resource_coordinator::PageSignalReceiver::AddObserver [0x00007FF6B2F06E26+134]
	resource_coordinator::TabManager::ResourceCoordinatorSignalObserver::ResourceCoordinatorSignalObserver [0x00007FF6B2F351D7+39]
	resource_coordinator::TabManager::TabManager [0x00007FF6B2F30D25+437]
	TestingBrowserProcess::GetTabManager [0x00007FF6B1134B5E+46]
	ChromeUnitTestSuite::`scalar deleting destructor' [0x00007FF6B11365AB+411]
	testing::internal::TestEventRepeater::OnTestStart [0x00007FF6AF93035C+44]
	testing::TestInfo::Run [0x00007FF6AF92E4BE+94]
	testing::TestCase::Run [0x00007FF6AF92E9E2+258]
	testing::internal::UnitTestImpl::RunAllTests [0x00007FF6AF935B3E+638]
	testing::UnitTest::Run [0x00007FF6AF9357B5+165]
	base::TestSuite::Run [0x00007FF6B113F966+118]
	base::LaunchUnitTests [0x00007FF6B114094A+554]
	base::LaunchUnitTests [0x00007FF6B11407C8+168]
	main [0x00007FF6B40A3471+269]
	__scrt_common_main_seh [0x00007FF6B4B72E39+285] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
	BaseThreadInitThunk [0x00007FFBA6D72774+20]
	RtlUserThreadStart [0x00007FFBA7190D51+33]
[3802/8842] SyncInternalsMessageHandlerTest.WriteUserEventBlank (UNKNOWN)
======
Suspects:
https://chromium-review.googlesource.com/c/chromium/src/+/1049984 
I am not sure what it is doing, but "SEE_MASK_NOCLOSEPROCESS" seems related :)

https://chromium-review.googlesource.com/c/chromium/src/+/1038226
ICU rollout, but I have no idea what ICU is.

Comment 7 by grt@chromium.org, May 9 2018

Cc: finnur@chromium.org
The failures are a result of a launch of chrome.exe with --notification-launch-id, which certainly is related to native notifications. I think it's due to a handle leak. The revert in r557125 should clear it up.

Sign in to add a comment