telemetry_perf_unittests extremely flaky on Windows, blocking CQ |
|||||||||
Issue descriptionhttps://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng?numbuilds=200 https://chromium-swarm.appspot.com/task?id=379c58e51f945c10&refresh=10&show_raw=1 The error seems to be that we are leaking browser processes at the end of the tests: Failed to delete e:\b\s\w\ir. The following files remain: - \\?\e:\b\s\w\ir - \\?\e:\b\s\w\ir\out - \\?\e:\b\s\w\ir\out\Release - \\?\e:\b\s\w\ir\out\Release\chrome.dll - \\?\e:\b\s\w\ir\out\Release\chrome.exe - \\?\e:\b\s\w\ir\out\Release\chrome_100_percent.pak - \\?\e:\b\s\w\ir\out\Release\chrome_child.dll - \\?\e:\b\s\w\ir\out\Release\chrome_elf.dll - \\?\e:\b\s\w\ir\out\Release\d3dcompiler_47.dll - \\?\e:\b\s\w\ir\out\Release\icudtl.dat - \\?\e:\b\s\w\ir\out\Release\locales - \\?\e:\b\s\w\ir\out\Release\locales\en-US.pak - \\?\e:\b\s\w\ir\out\Release\natives_blob.bin - \\?\e:\b\s\w\ir\out\Release\resources.pak - \\?\e:\b\s\w\ir\out\Release\snapshot_blob.bin - \\?\e:\b\s\w\ir\out\Release\swiftshader - \\?\e:\b\s\w\ir\out\Release\swiftshader\libEGL.dll - \\?\e:\b\s\w\ir\out\Release\swiftshader\libGLESv2.dll Enumerating processes: - pid 932; Handles: 200; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=renderer --enable-logging --trace-config-file="e:\b\s\w\it6igb7o\tmpqcmcyp\chrome-trace-config.json" --v=0 --enable-gpu-benchmarking --field-trial-handle=1336,16828144388170565451,2606585767660807311,131072 --service-pipe-token=042B0DB4CE61F2E39D0D5E3E5940DEC5 --lang=en-US --noerrdialogs --user-data-dir="e:\b\s\w\it6igb7o\tmpmlxuv8" --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true,cssExternalScannerNoPreload=false,cssExternalScannerPreload=true --enable-net-benchmarking --enable-pinch --device-scale-factor=1 --num-raster-threads=4 --enable-main-frame-before-activation --enable-checker-imaging --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553 --enable-gpu-async-worker-context --service-request-channel-token=042B0DB4CE61F2E39D0D5E3E5940DEC5 --renderer-client-id=3 --mojo-platform-channel-handle=2188 /prefetch:1 - pid 4284; Handles: 91; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 /prefetch:7 --monitor-self --monitor-self-argument=--type=crashpad-handler --monitor-self-argument=--user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 --monitor-self-argument=/prefetch:7 --monitor-self-annotation=ptype=crashpad-handler --database=e:\b\s\w\it6igb7o\tmpqquqrw --metrics-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 --annotation=plat=Win32 --annotation=prod=Chromium --annotation=ver=62.0.3169.0-devel --initial-client-data=0x9c,0xa0,0xa4,0x98,0xa8,0x745b0a34,0x745b0a44,0x745b0a54 - pid 4296; Handles: 73; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 /prefetch:7 --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=e:\b\s\w\it6igb7o\tmpqquqrw --annotation=plat=Win32 --annotation=prod=Chromium --annotation=ver=62.0.3169.0-devel --initial-client-data=0xc0,0xc4,0xc8,0xbc,0xb8,0x1050a44,0x1050a54,0x1050a64 - pid 4300; Handles: 629; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --disable-features=LazyParseCSS<LazyParseCSS,TranslateRankerEnforcement<TranslateRankerModel --force-fieldtrials=AlternateComponentUrls/AlternateComponentUrls/AutofillClassifier/Enabled/AutofillCreditCardSigninPromo/EnabledThree/AutofillFieldMetadata/Enabled/AutofillProfileCleanup/Enabled/AutofillProfileOrderByFrecency/EnabledLimitTo3/AutomaticTabDiscarding/Enabled_Once_10-gen2/BackgroundVideoOptimizations/BackgroundOptimizationEnabled1sOrLessMediaSource/BrowserBlacklist/Enabled/BrowserScheduler/RedirectWithDefaultInitParams/CSSExternalScanner/Enabled_ScanAndPreload/CaptivePortalCertificateList/Enabled/CaptivePortalInterstitial/Enabled/CheckerImaging/CheckerImaging/ChildAccountDetection/Disabled/ChromeDashboard/Enabled/ChromotingQUIC/Enabled/DataReductionProxyUseQuic/Enabled/DefaultBrowserInfobar/SettingsTextNotNow/DefaultBrowserPromptStyle/ColoredLogoOnWhiteBackground/DefaultEnableGpuRasterization/DefaultEnableGpuRasterization/DelayNavigation/DelayNavigation/DesktopIOSPromotion/SavePasswordsBubblePromotionV1/DirectCompositionOverlays/DirectCompositionOverlays/DisallowFetchForDocWrittenScriptsInMainFrame/DocumentWriteScriptBlockGroup_20161208_Launch/EnableAppContainer/Enabled/EnableGoogleCachedCopyTextExperiment/Button/EnableMediaRemoting/Enabled/EnableMediaRouter/Enabled/EnableMediaRouterWithCastExtension/Enabled/EnableSessionCrashedBubbleUI/Enabled/EnableSyncClientToServerCompression/Enabled/EnableSyncUSSDeviceInfo/Enabled/ExpectCTDynamic/DynamicExpectCTEnabled/ExpectCTReporting/ExpectCTReportingEnabled/ExtensionContentVerification/Enforce/ExtensionInstallVerification/Enforce/FasterLocationReload/Enabled/GoogleBrandedContextMenu/branded/GoogleNow/Enable/GpuScheduler/Enabled/Html5ByDefault/Enabled/HttpFormWarning/Enabled/IdleTimeSpellChecking/Enabled/ImprovedRecoveryComponent/ImprovedRecoveryComponent/IncludeBetaForumMenuItem/Enabled/IncognitoWindowPromo/Enabled/InstanceID/Enabled/IntelligentSessionRestore/Enabled/LazyParseCSS/Control/LoadingWithMojo/Enabled/MaterialDesignDownloads/Enabled/MaterialDesignUserManager/Enabled/MediaFoundationH264Encoding/Enabled/MidiManagerDynamicInstantiation/Enabled/MojoChannel/Enabled/NTPCaptureThumbnail/Enabled/NTPTilesInInstantService/Enabled/NetDelayableH2AndQuicRequests/Enabled/NetworkQualityEstimator/Enabled/NetworkSchedulerYielding/Enabled/NewProfileManagement/Enabled/NewTabInProductHelp/Enabled_1/NoStatePrefetchValidation/PrerenderDisabled_R2/OfferUploadCreditCards/Enabled/OneGoogleBarOnLocalNtp/Enabled/OutOfProcessPac/Enabled/PageRevisitInstrumentation/Enabled/PassiveDocumentEventListeners/Enabled/PassiveEventListenersDueToFling/Enabled/PasswordBranding/SmartLockBrandingSavePromptOnly/PasswordGeneration/Disabled/PasswordMetadataFilling/Enabled/PasswordProtectionPasswordFieldOnFocusPing/SBEROnlyNoIncognito2/PasswordProtectionProtectedPasswordEntryPing/AllPopulation/PasswordSeparatedSigninFlow/Enabled/PasswordSmartBubble/3-Times/PermissionPromptUIViews/BlockPromptsEnabled/PersistentHistograms/EnabledOnDisk5/PreconnectMore/Enabled/PrintScaling/Enabled/QUIC/Enabled/RafAlignedMouseInput/Enabled/RafAlignedTouchInput/Enabled/RefreshTokenDeviceId/Enabled/ReportCertificateErrors/ShowAndPossiblySend/ReportingAPI/ReportingEnabled/SRTPromptFieldTrial/On/SSLCommonNameMismatchHandling/Enabled/SafeBrowsingIncidentReportingService/Enabled/SafeBrowsingIncidentReportingServiceFeatures/WithSuspiciousModuleReporting/SafeBrowsingScoutTransitionStudy/CanShowScoutOptInGroup2/SafeBrowsingThreatDomDetailsTagAttributes/AdIdentifiers/SafeBrowsingV4LocalDatabaseManagerEnabled/V4Only2/SaveAsMenuText/download/SchedulerExpensiveTaskBlocking/Enabled/SdchPersistence/Enabled/SecurityChip/Enabled/ServiceWorkerNavigationPreload/Enabled/SettingsEnforcement/enforce_always_with_extensions_and_dse/SettingsResetPrompt/Enabled/SignInPasswordPromo/Enable3/SocketReadIfReady/Enabled/SpeculativeLaunchServiceWorker/Enabled/StrictSecureCookies/Enabled/SubresourceFilter/EnabledForPhishingSites/SuperfishInterstitial/Enabled/SyncUSSAutocomplete/Enabled/SyncUserEvents/Enabled/TLS13Variant/DraftEnabled/TabSyncByRecency/Enabled/TokenBinding/TokenBinding/TranslateRankerModel/QueryExperiment2/TranslateServerStudy/SmartRendering/TranslateUI2016Q2/TranslateUI2016Q2OnGroup/TriggeredResetFieldTrial/On/UKM/Enabled/V8AsmJSToWasm/AsmJsToWebAssembly/V8CacheStrategiesForCacheStorage/default/V8Ignition/Future/VideoCaptureService/Enabled/VsyncAlignedInput/Enable/WebBluetoothBlocklist/TestGroup/WebFontsInterventionV2/Enabled-slow2g/WebRTC-EnableWebRtcEcdsa/Enabled/WebRTC-H264WithOpenH264FFmpeg/Enabled/WebRTC-LocalIPPermissionCheck/Enabled/use-new-media-cache/Enabled --enable-features=AlternateComponentUrls<AlternateComponentUrls,AutofillCreditCardSigninPromo<AutofillCreditCardSigninPromo,AutofillProfileCleanup<AutofillProfileCleanup,AutomaticTabDiscarding<AutomaticTabDiscarding,BackgroundVideoTrackOptimization<BackgroundVideoOptimizations,CaptivePortalCertificateList<CaptivePortalCertificateList,CheckerImaging<CheckerImaging,DefaultEnableGpuRasterization<DefaultEnableGpuRasterization,DelayNavigation<DelayNavigation,DesktopIOSPromotion<DesktopIOSPromotion,DirectCompositionOverlays<DirectCompositionOverlays,MediaRemoting<EnableMediaRemoting,EnableSyncClientToServerCompression<EnableSyncClientToServerCompression,EnableSyncUSSDeviceInfo<EnableSyncUSSDeviceInfo,DynamicExpectCT<ExpectCTDynamic,ExpectCTReporting<ExpectCTReporting,FasterLocationReload<FasterLocationReload,GpuScheduler<GpuScheduler,PreferHtmlOverPlugins<Html5ByDefault,HttpFormWarning<HttpFormWarning,IdleTimeSpellChecking<IdleTimeSpellChecking,ImprovedRecoveryComponent<ImprovedRecoveryComponent,IncludeBetaForumMenuItem<IncludeBetaForumMenuItem,IPH_IncognitoWindow<IncognitoWindowPromo,LoadingWithMojo<LoadingWithMojo,MaterialDesignUserManager<MaterialDesignUserManager,MediaFoundationH264Encoding<MediaFoundationH264Encoding,MidiManagerDynamicInstantiation<MidiManagerDynamicInstantiation,CaptureThumbnailDependingOnTransitionType<NTPCaptureThumbnail,CaptureThumbnailOnNavigatingAway<NTPCaptureThumbnail,NTPTilesInInstantService<NTPTilesInInstantService,PrioritySupportedRequestsDelayable<NetDelayableH2AndQuicRequests,NetworkSchedulerYielding<NetworkSchedulerYielding,IPH_NewTab<NewTabInProductHelp,NoStatePrefetch<NoStatePrefetchValidation,OneGoogleBarOnLocalNtp<OneGoogleBarOnLocalNtp,PassiveDocumentEventListeners<PassiveDocumentEventListeners,PassiveEventListenersDueToFling<PassiveEventListenersDueToFling,PasswordMetadataFilling<PasswordMetadataFilling,PasswordFieldOnFocusPinging<PasswordProtectionPasswordFieldOnFocusPing,ProtectedPasswordEntryPinging<PasswordProtectionProtectedPasswordEntryPing,UsePasswordSeparatedSigninFlow<PasswordSeparatedSigninFlow,BlockPromptsIfDismissedOften<PermissionPromptUIViews,PersistentHistograms<PersistentHistograms,PreconnectMore<PreconnectMore,PrintScaling<PrintScaling,RafAlignedMouseInput<RafAlignedMouseInput,RafAlignedTouchInput<RafAlignedTouchInput,Reporting<ReportingAPI,IncidentReportingDisableUpload<SafeBrowsingIncidentReportingService,IncidentReportingModuleLoadAnalysis<SafeBrowsingIncidentReportingServiceFeatures,IncidentReportingSuspiciousModuleReporting<SafeBrowsingIncidentReportingServiceFeatures,CanShowScoutOptIn<SafeBrowsingScoutTransitionStudy,ThreatDomDetailsTagAttributes<SafeBrowsingThreatDomDetailsTagAttributes,SafeBrowsingV4LocalDatabaseManagerEnabled<SafeBrowsingV4LocalDatabaseManagerEnabled,SafeBrowsingV4OnlyEnabled<SafeBrowsingV4LocalDatabaseManagerEnabled,SecurityChip<SecurityChip,ServiceWorkerNavigationPreload<ServiceWorkerNavigationPreload,SettingsResetPrompt<SettingsResetPrompt,SocketReadIfReady<SocketReadIfReady,SpeculativeLaunchServiceWorker<SpeculativeLaunchServiceWorker,SubresourceFilter<SubresourceFilter,SuperfishInterstitial<SuperfishInterstitial,SyncUSSAutocomplete<SyncUSSAutocomplete,SyncUserEvents<SyncUserEvents,token-binding<TokenBinding,TranslateRankerQuery<TranslateRankerModel,TranslateUI2016Q2<TranslateUI2016Q2,Ukm<UKM,AsmJsToWebAssembly<V8AsmJSToWasm,V8Future<V8Ignition,MojoVideoCapture<VideoCaptureService,RequestUnbufferedDispatch<VsyncAlignedInput,VsyncAlignedInput<VsyncAlignedInput,WebRTC-EnableWebRtcEcdsa<WebRTC-EnableWebRtcEcdsa,WebRTC-H264WithOpenH264FFmpeg<WebRTC-H264WithOpenH264FFmpeg,use-new-media-cache<use-new-media-cache --force-fieldtrial-params=AutofillCreditCardSigninPromo.EnabledThree:impression_limit/3,AutofillProfileOrderByFrecency.EnabledLimitTo3:limit/3,AutomaticTabDiscarding.Enabled_Once_10-gen2:MinimumProtectionTime/600,BackgroundVideoOptimizations.BackgroundOptimizationEnabled1sOrLessMediaSource:max_keyframe_distance_media_source_ms/1000/max_keyframe_distance_ms/0,CSSExternalScanner.Enabled_ScanAndPreload:cssExternalScannerNoPreload/false/cssExternalScannerPreload/true,DataReductionProxyUseQuic.Enabled:enable_warmup/true,DefaultBrowserPromptStyle.ColoredLogoOnWhiteBackground:IconColor/Colored/InfoBarType/PageAction,DelayNavigation.DelayNavigation:delay_millis/4/delay_probability/0%2E5/randomize_delay/true,DesktopIOSPromotion.SavePasswordsBubblePromotionV1:body_text_id/0/entry_point/SavePasswordsBubblePromotion/max_views/3/promo_variation_id/1/sms_id/19001508/title_text_id/0,DisallowFetchForDocWrittenScriptsInMainFrame.DocumentWriteScriptBlockGroup_20161208_Launch:disallowFetchForDocWrittenScriptsInMainFrame/false/disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections/true,Html5ByDefault.Enabled:engagement_threshold_for_flash/32,IncognitoWindowPromo.Enabled:availability/any/event_history_deleted/name%3Ahistory_deleted;comparator%3A>=1;window%3A3650;storage%3A3650/event_incognito_window_opened/name%3Aincognito_window_opened;comparator%3A==0;window%3A3650;storage%3A3650/event_session_time/name%3Asession_time;comparator%3A>=1;window%3A3650;storage%3A3650/event_trigger/name%3Aincognito_window_trigger;comparator%3Aany;window%3A3650;storage%3A3650/event_used/name%3Aincognito_window_clicked;comparator%3Aany;window%3A3650;storage%3A3650/session_rate/<=3,NetworkQualityEstimator.Enabled:2G%2EDefaultMedianKbps/81/2G%2EDefaultMedianRTTMsec/1337/3G%2EDefaultMedianKbps/658/3G%2EDefaultMedianRTTMsec/297/4G%2EDefaultMedianKbps/1756/4G%2EDefaultMedianRTTMsec/159/Bluetooth%2EDefaultMedianKbps/449/Bluetooth%2EDefaultMedianRTTMsec/135/Ethernet%2EDefaultMedianKbps/3263/Ethernet%2EDefaultMedianRTTMsec/106/HalfLifeSeconds/60/None%2EDefaultMedianKbps/567/None%2EDefaultMedianRTTMsec/272/Slow2G%2EThresholdMedianHttpRTTMsec/2000/Unknown%2EDefaultMedianKbps/1916/Unknown%2EDefaultMedianRTTMsec/121/WiFi%2EDefaultMedianKbps/2736/WiFi%2EDefaultMedianRTTMsec/168/correlation_logging_probability/0%2E001,NewTabInProductHelp.Enabled_1:availability/any/event_new_tab_opened/name%3Anew_tab_opened;comparator%3A==0;window%3A3650;storage%3A3650/event_omnibox_used/name%3Aomnibox_used;comparator%3A>=1;window%3A3650;storage%3A3650/event_session_time/name%3Asession_time;comparator%3A>=1;window%3A3650;storage%3A3650/event_trigger/name%3Anew_tab_trigger;comparator%3Aany;window%3A3650;storage%3A3650/event_used/name%3Anew_tab_clicked;comparator%3Aany;window%3A3650;storage%3A3650/session_rate/<=3/x_promo_string/0,NoStatePrefetchValidation.PrerenderDisabled_R2:instant_mode/simple_load/mode/simple_load/omnibox_mode/simple_load,PasswordProtectionPasswordFieldOnFocusPing.SBEROnlyNoIncognito2:extended_reporting/true/history_sync/false/incognito/false,PasswordProtectionProtectedPasswordEntryPing.AllPopulation:all_population/true/incognito/true,PasswordSmartBubble.3-Times:dismissal_count/3,PersistentHistograms.EnabledOnDisk5:send_unreported_metrics/yes/storage/MappedFile,ReportCertificateErrors.ShowAndPossiblySend:sendingThreshold/1%2E0,SafeBrowsingIncidentReportingService.Enabled:collect_reg_data/true,SafeBrowsingThreatDomDetailsTagAttributes.AdIdentifiers:tag_attribute_csv/div%2Cdata-google-query-id%2Cdiv%2Cid%2Ciframe%2Cid,SecurityChip.Enabled:animation/animate-nonsecure-only/visibility/show-all,SettingsResetPrompt.Enabled:delay_before_prompt_seconds/1/domain_hashes/{\"c027a731da03fee479b2ba00b3afe74b3fd9190f0622755db8ffa504efbde66e\"%3A\"3\"}/prompt_wave/20170101/time_between_prompts_seconds/3600,SignInPasswordPromo.Enable3:dismissal_threshold/3,SubresourceFilter.EnabledForPhishingSites:enable_presets/liverun_on_phishing_sites,TLS13Variant.DraftEnabled:variant/draft,TranslateRankerModel.QueryExperiment2:translate-ranker-model-url/https%3A%2F%2Fwww%2Egstatic%2Ecom%2Fchrome%2Fintelligence%2Fassist%2Franker%2Fmodels%2Ftranslate%2F2017%2F03%2Ftranslate_ranker_model_20170329%2Epb%2Ebin,TranslateServerStudy.SmartRendering:server_params/smrd,UKM.Enabled:RecordSessionId/true,WebBluetoothBlocklist.TestGroup:blocklist_additions/00060000%3Ae%2Cfffd%3Ae%2Ced5f25a4%3Ae --enable-net-benchmarking --metrics-recording-only --no-default-browser-check --no-first-run --enable-gpu-benchmarking --disable-background-networking --proxy-server=socks://localhost:49552 --ignore-certificate-errors --disable-component-extensions-with-background-pages --disable-default-apps --disable-search-geolocation-disclosure --enable-logging --v=0 --remote-debugging-port=0 --enable-crash-reporter-for-testing --disable-component-update --window-size=1280,1024 --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 --trace-config-file=e:\b\s\w\it6igb7o\tmpqcmcyp\chrome-trace-config.json about:blank - pid 5312; Handles: 176; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=gpu-process --field-trial-handle=1336,16828144388170565451,2606585767660807311,131072 --enable-logging --trace-config-file="e:\b\s\w\it6igb7o\tmpqcmcyp\chrome-trace-config.json" --v=0 --noerrdialogs --user-data-dir="e:\b\s\w\it6igb7o\tmpmlxuv8" --disable-direct-composition --use-gl=swiftshader-webgl --supports-dual-gpus=false --gpu-driver-bug-workarounds=9,12,13,22,23,24,27,49,84 --disable-gl-extensions="GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent" --disable-accelerated-video-decode --gpu-vendor-id=0x15ad --gpu-device-id=0x0405 --gpu-driver-vendor="Google Inc." --gpu-driver-version=3.3.0.2 --gpu-driver-date=2017/04/07 --noerrdialogs --user-data-dir="e:\b\s\w\it6igb7o\tmpmlxuv8" --enable-logging --v=0 --service-request-channel-token=CC943C7FF8E3B2634D15AD32399BD0BA --mojo-platform-channel-handle=1408 --ignored=" --type=renderer " /prefetch:2 Terminating 5 processes: - 932 killed - 4284 killed - 4296 killed - 4300 killed - 5312 killed *** Swarming tried multiple times to delete the run directory and failed *** *** Hard failing the task *** Swarming detected that your testing script ran an executable, which may have started a child executable, and the main script returned early, leaving the children executables playing around unguided. You don't want to leave children processes outliving the task on the Swarming bot, do you? The Swarming bot doesn't. How to fix? - For any process that starts children processes, make sure all children processes terminated properly before each parent process exits. This is especially important in very deep process trees. - This must be done properly both in normal successful task and in case of task failure. Cleanup is very important. - The Swarming bot sends a SIGTERM in case of timeout. - You have 30.0 seconds to comply after the signal was sent to the process before the process is forcibly killed. - To achieve not leaking children processes in case of signals on timeout, you MUST handle signals in each executable / python script and propagate them to children processes. - When your test script (python or binary) receives a signal like SIGTERM or CTRL_BREAK_EVENT on Windows), send it to all children processes and wait for them to terminate before quitting. See https://github.com/luci/luci-py/blob/master/appengine/swarming/doc/Bot.md#graceful-termination-aka-the-sigterm-and-sigkill-dance for more information. *** May the SIGKILL force be with you ***
,
Jul 27 2017
Yes, possibly related to my latest changes. Having a look ....
,
Jul 27 2017
Disabling CL: https://chromium-review.googlesource.com/c/589349/
,
Jul 27 2017
,
Jul 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9fadd5c4521bfd98984f92d800b75c49952b2dff commit 9fadd5c4521bfd98984f92d800b75c49952b2dff Author: nednguyen <nednguyen@google.com> Date: Thu Jul 27 15:44:57 2017 Remove telemetry_unittests & telemetry_perf_unittests from win CQ bots TBR=jbudorick@chromium.org Bug: chromium:749504 Change-Id: Ic53bdccc7f77e1aa1be05bc0ee85bd200bb97317 Reviewed-on: https://chromium-review.googlesource.com/589349 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#489939} [modify] https://crrev.com/9fadd5c4521bfd98984f92d800b75c49952b2dff/testing/buildbot/chromium.win.json
,
Jul 27 2017
Hey, looking at the logs from the run in #0 and carefully comparing the args on all Chrome invocations, I found what the problem is:
One of the tests (browse:media:flickr_infinite_scroll in particular) tries to launch the browser and _thinks_ that it failed, so it tries a second time when it succeeds. It then dutifully keeps track of and closes that second browser when it's done.
The problem is that the first browser _did_ eventually manage to run and lingers around till the very end when swarming finds it. Other chrome processes left around, I assume, are children of that main browser process (all have --type=... args).
The temp dirs in --user-data-dir give away what each of the browsers is.
Lowering Pri for now as the test is disabled in CQ. Will think about possible solutions tomorrow. (The "Failed with WebSocketTimeoutException while starting the browser backend." should probably try to find and kill the browser before retrying.)
Useful bits from the log that tell this story follow:
[26/27] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:media:flickr_infinite_scroll passed 146.8450s:
[ RUN ] browse:media:flickr_infinite_scroll@{'case': 'browse', 'group': 'media'}
[ RUN ] e:\b\s\w\it6igb7o\tmp4zvixq.html
[ OK ] e:\b\s\w\it6igb7o\tmp4zvixq.html (2682 ms)
[ OK ] browse:media:flickr_infinite_scroll@{'case': 'browse', 'group': 'media'} (118126 ms)
[ PASSED ] 1 test.
[... snip ...]
Issuing command to ts_proxy_server: set mapports 443:49555,*:49554
Requested remote debugging port: 0
Chrome log file will be saved in e:\b\s\w\it6igb7o\tmppwqhgo\chrome.log
Starting Chrome ['e:\\b\\s\\w\\ir\\out\\Release\\chrome.exe', [...], '--user-data-dir=e:\\b\\s\\w\\it6igb7o\\tmpmlxuv8', ...]
Discovered ephemeral port 49558
OS: win win7
Detailed OS version: 6.1.7601
Failed with WebSocketTimeoutException while starting the browser backend.
Browser creation failed (attempt 1 of 3), retrying
Requested remote debugging port: 0
Chrome log file will be saved in e:\b\s\w\it6igb7o\tmpqlhxid\chrome.log
Starting Chrome ['e:\\b\\s\\w\\ir\\out\\Release\\chrome.exe', [...]. '--user-data-dir=e:\\b\\s\\w\\it6igb7o\\tmp4ktu7b', ...]
Discovered ephemeral port 49790
OS: win win7
Detailed OS version: 6.1.7601
Browser command line: "e:\b\s\w\ir\out\Release\chrome.exe" [...] --user-data-dir="e:\b\s\w\it6igb7o\tmp4ktu7b" ...
GPU device 0: VENDOR = 0x15ad, DEVICE = 0x405
[... snip ...]
Uploading e:\b\s\w\it6igb7o\tmpqlhxid\chrome.log to gs://chrome-telemetry-output/log_c6299206-cb91-40d1-ab25-8e225cfa46b1
Uploading browser log to https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/log_c6299206-cb91-40d1-ab25-8e225cfa46b1
Successfully shut down browser cooperatively
[... snip ...]
Enumerating processes:
- pid 932; Handles: 200; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=renderer [...]
- pid 4284; Handles: 91; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler [...]
- pid 4296; Handles: 73; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler [...]
- pid 4300; Handles: 629; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe [...] --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 [...]
- pid 5312; Handles: 176; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=gpu-process [...]
,
Jul 27 2017
Thanks for your investigation, Juan! Reassign the bug to you since that's most of the work here.
,
Jul 27 2017
"One of the tests (browse:media:flickr_infinite_scroll in particular) tries to launch the browser and _thinks_ that it failed, so it tries a second time when it succeeds. It then dutifully keeps track of and closes that second browser when it's done. The problem is that the first browser _did_ eventually manage to run and lingers around till the very end when swarming finds it." --> Upon test failure, the teardown logic should properly clean up everything. If it doesn't, then this is a new regression Telemetry state machine (probably story_runner or shared_page_state).
,
Jul 27 2017
Issue 749297 has been merged into this issue.
,
Jul 27 2017
danajk@, I don't this issue should be merged to Issue 749297 . This one is more about the test controller (Telemetry) failing to clean up the browser process, whereas the other one is about Chrome actually crashing.
,
Jul 27 2017
Yeah, erikchen already unmerged it, thanks.
,
Jul 27 2017
,
Jul 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/383bc1fa89311c568194da1a9ecb66ab5ccb9823 commit 383bc1fa89311c568194da1a9ecb66ab5ccb9823 Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Fri Jul 28 13:53:56 2017 Roll src/third_party/catapult/ a71decf74..507c4af8b (2 commits) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/a71decf74249..507c4af8b4ed $ git log a71decf74..507c4af8b --date=short --no-merges --format='%ad %ae %s' 2017-07-28 perezju [Telemetry] Do close browser upon exceptions while creating 2017-07-27 benjhayden Ungroup unrelated memory alerts. Created with: roll-dep src/third_party/catapult BUG= 749504 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: I3a9ffc2b7585c9f4da20883b9aff4cbbddc50309 Reviewed-on: https://chromium-review.googlesource.com/591387 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#490381} [modify] https://crrev.com/383bc1fa89311c568194da1a9ecb66ab5ccb9823/DEPS
,
Jul 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8385e92c1d5e573d4ec930e566ba1122a5e5e4af commit 8385e92c1d5e573d4ec930e566ba1122a5e5e4af Author: Ned Nguyen <nednguyen@google.com> Date: Fri Jul 28 15:43:35 2017 Revert "Remove telemetry_unittests & telemetry_perf_unittests from win CQ bots" This reverts commit 9fadd5c4521bfd98984f92d800b75c49952b2dff. Reason for revert: Root cause should be fixed. Original change's description: > Remove telemetry_unittests & telemetry_perf_unittests from win CQ bots > > TBR=jbudorick@chromium.org > > Bug: chromium:749504 > Change-Id: Ic53bdccc7f77e1aa1be05bc0ee85bd200bb97317 > Reviewed-on: https://chromium-review.googlesource.com/589349 > Reviewed-by: Ned Nguyen <nednguyen@google.com> > Commit-Queue: Ned Nguyen <nednguyen@google.com> > Cr-Commit-Position: refs/heads/master@{#489939} TBR=nednguyen@google.com Change-Id: I58973ff1d93435ca9975eb0ae15483a32dfb2bf1 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: chromium:749504 Reviewed-on: https://chromium-review.googlesource.com/591947 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#490407} [modify] https://crrev.com/8385e92c1d5e573d4ec930e566ba1122a5e5e4af/testing/buildbot/chromium.win.json
,
Jul 28 2017
nednguyen: What was the "root cause"? Can you post a link?
,
Jul 28 2017
Sorry for not posting the link. The root cause is addressed in https://codereview.chromium.org/2992603003. From Juan: "Exceptions raised during _browser_backend.Start() cause the browser to be closed before continuing (due to code within browser_backend itself). In the bug that triggered this issue, however, the exception was raised during _LogBrowserInfo(); which just falls through the except block, calls "WillCloseBrowser" but doesn't actually do anything to close the browser. I was able to reproduce the issue by injecting a fake exception on _LogBrowserInfo (a zombie browser instance is left open). And the issue is indeed fixed by changing the "WillCloseBrowser" with plain "Close"."
,
Jul 28 2017
Few hours & CQ are functioning well now. Thanks Juan for the quick fix!
,
Jul 29 2017
I did to this a little bit more to try to understand why this issue suddenly pop up (since the bug Juan fixed in #13 has existed forever). My theory is that the bug is due to the action of logging the browser's info (gpu info, browser cmdline flag..) become very flaky at some point recently. However none of the telemetry test are failing because we have a retry logic that tries start the browser 3 times (https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_finder.py?rcl=8b13c6e36285df14197824d078f42854c2e40ad0&l=79) This bug is surfaced because the process leak when browser startup is retries multiple times. Ken: FYI
,
Jan 16
,
Jan 16
|
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by nedngu...@google.com
, Jul 27 2017