New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 749504 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jul 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

telemetry_perf_unittests extremely flaky on Windows, blocking CQ

Project Member Reported by nedngu...@google.com, Jul 27 2017

Issue description

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng?numbuilds=200

https://chromium-swarm.appspot.com/task?id=379c58e51f945c10&refresh=10&show_raw=1

The error seems to be that we are leaking browser processes at the end of the tests:


Failed to delete e:\b\s\w\ir. The following files remain:
- \\?\e:\b\s\w\ir
- \\?\e:\b\s\w\ir\out
- \\?\e:\b\s\w\ir\out\Release
- \\?\e:\b\s\w\ir\out\Release\chrome.dll
- \\?\e:\b\s\w\ir\out\Release\chrome.exe
- \\?\e:\b\s\w\ir\out\Release\chrome_100_percent.pak
- \\?\e:\b\s\w\ir\out\Release\chrome_child.dll
- \\?\e:\b\s\w\ir\out\Release\chrome_elf.dll
- \\?\e:\b\s\w\ir\out\Release\d3dcompiler_47.dll
- \\?\e:\b\s\w\ir\out\Release\icudtl.dat
- \\?\e:\b\s\w\ir\out\Release\locales
- \\?\e:\b\s\w\ir\out\Release\locales\en-US.pak
- \\?\e:\b\s\w\ir\out\Release\natives_blob.bin
- \\?\e:\b\s\w\ir\out\Release\resources.pak
- \\?\e:\b\s\w\ir\out\Release\snapshot_blob.bin
- \\?\e:\b\s\w\ir\out\Release\swiftshader
- \\?\e:\b\s\w\ir\out\Release\swiftshader\libEGL.dll
- \\?\e:\b\s\w\ir\out\Release\swiftshader\libGLESv2.dll
Enumerating processes:
- pid 932; Handles: 200; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=renderer --enable-logging --trace-config-file="e:\b\s\w\it6igb7o\tmpqcmcyp\chrome-trace-config.json" --v=0 --enable-gpu-benchmarking --field-trial-handle=1336,16828144388170565451,2606585767660807311,131072 --service-pipe-token=042B0DB4CE61F2E39D0D5E3E5940DEC5 --lang=en-US --noerrdialogs --user-data-dir="e:\b\s\w\it6igb7o\tmpmlxuv8" --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=disallowFetchForDocWrittenScriptsInMainFrame=false,disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections=true,cssExternalScannerNoPreload=false,cssExternalScannerPreload=true --enable-net-benchmarking --enable-pinch --device-scale-factor=1 --num-raster-threads=4 --enable-main-frame-before-activation --enable-checker-imaging --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;0,17,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;1,17,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;2,17,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;3,17,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553;4,17,3553 --enable-gpu-async-worker-context --service-request-channel-token=042B0DB4CE61F2E39D0D5E3E5940DEC5 --renderer-client-id=3 --mojo-platform-channel-handle=2188 /prefetch:1
- pid 4284; Handles: 91; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 /prefetch:7 --monitor-self --monitor-self-argument=--type=crashpad-handler --monitor-self-argument=--user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 --monitor-self-argument=/prefetch:7 --monitor-self-annotation=ptype=crashpad-handler --database=e:\b\s\w\it6igb7o\tmpqquqrw --metrics-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 --annotation=plat=Win32 --annotation=prod=Chromium --annotation=ver=62.0.3169.0-devel --initial-client-data=0x9c,0xa0,0xa4,0x98,0xa8,0x745b0a34,0x745b0a44,0x745b0a54
- pid 4296; Handles: 73; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 /prefetch:7 --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=e:\b\s\w\it6igb7o\tmpqquqrw --annotation=plat=Win32 --annotation=prod=Chromium --annotation=ver=62.0.3169.0-devel --initial-client-data=0xc0,0xc4,0xc8,0xbc,0xb8,0x1050a44,0x1050a54,0x1050a64
- pid 4300; Handles: 629; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --disable-features=LazyParseCSS<LazyParseCSS,TranslateRankerEnforcement<TranslateRankerModel --force-fieldtrials=AlternateComponentUrls/AlternateComponentUrls/AutofillClassifier/Enabled/AutofillCreditCardSigninPromo/EnabledThree/AutofillFieldMetadata/Enabled/AutofillProfileCleanup/Enabled/AutofillProfileOrderByFrecency/EnabledLimitTo3/AutomaticTabDiscarding/Enabled_Once_10-gen2/BackgroundVideoOptimizations/BackgroundOptimizationEnabled1sOrLessMediaSource/BrowserBlacklist/Enabled/BrowserScheduler/RedirectWithDefaultInitParams/CSSExternalScanner/Enabled_ScanAndPreload/CaptivePortalCertificateList/Enabled/CaptivePortalInterstitial/Enabled/CheckerImaging/CheckerImaging/ChildAccountDetection/Disabled/ChromeDashboard/Enabled/ChromotingQUIC/Enabled/DataReductionProxyUseQuic/Enabled/DefaultBrowserInfobar/SettingsTextNotNow/DefaultBrowserPromptStyle/ColoredLogoOnWhiteBackground/DefaultEnableGpuRasterization/DefaultEnableGpuRasterization/DelayNavigation/DelayNavigation/DesktopIOSPromotion/SavePasswordsBubblePromotionV1/DirectCompositionOverlays/DirectCompositionOverlays/DisallowFetchForDocWrittenScriptsInMainFrame/DocumentWriteScriptBlockGroup_20161208_Launch/EnableAppContainer/Enabled/EnableGoogleCachedCopyTextExperiment/Button/EnableMediaRemoting/Enabled/EnableMediaRouter/Enabled/EnableMediaRouterWithCastExtension/Enabled/EnableSessionCrashedBubbleUI/Enabled/EnableSyncClientToServerCompression/Enabled/EnableSyncUSSDeviceInfo/Enabled/ExpectCTDynamic/DynamicExpectCTEnabled/ExpectCTReporting/ExpectCTReportingEnabled/ExtensionContentVerification/Enforce/ExtensionInstallVerification/Enforce/FasterLocationReload/Enabled/GoogleBrandedContextMenu/branded/GoogleNow/Enable/GpuScheduler/Enabled/Html5ByDefault/Enabled/HttpFormWarning/Enabled/IdleTimeSpellChecking/Enabled/ImprovedRecoveryComponent/ImprovedRecoveryComponent/IncludeBetaForumMenuItem/Enabled/IncognitoWindowPromo/Enabled/InstanceID/Enabled/IntelligentSessionRestore/Enabled/LazyParseCSS/Control/LoadingWithMojo/Enabled/MaterialDesignDownloads/Enabled/MaterialDesignUserManager/Enabled/MediaFoundationH264Encoding/Enabled/MidiManagerDynamicInstantiation/Enabled/MojoChannel/Enabled/NTPCaptureThumbnail/Enabled/NTPTilesInInstantService/Enabled/NetDelayableH2AndQuicRequests/Enabled/NetworkQualityEstimator/Enabled/NetworkSchedulerYielding/Enabled/NewProfileManagement/Enabled/NewTabInProductHelp/Enabled_1/NoStatePrefetchValidation/PrerenderDisabled_R2/OfferUploadCreditCards/Enabled/OneGoogleBarOnLocalNtp/Enabled/OutOfProcessPac/Enabled/PageRevisitInstrumentation/Enabled/PassiveDocumentEventListeners/Enabled/PassiveEventListenersDueToFling/Enabled/PasswordBranding/SmartLockBrandingSavePromptOnly/PasswordGeneration/Disabled/PasswordMetadataFilling/Enabled/PasswordProtectionPasswordFieldOnFocusPing/SBEROnlyNoIncognito2/PasswordProtectionProtectedPasswordEntryPing/AllPopulation/PasswordSeparatedSigninFlow/Enabled/PasswordSmartBubble/3-Times/PermissionPromptUIViews/BlockPromptsEnabled/PersistentHistograms/EnabledOnDisk5/PreconnectMore/Enabled/PrintScaling/Enabled/QUIC/Enabled/RafAlignedMouseInput/Enabled/RafAlignedTouchInput/Enabled/RefreshTokenDeviceId/Enabled/ReportCertificateErrors/ShowAndPossiblySend/ReportingAPI/ReportingEnabled/SRTPromptFieldTrial/On/SSLCommonNameMismatchHandling/Enabled/SafeBrowsingIncidentReportingService/Enabled/SafeBrowsingIncidentReportingServiceFeatures/WithSuspiciousModuleReporting/SafeBrowsingScoutTransitionStudy/CanShowScoutOptInGroup2/SafeBrowsingThreatDomDetailsTagAttributes/AdIdentifiers/SafeBrowsingV4LocalDatabaseManagerEnabled/V4Only2/SaveAsMenuText/download/SchedulerExpensiveTaskBlocking/Enabled/SdchPersistence/Enabled/SecurityChip/Enabled/ServiceWorkerNavigationPreload/Enabled/SettingsEnforcement/enforce_always_with_extensions_and_dse/SettingsResetPrompt/Enabled/SignInPasswordPromo/Enable3/SocketReadIfReady/Enabled/SpeculativeLaunchServiceWorker/Enabled/StrictSecureCookies/Enabled/SubresourceFilter/EnabledForPhishingSites/SuperfishInterstitial/Enabled/SyncUSSAutocomplete/Enabled/SyncUserEvents/Enabled/TLS13Variant/DraftEnabled/TabSyncByRecency/Enabled/TokenBinding/TokenBinding/TranslateRankerModel/QueryExperiment2/TranslateServerStudy/SmartRendering/TranslateUI2016Q2/TranslateUI2016Q2OnGroup/TriggeredResetFieldTrial/On/UKM/Enabled/V8AsmJSToWasm/AsmJsToWebAssembly/V8CacheStrategiesForCacheStorage/default/V8Ignition/Future/VideoCaptureService/Enabled/VsyncAlignedInput/Enable/WebBluetoothBlocklist/TestGroup/WebFontsInterventionV2/Enabled-slow2g/WebRTC-EnableWebRtcEcdsa/Enabled/WebRTC-H264WithOpenH264FFmpeg/Enabled/WebRTC-LocalIPPermissionCheck/Enabled/use-new-media-cache/Enabled --enable-features=AlternateComponentUrls<AlternateComponentUrls,AutofillCreditCardSigninPromo<AutofillCreditCardSigninPromo,AutofillProfileCleanup<AutofillProfileCleanup,AutomaticTabDiscarding<AutomaticTabDiscarding,BackgroundVideoTrackOptimization<BackgroundVideoOptimizations,CaptivePortalCertificateList<CaptivePortalCertificateList,CheckerImaging<CheckerImaging,DefaultEnableGpuRasterization<DefaultEnableGpuRasterization,DelayNavigation<DelayNavigation,DesktopIOSPromotion<DesktopIOSPromotion,DirectCompositionOverlays<DirectCompositionOverlays,MediaRemoting<EnableMediaRemoting,EnableSyncClientToServerCompression<EnableSyncClientToServerCompression,EnableSyncUSSDeviceInfo<EnableSyncUSSDeviceInfo,DynamicExpectCT<ExpectCTDynamic,ExpectCTReporting<ExpectCTReporting,FasterLocationReload<FasterLocationReload,GpuScheduler<GpuScheduler,PreferHtmlOverPlugins<Html5ByDefault,HttpFormWarning<HttpFormWarning,IdleTimeSpellChecking<IdleTimeSpellChecking,ImprovedRecoveryComponent<ImprovedRecoveryComponent,IncludeBetaForumMenuItem<IncludeBetaForumMenuItem,IPH_IncognitoWindow<IncognitoWindowPromo,LoadingWithMojo<LoadingWithMojo,MaterialDesignUserManager<MaterialDesignUserManager,MediaFoundationH264Encoding<MediaFoundationH264Encoding,MidiManagerDynamicInstantiation<MidiManagerDynamicInstantiation,CaptureThumbnailDependingOnTransitionType<NTPCaptureThumbnail,CaptureThumbnailOnNavigatingAway<NTPCaptureThumbnail,NTPTilesInInstantService<NTPTilesInInstantService,PrioritySupportedRequestsDelayable<NetDelayableH2AndQuicRequests,NetworkSchedulerYielding<NetworkSchedulerYielding,IPH_NewTab<NewTabInProductHelp,NoStatePrefetch<NoStatePrefetchValidation,OneGoogleBarOnLocalNtp<OneGoogleBarOnLocalNtp,PassiveDocumentEventListeners<PassiveDocumentEventListeners,PassiveEventListenersDueToFling<PassiveEventListenersDueToFling,PasswordMetadataFilling<PasswordMetadataFilling,PasswordFieldOnFocusPinging<PasswordProtectionPasswordFieldOnFocusPing,ProtectedPasswordEntryPinging<PasswordProtectionProtectedPasswordEntryPing,UsePasswordSeparatedSigninFlow<PasswordSeparatedSigninFlow,BlockPromptsIfDismissedOften<PermissionPromptUIViews,PersistentHistograms<PersistentHistograms,PreconnectMore<PreconnectMore,PrintScaling<PrintScaling,RafAlignedMouseInput<RafAlignedMouseInput,RafAlignedTouchInput<RafAlignedTouchInput,Reporting<ReportingAPI,IncidentReportingDisableUpload<SafeBrowsingIncidentReportingService,IncidentReportingModuleLoadAnalysis<SafeBrowsingIncidentReportingServiceFeatures,IncidentReportingSuspiciousModuleReporting<SafeBrowsingIncidentReportingServiceFeatures,CanShowScoutOptIn<SafeBrowsingScoutTransitionStudy,ThreatDomDetailsTagAttributes<SafeBrowsingThreatDomDetailsTagAttributes,SafeBrowsingV4LocalDatabaseManagerEnabled<SafeBrowsingV4LocalDatabaseManagerEnabled,SafeBrowsingV4OnlyEnabled<SafeBrowsingV4LocalDatabaseManagerEnabled,SecurityChip<SecurityChip,ServiceWorkerNavigationPreload<ServiceWorkerNavigationPreload,SettingsResetPrompt<SettingsResetPrompt,SocketReadIfReady<SocketReadIfReady,SpeculativeLaunchServiceWorker<SpeculativeLaunchServiceWorker,SubresourceFilter<SubresourceFilter,SuperfishInterstitial<SuperfishInterstitial,SyncUSSAutocomplete<SyncUSSAutocomplete,SyncUserEvents<SyncUserEvents,token-binding<TokenBinding,TranslateRankerQuery<TranslateRankerModel,TranslateUI2016Q2<TranslateUI2016Q2,Ukm<UKM,AsmJsToWebAssembly<V8AsmJSToWasm,V8Future<V8Ignition,MojoVideoCapture<VideoCaptureService,RequestUnbufferedDispatch<VsyncAlignedInput,VsyncAlignedInput<VsyncAlignedInput,WebRTC-EnableWebRtcEcdsa<WebRTC-EnableWebRtcEcdsa,WebRTC-H264WithOpenH264FFmpeg<WebRTC-H264WithOpenH264FFmpeg,use-new-media-cache<use-new-media-cache --force-fieldtrial-params=AutofillCreditCardSigninPromo.EnabledThree:impression_limit/3,AutofillProfileOrderByFrecency.EnabledLimitTo3:limit/3,AutomaticTabDiscarding.Enabled_Once_10-gen2:MinimumProtectionTime/600,BackgroundVideoOptimizations.BackgroundOptimizationEnabled1sOrLessMediaSource:max_keyframe_distance_media_source_ms/1000/max_keyframe_distance_ms/0,CSSExternalScanner.Enabled_ScanAndPreload:cssExternalScannerNoPreload/false/cssExternalScannerPreload/true,DataReductionProxyUseQuic.Enabled:enable_warmup/true,DefaultBrowserPromptStyle.ColoredLogoOnWhiteBackground:IconColor/Colored/InfoBarType/PageAction,DelayNavigation.DelayNavigation:delay_millis/4/delay_probability/0%2E5/randomize_delay/true,DesktopIOSPromotion.SavePasswordsBubblePromotionV1:body_text_id/0/entry_point/SavePasswordsBubblePromotion/max_views/3/promo_variation_id/1/sms_id/19001508/title_text_id/0,DisallowFetchForDocWrittenScriptsInMainFrame.DocumentWriteScriptBlockGroup_20161208_Launch:disallowFetchForDocWrittenScriptsInMainFrame/false/disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections/true,Html5ByDefault.Enabled:engagement_threshold_for_flash/32,IncognitoWindowPromo.Enabled:availability/any/event_history_deleted/name%3Ahistory_deleted;comparator%3A>=1;window%3A3650;storage%3A3650/event_incognito_window_opened/name%3Aincognito_window_opened;comparator%3A==0;window%3A3650;storage%3A3650/event_session_time/name%3Asession_time;comparator%3A>=1;window%3A3650;storage%3A3650/event_trigger/name%3Aincognito_window_trigger;comparator%3Aany;window%3A3650;storage%3A3650/event_used/name%3Aincognito_window_clicked;comparator%3Aany;window%3A3650;storage%3A3650/session_rate/<=3,NetworkQualityEstimator.Enabled:2G%2EDefaultMedianKbps/81/2G%2EDefaultMedianRTTMsec/1337/3G%2EDefaultMedianKbps/658/3G%2EDefaultMedianRTTMsec/297/4G%2EDefaultMedianKbps/1756/4G%2EDefaultMedianRTTMsec/159/Bluetooth%2EDefaultMedianKbps/449/Bluetooth%2EDefaultMedianRTTMsec/135/Ethernet%2EDefaultMedianKbps/3263/Ethernet%2EDefaultMedianRTTMsec/106/HalfLifeSeconds/60/None%2EDefaultMedianKbps/567/None%2EDefaultMedianRTTMsec/272/Slow2G%2EThresholdMedianHttpRTTMsec/2000/Unknown%2EDefaultMedianKbps/1916/Unknown%2EDefaultMedianRTTMsec/121/WiFi%2EDefaultMedianKbps/2736/WiFi%2EDefaultMedianRTTMsec/168/correlation_logging_probability/0%2E001,NewTabInProductHelp.Enabled_1:availability/any/event_new_tab_opened/name%3Anew_tab_opened;comparator%3A==0;window%3A3650;storage%3A3650/event_omnibox_used/name%3Aomnibox_used;comparator%3A>=1;window%3A3650;storage%3A3650/event_session_time/name%3Asession_time;comparator%3A>=1;window%3A3650;storage%3A3650/event_trigger/name%3Anew_tab_trigger;comparator%3Aany;window%3A3650;storage%3A3650/event_used/name%3Anew_tab_clicked;comparator%3Aany;window%3A3650;storage%3A3650/session_rate/<=3/x_promo_string/0,NoStatePrefetchValidation.PrerenderDisabled_R2:instant_mode/simple_load/mode/simple_load/omnibox_mode/simple_load,PasswordProtectionPasswordFieldOnFocusPing.SBEROnlyNoIncognito2:extended_reporting/true/history_sync/false/incognito/false,PasswordProtectionProtectedPasswordEntryPing.AllPopulation:all_population/true/incognito/true,PasswordSmartBubble.3-Times:dismissal_count/3,PersistentHistograms.EnabledOnDisk5:send_unreported_metrics/yes/storage/MappedFile,ReportCertificateErrors.ShowAndPossiblySend:sendingThreshold/1%2E0,SafeBrowsingIncidentReportingService.Enabled:collect_reg_data/true,SafeBrowsingThreatDomDetailsTagAttributes.AdIdentifiers:tag_attribute_csv/div%2Cdata-google-query-id%2Cdiv%2Cid%2Ciframe%2Cid,SecurityChip.Enabled:animation/animate-nonsecure-only/visibility/show-all,SettingsResetPrompt.Enabled:delay_before_prompt_seconds/1/domain_hashes/{\"c027a731da03fee479b2ba00b3afe74b3fd9190f0622755db8ffa504efbde66e\"%3A\"3\"}/prompt_wave/20170101/time_between_prompts_seconds/3600,SignInPasswordPromo.Enable3:dismissal_threshold/3,SubresourceFilter.EnabledForPhishingSites:enable_presets/liverun_on_phishing_sites,TLS13Variant.DraftEnabled:variant/draft,TranslateRankerModel.QueryExperiment2:translate-ranker-model-url/https%3A%2F%2Fwww%2Egstatic%2Ecom%2Fchrome%2Fintelligence%2Fassist%2Franker%2Fmodels%2Ftranslate%2F2017%2F03%2Ftranslate_ranker_model_20170329%2Epb%2Ebin,TranslateServerStudy.SmartRendering:server_params/smrd,UKM.Enabled:RecordSessionId/true,WebBluetoothBlocklist.TestGroup:blocklist_additions/00060000%3Ae%2Cfffd%3Ae%2Ced5f25a4%3Ae --enable-net-benchmarking --metrics-recording-only --no-default-browser-check --no-first-run --enable-gpu-benchmarking --disable-background-networking --proxy-server=socks://localhost:49552 --ignore-certificate-errors --disable-component-extensions-with-background-pages --disable-default-apps --disable-search-geolocation-disclosure --enable-logging --v=0 --remote-debugging-port=0 --enable-crash-reporter-for-testing --disable-component-update --window-size=1280,1024 --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 --trace-config-file=e:\b\s\w\it6igb7o\tmpqcmcyp\chrome-trace-config.json about:blank
- pid 5312; Handles: 176; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=gpu-process --field-trial-handle=1336,16828144388170565451,2606585767660807311,131072 --enable-logging --trace-config-file="e:\b\s\w\it6igb7o\tmpqcmcyp\chrome-trace-config.json" --v=0 --noerrdialogs --user-data-dir="e:\b\s\w\it6igb7o\tmpmlxuv8" --disable-direct-composition --use-gl=swiftshader-webgl --supports-dual-gpus=false --gpu-driver-bug-workarounds=9,12,13,22,23,24,27,49,84 --disable-gl-extensions="GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent" --disable-accelerated-video-decode --gpu-vendor-id=0x15ad --gpu-device-id=0x0405 --gpu-driver-vendor="Google Inc." --gpu-driver-version=3.3.0.2 --gpu-driver-date=2017/04/07 --noerrdialogs --user-data-dir="e:\b\s\w\it6igb7o\tmpmlxuv8" --enable-logging --v=0 --service-request-channel-token=CC943C7FF8E3B2634D15AD32399BD0BA --mojo-platform-channel-handle=1408 --ignored=" --type=renderer " /prefetch:2
Terminating 5 processes:
- 932 killed
- 4284 killed
- 4296 killed
- 4300 killed
- 5312 killed
*** Swarming tried multiple times to delete the run directory and failed ***
*** Hard failing the task ***

Swarming detected that your testing script ran an executable, which may have
started a child executable, and the main script returned early, leaving the
children executables playing around unguided.

You don't want to leave children processes outliving the task on the Swarming
bot, do you? The Swarming bot doesn't.

How to fix?
- For any process that starts children processes, make sure all children
  processes terminated properly before each parent process exits. This is
  especially important in very deep process trees.
  - This must be done properly both in normal successful task and in case of
    task failure. Cleanup is very important.
- The Swarming bot sends a SIGTERM in case of timeout.
  - You have 30.0 seconds to comply after the signal was sent to the process
    before the process is forcibly killed.
- To achieve not leaking children processes in case of signals on timeout, you
  MUST handle signals in each executable / python script and propagate them to
  children processes.
  - When your test script (python or binary) receives a signal like SIGTERM or
    CTRL_BREAK_EVENT on Windows), send it to all children processes and wait for
    them to terminate before quitting.

See
https://github.com/luci/luci-py/blob/master/appengine/swarming/doc/Bot.md#graceful-termination-aka-the-sigterm-and-sigkill-dance
for more information.

*** May the SIGKILL force be with you ***
 
I think I need to remove telemetry_perf_uittest from Windows CQ first to unblock people :-(
Yes, possibly related to my latest changes. Having a look ....
Project Member

Comment 4 by sheriffbot@chromium.org, Jul 27 2017

Labels: Hotlist-Google
Project Member

Comment 5 by bugdroid1@chromium.org, Jul 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9fadd5c4521bfd98984f92d800b75c49952b2dff

commit 9fadd5c4521bfd98984f92d800b75c49952b2dff
Author: nednguyen <nednguyen@google.com>
Date: Thu Jul 27 15:44:57 2017

Remove telemetry_unittests & telemetry_perf_unittests from win CQ bots

TBR=jbudorick@chromium.org

Bug:  chromium:749504 
Change-Id: Ic53bdccc7f77e1aa1be05bc0ee85bd200bb97317
Reviewed-on: https://chromium-review.googlesource.com/589349
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#489939}
[modify] https://crrev.com/9fadd5c4521bfd98984f92d800b75c49952b2dff/testing/buildbot/chromium.win.json

Labels: -Pri-0 Pri-1
Hey, looking at the logs from the run in #0 and carefully comparing the args on all Chrome invocations, I found what the problem is:

One of the tests (browse:media:flickr_infinite_scroll in particular) tries to launch the browser and _thinks_ that it failed, so it tries a second time when it succeeds. It then dutifully keeps track of and closes that second browser when it's done.

The problem is that the first browser _did_ eventually manage to run and lingers around till the very end when swarming finds it. Other chrome processes left around, I assume, are children of that main browser process (all have --type=... args).

The temp dirs in --user-data-dir give away what each of the browsers is.

Lowering Pri for now as the test is disabled in CQ. Will think about possible solutions tomorrow. (The "Failed with WebSocketTimeoutException while starting the browser backend." should probably try to find and kill the browser before retrying.)

Useful bits from the log that tell this story follow:

[26/27] benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.browse:media:flickr_infinite_scroll passed 146.8450s:
  [ RUN      ] browse:media:flickr_infinite_scroll@{'case': 'browse', 'group': 'media'}
  [ RUN      ] e:\b\s\w\it6igb7o\tmp4zvixq.html
  [       OK ] e:\b\s\w\it6igb7o\tmp4zvixq.html (2682 ms)
  [       OK ] browse:media:flickr_infinite_scroll@{'case': 'browse', 'group': 'media'} (118126 ms)
  [  PASSED  ] 1 test.

[... snip ...]

  Issuing command to ts_proxy_server: set mapports 443:49555,*:49554
  Requested remote debugging port: 0
  Chrome log file will be saved in e:\b\s\w\it6igb7o\tmppwqhgo\chrome.log
  Starting Chrome ['e:\\b\\s\\w\\ir\\out\\Release\\chrome.exe', [...], '--user-data-dir=e:\\b\\s\\w\\it6igb7o\\tmpmlxuv8', ...]
  Discovered ephemeral port 49558
  OS: win win7
  Detailed OS version: 6.1.7601
  Failed with WebSocketTimeoutException while starting the browser backend.
  Browser creation failed (attempt 1 of 3), retrying
  Requested remote debugging port: 0
  Chrome log file will be saved in e:\b\s\w\it6igb7o\tmpqlhxid\chrome.log
  Starting Chrome ['e:\\b\\s\\w\\ir\\out\\Release\\chrome.exe', [...]. '--user-data-dir=e:\\b\\s\\w\\it6igb7o\\tmp4ktu7b', ...]
  Discovered ephemeral port 49790
  OS: win win7
  Detailed OS version: 6.1.7601
  Browser command line: "e:\b\s\w\ir\out\Release\chrome.exe" [...] --user-data-dir="e:\b\s\w\it6igb7o\tmp4ktu7b" ...
  GPU device 0: VENDOR = 0x15ad, DEVICE = 0x405

[... snip ...]

  Uploading e:\b\s\w\it6igb7o\tmpqlhxid\chrome.log to gs://chrome-telemetry-output/log_c6299206-cb91-40d1-ab25-8e225cfa46b1
  Uploading browser log to https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/log_c6299206-cb91-40d1-ab25-8e225cfa46b1
  Successfully shut down browser cooperatively

[... snip ...]

  Enumerating processes:
  - pid 932; Handles: 200; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=renderer [...]
  - pid 4284; Handles: 91; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler [...]
  - pid 4296; Handles: 73; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe --type=crashpad-handler [...]
  - pid 4300; Handles: 629; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: e:\b\s\w\ir\out\Release\chrome.exe [...] --user-data-dir=e:\b\s\w\it6igb7o\tmpmlxuv8 [...]
  - pid 5312; Handles: 176; Exe: e:\b\s\w\ir\out\Release\chrome.exe; Cmd: "e:\b\s\w\ir\out\Release\chrome.exe" --type=gpu-process [...]

Cc: nedngu...@google.com
Owner: perezju@chromium.org
Thanks for your investigation, Juan! Reassign the bug to you since that's most of the work here.
"One of the tests (browse:media:flickr_infinite_scroll in particular) tries to launch the browser and _thinks_ that it failed, so it tries a second time when it succeeds. It then dutifully keeps track of and closes that second browser when it's done.

The problem is that the first browser _did_ eventually manage to run and lingers around till the very end when swarming finds it."

--> Upon test failure, the teardown logic should properly clean up everything. If it doesn't, then this is a new regression Telemetry state machine (probably story_runner or shared_page_state).

Comment 9 by danakj@chromium.org, Jul 27 2017

Cc: thakis@chromium.org danakj@chromium.org mark@chromium.org
 Issue 749297  has been merged into this issue.
danajk@, I don't this issue should be merged to  Issue 749297 . This one is more about the test controller (Telemetry) failing to clean up the browser process, whereas the other one is about Chrome actually crashing.
Yeah, erikchen already unmerged it, thanks.
Cc: -danakj@chromium.org -thakis@chromium.org -mark@chromium.org
Project Member

Comment 13 by bugdroid1@chromium.org, Jul 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/383bc1fa89311c568194da1a9ecb66ab5ccb9823

commit 383bc1fa89311c568194da1a9ecb66ab5ccb9823
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Fri Jul 28 13:53:56 2017

Roll src/third_party/catapult/ a71decf74..507c4af8b (2 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/a71decf74249..507c4af8b4ed

$ git log a71decf74..507c4af8b --date=short --no-merges --format='%ad %ae %s'
2017-07-28 perezju [Telemetry] Do close browser upon exceptions while creating
2017-07-27 benjhayden Ungroup unrelated memory alerts.

Created with:
  roll-dep src/third_party/catapult
BUG= 749504 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I3a9ffc2b7585c9f4da20883b9aff4cbbddc50309
Reviewed-on: https://chromium-review.googlesource.com/591387
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#490381}
[modify] https://crrev.com/383bc1fa89311c568194da1a9ecb66ab5ccb9823/DEPS

Project Member

Comment 14 by bugdroid1@chromium.org, Jul 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8385e92c1d5e573d4ec930e566ba1122a5e5e4af

commit 8385e92c1d5e573d4ec930e566ba1122a5e5e4af
Author: Ned Nguyen <nednguyen@google.com>
Date: Fri Jul 28 15:43:35 2017

Revert "Remove telemetry_unittests & telemetry_perf_unittests from win CQ bots"

This reverts commit 9fadd5c4521bfd98984f92d800b75c49952b2dff.

Reason for revert: Root cause should be fixed.

Original change's description:
> Remove telemetry_unittests & telemetry_perf_unittests from win CQ bots
> 
> TBR=jbudorick@chromium.org
> 
> Bug:  chromium:749504 
> Change-Id: Ic53bdccc7f77e1aa1be05bc0ee85bd200bb97317
> Reviewed-on: https://chromium-review.googlesource.com/589349
> Reviewed-by: Ned Nguyen <nednguyen@google.com>
> Commit-Queue: Ned Nguyen <nednguyen@google.com>
> Cr-Commit-Position: refs/heads/master@{#489939}

TBR=nednguyen@google.com

Change-Id: I58973ff1d93435ca9975eb0ae15483a32dfb2bf1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  chromium:749504 
Reviewed-on: https://chromium-review.googlesource.com/591947
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#490407}
[modify] https://crrev.com/8385e92c1d5e573d4ec930e566ba1122a5e5e4af/testing/buildbot/chromium.win.json

nednguyen: What was the "root cause"? Can you post a link?
Sorry for not posting the link. The root cause is addressed in https://codereview.chromium.org/2992603003.

From Juan:
"Exceptions raised during _browser_backend.Start() cause the browser to be closed
before continuing (due to code within browser_backend itself).

In the bug that triggered this issue, however, the exception was raised during
_LogBrowserInfo(); which just falls through the except block, calls
"WillCloseBrowser" but doesn't actually do anything to close the browser.

I was able to reproduce the issue by injecting a fake exception on
_LogBrowserInfo (a zombie browser instance is left open). And the issue is
indeed fixed by changing the "WillCloseBrowser" with plain "Close"."
Status: Fixed (was: Assigned)
Few hours & CQ are functioning well now. Thanks Juan for the quick fix!
I did to this a little bit more to try to understand why this issue suddenly pop up (since the bug Juan fixed in #13 has existed forever). 

My theory is that the bug is due to the action of logging the browser's info (gpu info, browser cmdline flag..) become very flaky at some point recently. 

However none of the telemetry test are failing because we have a retry logic that tries start the browser 3 times (https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_finder.py?rcl=8b13c6e36285df14197824d078f42854c2e40ad0&l=79)

This bug is surfaced because the process leak when browser startup is retries multiple times.

Ken: FYI
Components: Test>Telemetry
Components: -Speed>Telemetry

Sign in to add a comment