ANGLE try bot win_angle_x64_dbg_ng failing webgl2_conformance_tests because of "Failed to delete the run directory" |
|||
Issue descriptionSeems to have started sometime before Mar 30 15:45. First occurrence: https://build.chromium.org/p/tryserver.chromium.angle/builders/win_angle_x64_dbg_ng/builds/919 Happened pretty consistently since then. Error message: [ RUN ] WebglConformance.conformance2_textures_image_bitmap_from_video_tex_image_and_sub_image_2d_with_image_bitmap_from_video_rgb9_e5_rgb_half_float Running ['c:\\b\\depot_tools\\python276_bin\\python.exe', '../content/test/gpu/run_gpu_test.py', 'webgl_conformance', '--show-stdout', '--browser=debug_x64', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--webgl-conformance-version=2.0.0', '--webgl2-only=true', '--output-dir', 'c:\\users\\chrome~1\\appdata\\local\\temp\\tmp63vxti\\tmpivu_uztelemetry', '--output-format=json'] Failed to delete c:\users\chrome~1\appdata\local\temp\runktfcax (148 files remaining). Maybe the test has a subprocess outliving it. Sleeping 2 seconds. Failed to delete c:\users\chrome~1\appdata\local\temp\runktfcax (148 files remaining). Maybe the test has a subprocess outliving it. Sleeping 4 seconds. Failed to delete c:\users\chrome~1\appdata\local\temp\runktfcax. The following files remain: - \\?\c:\users\chrome~1\appdata\local\temp\runktfcax\chrome <snip> - \\?\c:\users\chrome~1\appdata\local\temp\runrmfgw2 Enumerating processes: <snip> Terminating 5 processes. - 748 killed - 1444 killed - 1736 killed - 2836 killed - 3428 killed Enumerating processes: Failed to delete the run directory, forcibly failing the task because of it. No zombie process can outlive a successful task run and still be marked as successful. Fix your stuff. This is blocking the ANGLE CQ because we run these tests on every CL. Only seems to happen on this config, Debug x64 Windows. It may not be happening on every try job, see https://build.chromium.org/p/tryserver.chromium.angle/builders/win_angle_x64_dbg_ng full log attached. Ken can you help with this? If it hasn't resolved itself we might want to raise to Pri-1.
,
Mar 31 2016
Looking at the logs, I think the problem is that the job is crossing Swarming's default one hour timeout: [ OK ] WebglConformance.conformance2_textures_image_bitmap_from_video_tex_image_and_sub_image_3d_with_image_bitmap_from_video_rgb565_rgb_unsigned_short_5_6_5 (1242 ms) [ RUN ] WebglConformance.conformance2_textures_image_bitmap_from_video_tex_image_and_sub_image_3d_with_image_bitmap_from_video_r11f_g11f_b10f_rgb_unsigned_int_10f_11f_11f_rev [2836:4056:0330/172714:WARNING:webmediaplayer_impl.cc(323)] Using MultibufferDataSource [748:3508:0330/172714:ERROR:audio_manager_win.cc(465)] GetPreferredAudioParameters failed: 88890004 [748:3508:0330/172714:ERROR:audio_manager_win.cc(465)] GetPreferredAudioParameters failed: 88890004 Running ['c:\\b\\depot_tools\\python276_bin\\python.exe', '../content/test/gpu/run_gpu_test.py', 'webgl_conformance', '--show-stdout', '--browser=debug_x64', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--webgl-conformance-version=2.0.0', '--webgl2-only=true', '--output-dir', 'c:\\users\\chrome~1\\appdata\\local\\temp\\tmplk3cir\\tmpk8atqotelemetry', '--output-format=json'] Failed to delete c:\users\chrome~1\appdata\local\temp\runrmfgw2 (148 files remaining). Maybe the test has a subprocess outliving it. Sleeping 2 seconds. Failed to delete c:\users\chrome~1\appdata\local\temp\runrmfgw2 (148 files remaining). Maybe the test has a subprocess outliving it. Sleeping 4 seconds. Failed to delete c:\users\chrome~1\appdata\local\temp\runrmfgw2. The following files remain: - \\?\c:\users\chrome~1\appdata\local\temp\runrmfgw2\chrome - \\?\c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\accessibility.dll - \\?\c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\api-ms-win-core-file-l1-2-0.dll ... - \\?\c:\users\chrome~1\appdata\local\temp\runrmfgw2\third_party - \\?\c:\users\chrome~1\appdata\local\temp\runrmfgw2 Enumerating processes: - pid 748; Handles: 812; Exe: c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe; Cmd: c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe --enable-unsafe-es3-apis --test-type=gpu --disable-domain-blocking-for-3d-apis --disable-gesture-requirement-for-media-playback --disable-gpu-process-crash-limit --enable-experimental-canvas-features --js-flags=--expose-gc --enable-logging=stderr --enable-net-benchmarking --metrics-recording-only --no-default-browser-check --no-first-run --enable-gpu-benchmarking --disable-background-networking "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2194.2 Safari/537.36" --no-proxy-server --disable-component-extensions-with-background-pages --disable-default-apps --remote-debugging-port=0 --enable-crash-reporter-for-testing --window-size=1280,1024 --user-data-dir=c:\users\chrome~1\appdata\local\temp\tmplk3cir\tmp179bjs about:blank - pid 1444; Handles: 109; Exe: c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe; Cmd: c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe --type=crashpad-handler --no-rate-limit "--database=C:\Users\chrome-bot\AppData\Local\Chromium\User Data\Crashpad" --annotation=channel= --annotation=plat=Win64 --annotation=prod=Chromium --annotation=ver=51.0.2695.0-devel --handshake-handle=0x118 - pid 1736; Handles: 122; Exe: c:\users\chrome~1\appdata\local\temp\runrmfgw2\third_party\catapult\telemetry\telemetry\internal\bin\win\AMD64\crash_service.exe; Cmd: c:\users\chrome~1\appdata\local\temp\runrmfgw2\third_party\catapult\telemetry\telemetry\internal\bin\win\AMD64\crash_service.exe --no-window --dumps-dir=c:\users\chrome~1\appdata\local\temp\tmplk3cir\tmpe1m6xi --pipe-name=\\.\pipe\tmpe1m6xi_service - pid 2836; Handles: 291; Exe: c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe; Cmd: "c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe" --type=renderer --disable-gesture-requirement-for-media-playback --enable-experimental-canvas-features --enable-logging=stderr --enable-unsafe-es3-apis --js-flags=--expose-gc --test-type=gpu --enable-gpu-benchmarking --enable-features=AutomaticTabDiscarding<AutomaticTabDiscarding,PreconnectMore<PreconnectMore,WebRTC-H264WithOpenH264FFmpeg<WebRTC-H264WithOpenH264FFmpeg,use-new-media-cache<use-new-media-cache --primordial-pipe-token=00C6E3EFAF70E80659C610F1BE1C14E8 --lang=en-US --force-fieldtrials=AutofillClassifier/Enabled/AutofillFieldMetadata/Enabled/AutofillProfileOrderByFrecency/EnabledLimitTo3/*AutomaticTabDiscarding/Enabled_Once_10-gen2/BrowserBlacklist/Enabled/CaptivePortalInterstitial/Enabled/ChildAccountDetection/Disabled/ChromeDashboard/Enabled/ChromotingQUIC/Enabled/*DataReductionProxyConfigService/Enabled/EnableGoogleCachedCopyTextExperiment/Button/EnableSessionCrashedBubbleUI/Enabled/*ExtensionActionRedesign/Enabled/*ExtensionContentVerification/Enforce/ExtensionInstallVerification/Enforce/GoogleNow/Enable/*IconNTP/Default/InstanceID/Enabled/IntelligentSessionRestore/Enabled/*LocalNTPSuggestionsService/Enabled/MaterialDesignDownloads/Enabled/*NetworkQualityEstimator/Enabled/*NewProfileManagement/Enabled/OfferUploadCreditCards/Enabled/*PageRevisitInstrumentation/Enabled/PasswordBranding/SmartLockBrandingSavePromptOnly/PasswordGeneration/Disabled/PreRead/NoPrefetchArgument/PreconnectMore/PreconnectMoreAllFrames/*QUIC/Enabled/RefreshTokenDeviceId/Enabled/RenderingPipelineThrottling/Enabled/ReportCertificateErrors/ShowAndPossiblySend/*ResourcePriorities/AllExceptAsyncScripts_11011_1_1_10/SRTPromptFieldTrial/On/SafeBrowsingReportPhishingErrorLink/Enabled/SafeBrowsingUnverifiedDownloads/DisableByParameterMostSbTypes2/SafeBrowsingUpdateFrequency/UpdateTime15m/SchedulerExpensiveTaskBlocking/Enabled/SdchPersistence/Enabled/SettingsEnforcement/enforce_always_with_extensions_and_dse/StrictSecureCookies/Enabled/SyncHttpContentCompression/Enabled/TabSyncByRecency/Enabled/*TriggeredResetFieldTrial/On/VarationsServiceControl/Interval_30min/WebFontsIntervention/Enabled/WebRTC-H264WithOpenH264FFmpeg/Enabled/WebRTC-LocalIPPermissionCheck/Enabled/WebRTC-PeerConnectionDTLS1.2/Enabled/use-new-media-cache/Enabled/ --noerrdialogs --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2194.2 Safari/537.36" --user-data-dir="c:\users\chrome~1\appdata\local\temp\tmplk3cir\tmp179bjs" --enable-offline-auto-reload --enable-offline-auto-reload-visible-only --blink-settings=fetchDeferLateScripts=true,fetchIncreaseFontPriority=true,fetchIncreasePriorities=true --enable-net-benchmarking --enable-pinch --device-scale-factor=1 --num-raster-threads=4 --disable-partial-raster --content-image-texture-target=3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553 --video-image-texture-target=3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553,3553 --channel="748.1.1661918517\449404499" --mojo-platform-channel-handle=2116 - pid 3428; Handles: 407; Exe: c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe; Cmd: "c:\users\chrome~1\appdata\local\temp\runrmfgw2\out\Debug_x64\chrome.exe" --type=gpu-process --channel="748.0.1703074855\1260576314" --enable-logging=stderr --enable-unsafe-es3-apis --noerrdialogs --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2194.2 Safari/537.36" --user-data-dir="c:\users\chrome~1\appdata\local\temp\tmplk3cir\tmp179bjs" --disable-direct-composition --supports-dual-gpus=false --gpu-driver-bug-workarounds=3,11,12,24,53,68 --gpu-vendor-id=0x10de --gpu-device-id=0x104a --gpu-driver-vendor=NVIDIA --gpu-driver-version=9.18.13.4788 --noerrdialogs --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2194.2 Safari/537.36" --user-data-dir="c:\users\chrome~1\appdata\local\temp\tmplk3cir\tmp179bjs" --enable-logging=stderr --mojo-platform-channel-handle=1308 --ignored=" --type=renderer " Terminating 5 processes. - 748 killed - 1444 killed - 1736 killed - 2836 killed - 3428 killed Enumerating processes: Failed to delete the run directory, forcibly failing the task because of it. No zombie process can outlive a successful task run and still be marked as successful. Fix your stuff. +---------------------------------------------------------------------------------------+ | End of shard 0 Pending: 18.9s Duration: 3667.3s Bot: build93-m4 Exit: -1073741510 | +---------------------------------------------------------------------------------------+ The duration is key. 3667 seconds is about an hour. The non-debug jobs take about 22 minutes; see similar runs from https://build.chromium.org/p/tryserver.chromium.angle/builders/win_angle_x64_rel_ng?numbuilds=200 . Also, the 32-bit debug versions of these tests take about 45 minutes: https://build.chromium.org/p/tryserver.chromium.angle/builders/win_angle_dbg_ng We can and will eventually need to shard these tests, but that's a larger task. For the moment the expedient thing to do would be to disable them on this particular bot.
,
Mar 31 2016
,
Mar 31 2016
Thanks for figuring this out - we have coverage in x64 in release and 32-bit in Debug so I think it's reasonable for now to disable them.
,
Mar 31 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8f0de46e67ce8338f1fd7861d4770f8fe30d98f2 commit 8f0de46e67ce8338f1fd7861d4770f8fe30d98f2 Author: kbr <kbr@chromium.org> Date: Thu Mar 31 22:54:13 2016 Stop running webgl2_conformance on the Win x64 Debug FYI bot. It's too slow and is triggering the default Swarming timeout. Need to add support for sharding Telemetry tests before turning it back on. BUG= 599451 TBR=zmo@chromium.org CQ_INCLUDE_TRYBOTS=tryserver.chromium.win:win_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel NOTRY=true Review URL: https://codereview.chromium.org/1848103003 Cr-Commit-Position: refs/heads/master@{#384425} [modify] https://crrev.com/8f0de46e67ce8338f1fd7861d4770f8fe30d98f2/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/8f0de46e67ce8338f1fd7861d4770f8fe30d98f2/testing/buildbot/chromium.gpu.fyi.json
,
Apr 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0d4b3081c987155b6e5ae3d2f9058ddf3af7013f commit 0d4b3081c987155b6e5ae3d2f9058ddf3af7013f Author: kbr <kbr@chromium.org> Date: Fri Apr 01 00:51:20 2016 Use wildcards for too-long suppressions. Bypassing the unittest, as expected, didn't help. Instead write these two suppressions using wildcards so that they still apply. BUG= 599451 CQ_INCLUDE_TRYBOTS=tryserver.chromium.win:win_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel Review URL: https://codereview.chromium.org/1846073002 Cr-Commit-Position: refs/heads/master@{#384452} [modify] https://crrev.com/0d4b3081c987155b6e5ae3d2f9058ddf3af7013f/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/0d4b3081c987155b6e5ae3d2f9058ddf3af7013f/content/test/gpu/gpu_tests/webgl_conformance_expectations.py
,
Apr 1 2016
The above CL in #6 should have referred to Issue 599333 .
,
Apr 18 2016
Considering this fixed at this point, since these tests aren't failing on this tryserver any more. |
|||
►
Sign in to add a comment |
|||
Comment 1 by jmad...@chromium.org
, Mar 31 2016