New issue
Advanced search Search tips

Issue 860828 link

Starred by 2 users

Issue metadata

Status: Duplicate
Owner: ----
Closed: Jul 24
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

interactive_ui_tests flakily fails on Win10 Tests x64 (dbg) with "timed out, took too much time to complete"

Project Member Reported by pdr@chromium.org, Jul 6

Issue description

interactive_ui_tests consistently fails on Win10 Tests x64 (dbg) with "timed out, took too much time to complete":
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1754

124/200 of the last runs failed for this reason:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29?limit=200
 
Labels: -Infra-Troopers Sheriff-Chromium
Something in the suite is either hanging or failing repeatedly, causing a shard to take an hour despite the others taking ~10 minutes. While this is worth looking into, it's more of a sheriff issue than a trooper one.
Cc: martiniss@chromium.org grt@chromium.org
Summary: interactive_ui_tests flakily fails on Win10 Tests x64 (dbg) with "timed out, took too much time to complete" (was: interactive_ui_tests consistently fails on Win10 Tests x64 (dbg) with "timed out, took too much time to complete")
Here's what I've learned so far. Bot:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29

Example passing runs:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1770
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1768

Example failing runs:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1766
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/1757

In the failing test runs, the tests start timing out right from the beginning:
[ RUN      ] WebViewFocusInteractiveTest.Focus_FocusBeforeNavigation
[5476:1296:0707/001030.452:WARNING:discovery_network_list_win.cc(195)] Failed to open Wlan client handle: 1062
[444:5564:0707/001030.499:WARNING:owning_audio_manager_accessor.cc(80)] Out of process audio service initializing.
[5476:7276:0707/001030.702:WARNING:chrome_browser_main_win.cc(653)] Command line too long for RegisterApplicationRestart:  --brave-new-test-launcher --cfi-diag=0 --gtest_also_run_disabled_tests --gtest_filter=WebViewFocusInteractiveTest.Focus_FocusBeforeNavigation --single_process --snapshot-output-dir="C:\b\s\w\iohcf8yq" --test-launcher-bot-mode --test-launcher-output="C:\Users\CHROME~1\AppData\Local\Temp\scoped_dir4180_30266\results4180_31406\test_results.xml" --test-launcher-summary-output="C:\b\s\w\iohcf8yq\output.json" --user-data-dir="C:\Users\CHROME~1\AppData\Local\Temp\scoped_dir4180_30266\d4180_29104" --disable-offline-auto-reload --no-first-run --no-default-browser-check --enable-logging=stderr --disable-default-apps --wm-window-animations-disabled --disable-component-update --test-type=browser --force-color-profile=srgb --disable-zero-browsers-open-for-tests --ipc-connection-timeout=45 --allow-file-access-from-files --dom-automation --log-gpu-control-list-decisions --disable-backgrounding-occluded-windows --disable-gl-drawing-for-tests --override-use-software-gl-for-tests --force-color-profile=srgb --disable-compositor-ukm-for-tests --enable-features=TestFeatureForBrowserTest1 --disable-features=NetworkPrediction,SpeculativePreconnect,TestFeatureForBrowserTest2 --flag-switches-begin --flag-switches-end --restore-last-session about:blank
[5476:2044:0707/001031.389:ERROR:service_manager_context.cc(264)] Attempting to run unsupported native service: C:\b\s\w\ir\out\Debug_x64\chrome_renderer.service.exe
[5476:2044:0707/001031.405:ERROR:service_manager_context.cc(264)] Attempting to run unsupported native service: C:\b\s\w\ir\out\Debug_x64\chrome_renderer.service.exe
[5476:7276:0707/001031.483:WARNING:gaia_auth_fetcher.cc(902)] Could not reach Google Accounts servers: errno -11
[5476:2044:0707/001031.696:ERROR:service_manager_context.cc(264)] Attempting to run unsupported native service: C:\b\s\w\ir\out\Debug_x64\chrome_renderer.service.exe
[5476:2044:0707/001031.697:ERROR:service_manager_context.cc(264)] Attempting to run unsupported native service: C:\b\s\w\ir\out\Debug_x64\chrome_renderer.service.exe
[5476:2044:0707/001031.943:ERROR:service_manager_context.cc(264)] Attempting to run unsupported native service: C:\b\s\w\ir\out\Debug_x64\chrome_renderer.service.exe
[5476:7592:0707/001031.951:ERROR:process_win.cc(151)] Unable to terminate process: Access is denied. (0x5)
[5476:7276:0707/001032.471:WARNING:gaia_auth_fetcher.cc(902)] Could not reach Google Accounts servers: errno -11
[5476:7276:0707/001034.983:WARNING:gaia_auth_fetcher.cc(902)] Could not reach Google Accounts servers: errno -11
[5476:7276:0707/001043.858:WARNING:gaia_auth_fetcher.cc(902)] Could not reach Google Accounts servers: errno -11
[5476:7276:0707/001107.015:WARNING:gaia_auth_fetcher.cc(902)] Could not reach Google Accounts servers: errno -11
[1/302] WebViewFocusInteractiveTest.Focus_FocusBeforeNavigation (TIMED OUT)

In the passing runs, this does not happen:
Using sharding settings from environment. This is shard 2/3
Using 1 parallel jobs.
[1/302] WebViewFocusInteractiveTest.Focus_FocusBeforeNavigation (4363 ms)

I tried to reproduce this locally on Win10 with a debug build but I do not see timeouts. The stderr output matches the failing run (i.e., "Command line too long for RegisterApplicationRestart", "Unable to terminate process: Access is denied. (0x5)", etc) so I do not think there are any clues in the stderr output, other than the test timeouts out. The same bot (e.g., "swarm770-c4") sometimes fails and sometimes passes, so I don't think this is something on the bot itself. A similar test, network_service_interactive_ui_tests (experimental), has the same issues. I do not see any likely changes in the test itself that would have caused this. The failures go back past the 200 run history of the bot. Because this is marked "experimental", these failures do not block the CQ or turn the tree red, but they do cause the bot to take a full hour to fail.

@grt and @martiniss, this experimental test suite was recently added in https://chromium-review.googlesource.com/c/chromium/src/+/1071568. Should it still be experimental? Do you have any suggestions for how I can debug this further?
Labels: -Sheriff-Chromium
Given that this is currently an experimental test step, the sheriffs should ignore it,  so I'm going to remove the "Sheriff-Chromium" label.
Mergedinto: 840369
Status: Duplicate (was: Available)

Sign in to add a comment