New issue
Advanced search Search tips

Issue 768409 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner: ----
Closed: Dec 4
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 3
Type: Bug



Sign in to add a comment

Android swarming trybots: shard #XX timed out, took too much time to complete

Project Member Reported by agrieve@chromium.org, Sep 25 2017

Issue description

I've been seeing this somewhat frequently lately.

Here's one from today:
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/391679

In this case, the logs show that this test timed out on each retry:
org.chromium.chrome.browser.omnibox.OmniboxUrlEmphasizerTest.testShortSecureHTTPSUrl
Log link: https://chromium-swarm.appspot.com/task?id=38d276869926f010&refresh=10&show_raw=1&wide_logs=true

However, I've seen this same error across different test suites. E.g., here's two where unit_tests failed:
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/266128
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/265908

And one where webview_instrumentation_test_apk failed:
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/266081


 
Two of the reported timeouts are due to network issues (you can tell from all the "TRANSIENT error publishing messages" errors at the bottom of the task log) and is being worked on in bug 761077.

Another one ran on a device that dropped offline in the middle of the test and hasn't been heard from since:
https://chromium-swarm.appspot.com/bot?id=build264-m4--device6
I'm working on having those types of infra-failures auto-retried, but that's a ways out.

Re https://chromium-swarm.appspot.com/task?id=38d276869926f010: The logcats for testShortSecureHTTPSUrl shows something going wrong in the system; not sure what tho:
https://luci-logdog.appspot.com/app/#!/stream/?s=chromium/android/swarming/logcats/38d276869926f011/+/logcat_org.chromium.chrome.browser.omnibox.OmniboxUrlEmphasizerTest.testShortSecureHTTPSUrl_20170925T153704-UTC_01ab471f0c375df4
(We're working on making that link to logdog more visible for each test.)
Thanks for filing this.

The test runner should be resilient to the TRANSIENT issues as of late last week: https://bugs.chromium.org/p/chromium/issues/detail?id=721889

That last testShortSecureHTTPSUrl failure is a DCHECK failure:

09-25 15:37:05.309 26078 26102 F chromium: [FATAL:command_line.cc(242)] Check failed: current_process_commandline_
Suspecting the DCHECK failure has to do with a test regression due to the J4 conversion in which the command-line no longer gets initialized consistently.
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/438c6ab463a2c46857ecddd955a8f5d0eae58d3c

commit 438c6ab463a2c46857ecddd955a8f5d0eae58d3c
Author: John Budorick <jbudorick@chromium.org>
Date: Tue Sep 26 22:55:19 2017

[android] Print LogcatdogMonitor stream URLs at stream opening.

Should improve logdog stream visibility in the event of a timeout.

Bug:  768409 
Change-Id: I94497c1b3fed557983c0d36112319fe2cb1bf282
Reviewed-on: https://chromium-review.googlesource.com/684055
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#504505}
[modify] https://crrev.com/438c6ab463a2c46857ecddd955a8f5d0eae58d3c/build/android/pylib/android/logdog_logcat_monitor.py

The failing DCHECK is  issue 771205 , where I've added some more detail.
Components: Infra>Client>Chrome
Moving Infra>Client>Android -> Infra>Client>Chrome+OS=Android
Components: -Infra>Client>Android
Project Member

Comment 8 by sheriffbot@chromium.org, Dec 4

Labels: Hotlist-Recharge-Cold
Status: Untriaged (was: Available)
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Status: Fixed (was: Untriaged)
Likely fixed by work in issue 721889 and  issue 771205 .

Sign in to add a comment