New issue
Advanced search Search tips

Issue 741194 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jul 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 2
Type: Bug

Blocked on:
issue 746775

Blocking:
issue 738275



Sign in to add a comment

Fuchsia test runner script very flaky

Project Member Reported by kmarshall@chromium.org, Jul 11 2017

Issue description

The Fuchsia test runner script, which calls qemu, is crashing hard (30% of the time?) in a manner that can't easily be caught with Python exceptions. It also renders the Python test runner's stdout and stderr inoperable (adding print debugging statements generates "[Errno 11] Resource temporarily unavailable)" error spew.

When the crashes occur, they tend to happen *around* test case #1320 (not necessarily the same entry). The test cases themselves don't appear to be the root cause of the error, since they can be run in isolation repeatedly without any issue.

I'm suspecting  that this is a POSIX pipe producer/consumer problem in the qemu configuration, or a bug in qemu itself. I have a local patch which routes the qemu terminal output out to a named pipe in the host OS. The issue ceases to repro under that configuration.


[00009.171] 02526.02752> [1314/2544] StringPiece16Test.CheckConversion (0 ms)
[00009.171] 02526.02752> [1315/2544] SplitStringIntoKeyValuePairsTest.EmptyString (0 ms)
[00009.198] 02526.02752> [1316/2544] SplitStringPieceUsingSubstrTest.TrimWhitespace (0 ms)
[00009.198] 02526.02752> [1317/2544] SplitStringPieceUsingSubstrTest.SplitWantAll (0 ms)
[00009.198] 02526.02752> [1318/2544] SplitStringPieceUsingSubstrTest.SplitWantNonEmpty (0 ms)
[00009.198] 02526.02752> [1319/2544] StringSplitTest.StringSplitKeepWhitespace (0 ms)
[00009.198] 02526.02752> [1320/2544] StringSplitTest.SplitStringAlongWhitespace (0 ms)
close failed in file object destructor:ringTokenizerTest.Simple (0 ms)
sys.excepthook is missing
lost sys.stderr

[00006.272] 02466.02599> [1320/2544] SplitStringIntoKeyValuePairsTest.EmptyValue (0 ms)
close failed in file object destructor:litStringIntoKeyValuePairsTest.UntrimmedWhitespace (0 ms)
sys.excepthook is missing
lost sys.stderr


[00005.439] 02389.02537> [1319/2544] SplitStringUsingSubstrTest.ConsecutiveDelimitersSkipped (0 ms)
close failed in file object destructor:litStringUsingSubstrTest.TrailingDelimitersSkipped (0 ms)
sys.excepthook is missing
lost sys.stderr
 
Another failure, from a totally different test suite. It's pretty clear that the unit tests themselves are probably not the culprit for this issue.

[00021.095] 02520.02734> [1657/2547] ThreadTest.StartWithOptions_NonJoinable (21 ms)
close failed in file object destructor:readTest.TwoTasksOnJoinableThread (21 ms)
sys.excepthook is missing
lost sys.stderr

Comment 2 by w...@chromium.org, Jul 12 2017

Blocking: 738275
OK, it appears that python and qemu can't reliably share the same stdin stream. We found that using a totally different stream for qemu's stdin (e.g. reading from /dev/null) stops the stream teardown and process destruction.
Project Member

Comment 4 by bugdroid1@chromium.org, Jul 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7913acc222c37941fb90f7e8813221c8cd83075c

commit 7913acc222c37941fb90f7e8813221c8cd83075c
Author: Kevin Marshall <kmarshall@chromium.org>
Date: Thu Jul 13 19:01:47 2017

Stability, usability improvements for Fuchsia test runner.

Stability, usability improvements for Fuchsia test runner.
* Fix test flakiness that seemed to be related to qemu sharing the
  stdin stream with Python. The OS would then tear down the scripts
  prematurely causing Python to panic.
  Specifying a subprocess PIPE for stdin fixes that.
* Disable the QEMU interactive monitor. It was overriding the user's
  terminal settings and blocking important keypresses like ^C, making
  it difficult to terminate tests early.

R: wez@chromium.org,thakis@chromium.org
BUG:  741194 , 740201 
Change-Id: Ibda6ac59d44b07c1b75902a7dc13db086841baa0
Reviewed-on: https://chromium-review.googlesource.com/568739
Commit-Queue: Kevin Marshall <kmarshall@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486451}
[modify] https://crrev.com/7913acc222c37941fb90f7e8813221c8cd83075c/build/fuchsia/test_runner.py

Comment 5 by w...@chromium.org, Jul 15 2017

Components: Internals>PlatformIntegration

Comment 6 by w...@chromium.org, Jul 20 2017

Blockedon: 746775
Status: Fixed (was: Assigned)
Are you still seeing any problem? I hadn't seen any of these errors, but it might depend on the host platform. Marking fixed assuming #c4 takes care of things.

Sign in to add a comment