Fuchsia test runner script very flaky |
|||||
Issue descriptionThe Fuchsia test runner script, which calls qemu, is crashing hard (30% of the time?) in a manner that can't easily be caught with Python exceptions. It also renders the Python test runner's stdout and stderr inoperable (adding print debugging statements generates "[Errno 11] Resource temporarily unavailable)" error spew. When the crashes occur, they tend to happen *around* test case #1320 (not necessarily the same entry). The test cases themselves don't appear to be the root cause of the error, since they can be run in isolation repeatedly without any issue. I'm suspecting that this is a POSIX pipe producer/consumer problem in the qemu configuration, or a bug in qemu itself. I have a local patch which routes the qemu terminal output out to a named pipe in the host OS. The issue ceases to repro under that configuration. [00009.171] 02526.02752> [1314/2544] StringPiece16Test.CheckConversion (0 ms) [00009.171] 02526.02752> [1315/2544] SplitStringIntoKeyValuePairsTest.EmptyString (0 ms) [00009.198] 02526.02752> [1316/2544] SplitStringPieceUsingSubstrTest.TrimWhitespace (0 ms) [00009.198] 02526.02752> [1317/2544] SplitStringPieceUsingSubstrTest.SplitWantAll (0 ms) [00009.198] 02526.02752> [1318/2544] SplitStringPieceUsingSubstrTest.SplitWantNonEmpty (0 ms) [00009.198] 02526.02752> [1319/2544] StringSplitTest.StringSplitKeepWhitespace (0 ms) [00009.198] 02526.02752> [1320/2544] StringSplitTest.SplitStringAlongWhitespace (0 ms) close failed in file object destructor:ringTokenizerTest.Simple (0 ms) sys.excepthook is missing lost sys.stderr [00006.272] 02466.02599> [1320/2544] SplitStringIntoKeyValuePairsTest.EmptyValue (0 ms) close failed in file object destructor:litStringIntoKeyValuePairsTest.UntrimmedWhitespace (0 ms) sys.excepthook is missing lost sys.stderr [00005.439] 02389.02537> [1319/2544] SplitStringUsingSubstrTest.ConsecutiveDelimitersSkipped (0 ms) close failed in file object destructor:litStringUsingSubstrTest.TrailingDelimitersSkipped (0 ms) sys.excepthook is missing lost sys.stderr
,
Jul 12 2017
,
Jul 12 2017
OK, it appears that python and qemu can't reliably share the same stdin stream. We found that using a totally different stream for qemu's stdin (e.g. reading from /dev/null) stops the stream teardown and process destruction.
,
Jul 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7913acc222c37941fb90f7e8813221c8cd83075c commit 7913acc222c37941fb90f7e8813221c8cd83075c Author: Kevin Marshall <kmarshall@chromium.org> Date: Thu Jul 13 19:01:47 2017 Stability, usability improvements for Fuchsia test runner. Stability, usability improvements for Fuchsia test runner. * Fix test flakiness that seemed to be related to qemu sharing the stdin stream with Python. The OS would then tear down the scripts prematurely causing Python to panic. Specifying a subprocess PIPE for stdin fixes that. * Disable the QEMU interactive monitor. It was overriding the user's terminal settings and blocking important keypresses like ^C, making it difficult to terminate tests early. R: wez@chromium.org,thakis@chromium.org BUG: 741194 , 740201 Change-Id: Ibda6ac59d44b07c1b75902a7dc13db086841baa0 Reviewed-on: https://chromium-review.googlesource.com/568739 Commit-Queue: Kevin Marshall <kmarshall@chromium.org> Reviewed-by: Nico Weber <thakis@chromium.org> Reviewed-by: Wez <wez@chromium.org> Cr-Commit-Position: refs/heads/master@{#486451} [modify] https://crrev.com/7913acc222c37941fb90f7e8813221c8cd83075c/build/fuchsia/test_runner.py
,
Jul 15 2017
,
Jul 20 2017
,
Jul 25 2017
Are you still seeing any problem? I hadn't seen any of these errors, but it might depend on the host platform. Marking fixed assuming #c4 takes care of things. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by kmarshall@chromium.org
, Jul 11 2017