net_unittests appears to hang frequently on Fuchsia/x64/FYI bot |
||||
Issue descriptione.g. https://ci.chromium.org/buildbot/chromium.fyi/Fuchsia/18065 shows the net_unittests step failing. There is a single shard, which has executed for ~8 minutes, before the connection to the VM is lost: [----------] 4 tests from DnsRecordParserTest [ RUN ] DnsRecordParserTest.Constructor [ OK ] DnsRecordParserTest.Constructor (1 ms) [ RUN ] DnsRecordParserTest.ReadName [ OK ] DnsRecordParserTest.ReadName (0 ms) [ RUN ] DnsRecordParserTest.ReadNameFail [ OK ] DnsRecordParserTest.ReadNameFail (0 ms) [ RUN ] DnsRecordParserTest.ReadRecord [ OK ] DnsRecordParserTest.ReadRecord (1 ms) [----------] 4 tests from Dns2018-05-29 22:37:00,879:WARNING:root:Process exited with status code 255. 2018-05-29 22:37:00,879:INFO:root:Terminating kernel log reader. 2018-05-29 22:37:00,879:INFO:root:Removing package source from device. 2018-05-29 22:38:06,357:INFO:root:Shutting down QEMU. RecordParserTest (167 ms total) [----------] 7 tests from DnsResponseTest [ RUN ] DnsResponseTest.InitParse [ OK ] WebSocketChannelReceiveUtf8Test.InvalidReceivedIncontinuation (185 ms) [ RUN ] WebSocketChannelReceiveUtf8Test.ReceivedBinaryNotUtf8Tested [ OK ] DnsResponseTest.InitParse (0 ms) [ RUN ] DnsResponseTest.InitParseWithoutQuery [ OK ] DnsResponseTest.InitParseWithoutQuery (0 ms) [ RUN ] DnsResponseTest.InitParseWithoutQueryNoQuestions [ OK ] DnsResponseTest.InitParseWithoutQueryNoQuestions (0 ms) [ RUN ] DnsResponseTest.InitParseWithoutQueryTwoQuestions [ OK ] DnsResponseTest.InitParseWithoutQueryTwoQuestions (0 ms) [ RUN ] DnsResponseTest.InitParseWithoutQueryPacketTooShort [ OK ] DnsResponseTest.InitParseWithoutQueryPacketTooShort (0 ms) [ RUN ] DnsResponseTest.ParseToAddressList [ OK ] DnsResponseTest.ParseToAddressList (1 ms) [ RUN ] DnsResponseTest.ParseToAddressListFail [ OK ] DnsResponseTest.ParseToAddressListFail (1 ms) [----------] 7 tests from DnsResponseTest (120 ms total) [----------] 5 tests from DnsSessionTest [ RUN ] DnsSessionTest.AllocateFree [ OK ] DnsSessionTest.AllocateFree (0 ms) [ RUN ] DnsSessionTest.HistogramTimeoutNormal [ OK ] DnsSessionTest.HistogramTimeoutNormal (0 ms) Connection to localhost closed by remote host. This indicates that the SSH process created by RunPackage() returned status 255, but it's not clear how we then continue to log output from the test run.
,
May 30 2018
sergeyu, marshallk and I noted that we may be passing the thread join() timeout at https://cs.chromium.org/chromium/src/build/fuchsia/run_package.py?l=135, which would explain the interleaved log output.
,
May 30 2018
,
May 30 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ccedd1633084ca86d92f5a778562a55779c9f23c commit ccedd1633084ca86d92f5a778562a55779c9f23c Author: Sergey Ulanov <sergeyu@chromium.org> Date: Wed May 30 23:22:49 2018 [Fuchsia] Increase SSH keep-alive interval Previously ServerAliveInterval and ServerAliveCountMax were both set to 1 in SSH config used to connect to Fuchsia from the runner script. This means that SSH diconnects after it fails to deliver 1 keep-alive message within 1 second. This interval is too short, which was causing timeouts and spurious test failures in net_unittests. Increase these values to 2 and 5. BUG= 847915 Change-Id: I88d786626512f08da19444615f13285aac461dcb Reviewed-on: https://chromium-review.googlesource.com/1079325 Reviewed-by: Kevin Marshall <kmarshall@chromium.org> Commit-Queue: Sergey Ulanov <sergeyu@chromium.org> Cr-Commit-Position: refs/heads/master@{#563058} [modify] https://crrev.com/ccedd1633084ca86d92f5a778562a55779c9f23c/build/fuchsia/boot_data.py
,
Jul 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1cf7a5e1f0f148c4dd59b2d4955389819a927f33 commit 1cf7a5e1f0f148c4dd59b2d4955389819a927f33 Author: Nico Weber <thakis@chromium.org> Date: Mon Jul 09 14:22:11 2018 Run more tests on Fuchsia bots. Adds ipc_tests, media_unittests, sql_unittests to the CQ fuchsia_x64 bot, where they all seem to pass. Add comments to the net_unittests suppressions explaining why they're there. Also adds these tests and content_unittests to some Fuchsia FYI bots (which don't have cq bots, so I don't know if they pass -- if not, we can disable them again). There is no comment explaining why these are disabled, and since they pass, let's run them. Bug: 843511, 847915 ,844416 Change-Id: Ife31eeb61a6e505e5e2c0b9a4efeda0af34e4d9f Reviewed-on: https://chromium-review.googlesource.com/1120753 Commit-Queue: Nico Weber <thakis@chromium.org> Reviewed-by: Wez <wez@chromium.org> Cr-Commit-Position: refs/heads/master@{#573299} [modify] https://crrev.com/1cf7a5e1f0f148c4dd59b2d4955389819a927f33/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/1cf7a5e1f0f148c4dd59b2d4955389819a927f33/testing/buildbot/chromium.linux.json [modify] https://crrev.com/1cf7a5e1f0f148c4dd59b2d4955389819a927f33/testing/buildbot/test_suite_exceptions.pyl
,
Jul 11
sergeyu: Is this Fixed now?
,
Jul 11
If it is, we should add net_unittests to the fuchsia bots.
,
Jul 11
Sadly, not yet; we have some other net_unittests flakes - this bug was specifically about the VM seemingly hanging/disconnecting, which I believe is resolve.
,
Dec 7
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9de2c414af3dcffebb07593f33da6d4a0815301a commit 9de2c414af3dcffebb07593f33da6d4a0815301a Author: Fabrice de Gans-Riberi <fdegans@chromium.org> Date: Fri Dec 07 01:01:59 2018 [Fuchsia] Remove the batch limit for net_unittests Bug: 844416, 847915 Change-Id: Id9b4d2a330978e81863557665ba0aa66889cc4bf Reviewed-on: https://chromium-review.googlesource.com/c/1364079 Reviewed-by: Wez <wez@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Fabrice de Gans-Riberi <fdegans@chromium.org> Cr-Commit-Position: refs/heads/master@{#614514} [modify] https://crrev.com/9de2c414af3dcffebb07593f33da6d4a0815301a/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/9de2c414af3dcffebb07593f33da6d4a0815301a/testing/buildbot/test_suites.pyl
,
Dec 20
We still have other issues in net_unittests but this specific one seems to no longer happen.
,
Jan 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a1541ca0168d7443878755b5ae9ec57f3feadb04 commit a1541ca0168d7443878755b5ae9ec57f3feadb04 Author: Fabrice de Gans-Riberi <fdegans@chromium.org> Date: Wed Jan 09 20:05:46 2019 [Fuchsia] Enable net_unittests on the main waterfall. In the past 2 weeks, the suite has only flaked on a single test, which is being investigated. Bug: 847915 , 877248, 884299 Change-Id: Ieb0cf0184b59d397a29c41ab079ecb048b409af2 Reviewed-on: https://chromium-review.googlesource.com/c/1394938 Reviewed-by: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Sergey Ulanov <sergeyu@chromium.org> Reviewed-by: Wez <wez@chromium.org> Commit-Queue: Fabrice de Gans-Riberi <fdegans@chromium.org> Cr-Commit-Position: refs/heads/master@{#621282} [modify] https://crrev.com/a1541ca0168d7443878755b5ae9ec57f3feadb04/testing/buildbot/chromium.linux.json [modify] https://crrev.com/a1541ca0168d7443878755b5ae9ec57f3feadb04/testing/buildbot/test_suite_exceptions.pyl |
||||
►
Sign in to add a comment |
||||
Comment 1 by w...@chromium.org
, May 30 2018Status: Started (was: Untriaged)