New issue
Advanced search Search tips
Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 20
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 1
Type: Bug



Sign in to add a comment
link

Issue 847915: net_unittests appears to hang frequently on Fuchsia/x64/FYI bot

Reported by w...@chromium.org, May 30 2018 Project Member

Issue description

e.g. https://ci.chromium.org/buildbot/chromium.fyi/Fuchsia/18065 shows the net_unittests step failing.  There is a single shard, which has executed for ~8 minutes, before the connection to the VM is lost:

[----------] 4 tests from DnsRecordParserTest
[ RUN      ] DnsRecordParserTest.Constructor
[       OK ] DnsRecordParserTest.Constructor (1 ms)
[ RUN      ] DnsRecordParserTest.ReadName
[       OK ] DnsRecordParserTest.ReadName (0 ms)
[ RUN      ] DnsRecordParserTest.ReadNameFail
[       OK ] DnsRecordParserTest.ReadNameFail (0 ms)
[ RUN      ] DnsRecordParserTest.ReadRecord
[       OK ] DnsRecordParserTest.ReadRecord (1 ms)
[----------] 4 tests from Dns2018-05-29 22:37:00,879:WARNING:root:Process exited with status code 255.
2018-05-29 22:37:00,879:INFO:root:Terminating kernel log reader.
2018-05-29 22:37:00,879:INFO:root:Removing package source from device.
2018-05-29 22:38:06,357:INFO:root:Shutting down QEMU.
RecordParserTest (167 ms total)

[----------] 7 tests from DnsResponseTest
[ RUN      ] DnsResponseTest.InitParse
[       OK ] WebSocketChannelReceiveUtf8Test.InvalidReceivedIncontinuation (185 ms)
[ RUN      ] WebSocketChannelReceiveUtf8Test.ReceivedBinaryNotUtf8Tested
[       OK ] DnsResponseTest.InitParse (0 ms)
[ RUN      ] DnsResponseTest.InitParseWithoutQuery
[       OK ] DnsResponseTest.InitParseWithoutQuery (0 ms)
[ RUN      ] DnsResponseTest.InitParseWithoutQueryNoQuestions
[       OK ] DnsResponseTest.InitParseWithoutQueryNoQuestions (0 ms)
[ RUN      ] DnsResponseTest.InitParseWithoutQueryTwoQuestions
[       OK ] DnsResponseTest.InitParseWithoutQueryTwoQuestions (0 ms)
[ RUN      ] DnsResponseTest.InitParseWithoutQueryPacketTooShort
[       OK ] DnsResponseTest.InitParseWithoutQueryPacketTooShort (0 ms)
[ RUN      ] DnsResponseTest.ParseToAddressList
[       OK ] DnsResponseTest.ParseToAddressList (1 ms)
[ RUN      ] DnsResponseTest.ParseToAddressListFail
[       OK ] DnsResponseTest.ParseToAddressListFail (1 ms)
[----------] 7 tests from DnsResponseTest (120 ms total)

[----------] 5 tests from DnsSessionTest
[ RUN      ] DnsSessionTest.AllocateFree
[       OK ] DnsSessionTest.AllocateFree (0 ms)
[ RUN      ] DnsSessionTest.HistogramTimeoutNormal
[       OK ] DnsSessionTest.HistogramTimeoutNormal (0 ms)
Connection to localhost closed by remote host.

This indicates that the SSH process created by RunPackage() returned status 255, but it's not clear how we then continue to log output from the test run.
 

Comment 1 by w...@chromium.org, May 30 2018

Owner: w...@chromium.org
Status: Started (was: Untriaged)

Comment 2 by w...@chromium.org, May 30 2018

sergeyu, marshallk and I noted that we may be passing the thread join() timeout at https://cs.chromium.org/chromium/src/build/fuchsia/run_package.py?l=135, which would explain the interleaved log output.

Comment 3 by w...@chromium.org, May 30 2018

Owner: sergeyu@chromium.org

Comment 4 by bugdroid1@chromium.org, May 30 2018

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ccedd1633084ca86d92f5a778562a55779c9f23c

commit ccedd1633084ca86d92f5a778562a55779c9f23c
Author: Sergey Ulanov <sergeyu@chromium.org>
Date: Wed May 30 23:22:49 2018

[Fuchsia] Increase SSH keep-alive interval

Previously ServerAliveInterval and ServerAliveCountMax were both set
to 1 in SSH config used to connect to Fuchsia from the runner script.
This means that SSH diconnects after it fails to deliver 1 keep-alive
message within 1 second.
This interval is too short, which was causing timeouts and spurious
test failures in net_unittests. Increase these values to 2 and 5.

BUG= 847915 

Change-Id: I88d786626512f08da19444615f13285aac461dcb
Reviewed-on: https://chromium-review.googlesource.com/1079325
Reviewed-by: Kevin Marshall <kmarshall@chromium.org>
Commit-Queue: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563058}
[modify] https://crrev.com/ccedd1633084ca86d92f5a778562a55779c9f23c/build/fuchsia/boot_data.py

Comment 5 by bugdroid1@chromium.org, Jul 9 2018

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1cf7a5e1f0f148c4dd59b2d4955389819a927f33

commit 1cf7a5e1f0f148c4dd59b2d4955389819a927f33
Author: Nico Weber <thakis@chromium.org>
Date: Mon Jul 09 14:22:11 2018

Run more tests on Fuchsia bots.

Adds ipc_tests, media_unittests, sql_unittests to
the CQ fuchsia_x64 bot, where they all seem to pass.

Add comments to the net_unittests suppressions explaining why they're
there.

Also adds these tests and content_unittests to some Fuchsia FYI
bots (which don't have cq bots, so I don't know if they pass --
if not, we can disable them again).

There is no comment explaining why these are disabled, and since
they pass, let's run them.

Bug: 843511, 847915 ,844416
Change-Id: Ife31eeb61a6e505e5e2c0b9a4efeda0af34e4d9f
Reviewed-on: https://chromium-review.googlesource.com/1120753
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#573299}
[modify] https://crrev.com/1cf7a5e1f0f148c4dd59b2d4955389819a927f33/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/1cf7a5e1f0f148c4dd59b2d4955389819a927f33/testing/buildbot/chromium.linux.json
[modify] https://crrev.com/1cf7a5e1f0f148c4dd59b2d4955389819a927f33/testing/buildbot/test_suite_exceptions.pyl

Comment 6 by w...@chromium.org, Jul 11 2018

sergeyu: Is this Fixed now?

Comment 7 by thakis@chromium.org, Jul 11 2018

If it is, we should add net_unittests to the fuchsia bots.

Comment 8 by w...@chromium.org, Jul 11 2018

Summary: net_unittests appears to hang frequently on Fuchsia/x64/FYI bot (was: net_unittests flakes frequently on Fuchsia/x64/FYI bot)
Sadly, not yet; we have some other net_unittests flakes - this bug was specifically about the VM seemingly hanging/disconnecting, which I believe is resolve.

Comment 9 by bugdroid1@chromium.org, Dec 7

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9de2c414af3dcffebb07593f33da6d4a0815301a

commit 9de2c414af3dcffebb07593f33da6d4a0815301a
Author: Fabrice de Gans-Riberi <fdegans@chromium.org>
Date: Fri Dec 07 01:01:59 2018

[Fuchsia] Remove the batch limit for net_unittests

Bug: 844416,  847915 
Change-Id: Id9b4d2a330978e81863557665ba0aa66889cc4bf
Reviewed-on: https://chromium-review.googlesource.com/c/1364079
Reviewed-by: Wez <wez@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Fabrice de Gans-Riberi <fdegans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#614514}
[modify] https://crrev.com/9de2c414af3dcffebb07593f33da6d4a0815301a/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/9de2c414af3dcffebb07593f33da6d4a0815301a/testing/buildbot/test_suites.pyl

Comment 10 by fdegans@chromium.org, Dec 20

Status: Fixed (was: Started)
We still have other issues in net_unittests but this specific one seems to no longer happen.

Comment 11 by bugdroid1@chromium.org, Jan 9

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a1541ca0168d7443878755b5ae9ec57f3feadb04

commit a1541ca0168d7443878755b5ae9ec57f3feadb04
Author: Fabrice de Gans-Riberi <fdegans@chromium.org>
Date: Wed Jan 09 20:05:46 2019

[Fuchsia] Enable net_unittests on the main waterfall.

In the past 2 weeks, the suite has only flaked on a single test, which
is being investigated.

Bug:  847915 , 877248,  884299 
Change-Id: Ieb0cf0184b59d397a29c41ab079ecb048b409af2
Reviewed-on: https://chromium-review.googlesource.com/c/1394938
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Reviewed-by: Wez <wez@chromium.org>
Commit-Queue: Fabrice de Gans-Riberi <fdegans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#621282}
[modify] https://crrev.com/a1541ca0168d7443878755b5ae9ec57f3feadb04/testing/buildbot/chromium.linux.json
[modify] https://crrev.com/a1541ca0168d7443878755b5ae9ec57f3feadb04/testing/buildbot/test_suite_exceptions.pyl

Sign in to add a comment