net_unittests hanging on Fuchsia |
||||
Issue descriptionFrom https://build.chromium.org/p/chromium.fyi/builders/Fuchsia?numbuilds=200 looks like it might have started at https://build.chromium.org/p/chromium.fyi/builders/Fuchsia/builds/10256 which suggests https://chromium-review.googlesource.com/c/chromium/src/+/719100 as a possible cause. Checking a local repro now...
,
Oct 18 2017
Running locally, netstack got OOM killed. Perhaps a leak? ... [00056.057] 02485.02619> [3812/18641] ThroughputAnalyzerTest.TestMinRequestsForThroughputSample (10 ms) [00056.058] 02485.02619> [3813/18641] ThroughputAnalyzerTest.TestHangingRequests (8221 ms) [00056.060] 02485.02619> [3814/18641] ThroughputAnalyzerTest.TestHangingRequestsCheckedOnlyPeriodically (2 ms) [00056.060] 02485.02619> [3815/18641] ThroughputAnalyzerTest.TestLastReceivedTimeIsUpdated (2 ms) [00056.060] 02485.02619> [3816/18641] ThroughputAnalyzerTest.TestRequestDeletedImmediately (1 ms) [00056.065] 02485.02619> [3817/18641] ThroughputAnalyzerTest.TestThroughputWithMultipleRequestsOverlap (15 ms) [00056.067] 02485.02619> [3818/18641] ThroughputAnalyzerTest.TestThroughputWithNetworkRequestsOverlap (14 ms) [00056.068] 02485.02619> [3819/18641] ThroughputAnalyzerTest.TestThroughputWithMultipleNetworkRequests (2 ms) [00056.070] 02485.02619> [3820/18641] NtlmBufferReaderTest.Initialization (0 ms) [00056.351] 161688.161719> PageFault:577: ERROR: failed to fault in or grab existing page [00056.352] 161826.161867> PageFault:577: ERROR: failed to fault in or grab existing page [00056.352] 161688.161719> PageFault:578: 0xffffff80734142e0 vmo_offset 0xf822347000, pf_flags 0x33 [00056.352] 161826.161867> PageFault:578: 0xffffff80734020c0 vmo_offset 0xc5e2a64000, pf_flags 0x33 [00056.356] 01210.01279> <== fatal exception: process net_unittests[161826] thread initial-thread[161867] [00056.356] 01210.01279> <== fatal page fault, PC at 0x4d8f9377e3a3 [00056.356] 01210.01279> CS: 0 RIP: 0x4d8f9377e3a3 EFL: 0x10202 CR2: 0x8bd376ca000 [00056.356] 01210.01279> RAX: 0x8bd376ca000 RBX: 0x1000 RCX: 0x1000 RDX: 0x1000 [00056.356] 01210.01279> RSI: 0x87853212440 RDI: 0x8bd376ca000 RBP: 0x1 RSP: 0x150c22058c18 [00056.356] 01210.01279> R8: 0xffffffffffffffff R9: 0x808644050a0 R10: 0x1000 R11: 0x80864405038 [00056.356] 01210.01279> R12: 0x1000 R13: 0 R14: 0x8bd376ca000 R15: 0 [00056.356] 01210.01279> errc: 0x6 [00056.356] 01210.01279> bottom of user stack: [00056.356] 01210.01279> 0x0000150c22058c18: 9b649f35 00002c59 9c4a5008 000008bd |5.d.Y,...PJ.....| [00056.356] 01210.01279> 0x0000150c22058c28: 00014000 00000000 00000003 00000000 |.@..............| [00056.356] 01210.01279> 0x0000150c22058c38: 0000001c 00000001 22059048 0000150c |........H.."....| [00056.356] 01210.01279> 0x0000150c22058c48: 00000000 00004d8f 22058e60 0000150c |.....M..`.."....| [00056.356] 01210.01279> 0x0000150c22058c58: 93760dce 00004d8f 51515151 51515151 |..v..M..QQQQQQQQ| [00056.356] 01210.01279> 0x0000150c22058c68: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00056.356] 01210.01279> 0x0000150c22058c78: 531cd120 00000878 00db1ee7 00000000 | ..Sx...........| [00056.356] 01210.01279> 0x0000150c22058c88: 531dd6a0 00000878 ffffffff 7fffffff |...Sx...........| [00056.356] 01210.01279> 0x0000150c22058c98: 531d7d58 00000878 531d8310 00000878 |X}.Sx......Sx...| [00056.356] 01210.01279> 0x0000150c22058ca8: 9a98f1d2 00002c59 44c177ec ab39b4da |....Y,...w.D..9.| [00056.356] 01210.01279> 0x0000150c22058cb8: 531d8950 00000878 44c177ec ab39b4da |P..Sx....w.D..9.| [00056.356] 01210.01279> 0x0000150c22058cc8: 531d8310 00000878 531d7f58 00000878 |...Sx...X..Sx...| [00056.356] 01210.01279> 0x0000150c22058cd8: 00648189 00000000 00000003 00000000 |..d.............| [00056.356] 01210.01279> 0x0000150c22058ce8: 9a989d33 00002c59 51515101 51515151 |3...Y,...QQQQQQQ| [00056.356] 01210.01279> 0x0000150c22058cf8: 51515151 51515151 51515101 51515151 |QQQQQQQQ.QQQQQQQ| [00056.356] 01210.01279> 0x0000150c22058d08: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00056.356] 01210.01279> arch: x86_64 [00056.356] 01210.01279> PageFault:577: ERROR: failed to fault in or grab existing page [00056.356] 01210.01279> PageFault:578: 0xffffff80118b0c60 vmo_offset 0xc3dc678000, pf_flags 0x33 [00056.358] 01210.01603> PageFault:577: ERROR: failed to fault in or grab existing page [00056.358] 01210.01603> PageFault:578: 0xffffff8073471710 vmo_offset 0x6d0f1f5000, pf_flags 0x33 [00056.358] 162158.162199> PageFault:577: ERROR: failed to fault in or grab existing page [00056.358] 162158.162199> PageFault:578: 0xffffff807345f2d0 vmo_offset 0x7f04fa9000, pf_flags 0x33 [00056.401] 00000.00000> OOM: 0M free (-902.9M) / 2046.9M total [00056.401] 00000.00000> OOM: oom_lowmem(shortfall_bytes=52428800) called [00056.401] 00000.00000> OOM: Process mapped committed bytes: [00056.401] 00000.00000> OOM: proc 2866 24M 'devhost:pci#4:8086:2922' [00056.401] 00000.00000> OOM: proc 1101 184M 'fshost' [00056.402] 00000.00000> OOM: proc 2371 15M 'netstack' [00056.403] 00000.00000> OOM: proc 2485 29M './net_unittests' [00056.403] 00000.00000> OOM: proc 155319 15M 'net_unittests' [00056.454] 00000.00000> OOM: proc 161688 909M 'net_unittests' [00056.480] 00000.00000> OOM: proc 161826 663M 'net_unittests' [00056.481] 00000.00000> OOM: proc 162158 20M 'net_unittests' [00056.481] 00000.00000> OOM: Finding a job to kill... [00056.481] 00000.00000> OOM: (skip) job 3014 'tcp:22' [00056.481] 00000.00000> OOM: *KILL* job 2154 'boot' [00056.481] 00000.00000> OOM: + proc 2304 run 'sh' [00056.481] 00000.00000> OOM: + proc 2334 run 'listen' [00056.481] 00000.00000> OOM: + proc 2371 run 'netstack' [00056.481] 00000.00000> OOM: + proc 2485 run './net_unittests' [00056.481] 00000.00000> OOM: + proc 155319 run 'net_unittests' [00056.481] 00000.00000> OOM: + proc 161688 run 'net_unittests' [00056.481] 00000.00000> OOM: + proc 161826 run 'net_unittests' [00056.481] 00000.00000> OOM: + proc 162158 run 'net_unittests' [00056.481] 00000.00000> OOM: + job 3014 'tcp:22' [00056.481] 00000.00000> OOM: = 8 running procs (8 total), 1 jobs [00056.482] 00000.00000> OOM: (next) job 1782 'root' [00056.482] 00000.00000> OOM: (next) job 1080 'fuchsia' [00056.482] 00000.00000> OOM: (next) job 1079 'zircon-services' [00056.548] 01805.01823> [ERROR:garnet/bin/bootstrap/app.cc(127)] Singleton /system/bin/netstack died [00056.674] 02600.03153> eth [netstack]: tx_thread: exit: 0
,
Oct 18 2017
If you cherrypick https://chromium-review.googlesource.com/c/chromium/src/+/724111 -- does that solve the OOM crash?
,
Oct 18 2017
Crashed at exactly the same place on the next run. With --test-launcher-jobs=1 it hung after running out of FDs [00163.114] 02402.02448> [3720/18641] ProxyScriptFetcherImplTest.HttpStatusCode (10 ms) [00163.332] 152249.152278> Note: Google Test filter = ProxyScriptFetcherImplTest.ContentDisposition:ProxyScriptFetcherImplTest.NoCache:ProxyScriptFetcherImplTest.TooLarge:ProxyScriptFetcherImplTest.Hang:ProxyScriptFetcherImplTest.Encodings:ProxyScript [00163.332] 152249.152278> FetcherImplTest.DataURLs:ProxyScriptFetcherImplTest.Priority:ProxyScriptFetcherImplTest.OnShutdown:ProxyScriptFetcherImplTest.OnShutdownWithNoLiveRequest:ProxyServerTest.FromURI [00163.332] 152249.152278> [==========] Running 10 tests from 2 test cases. [00163.332] 152249.152278> [----------] Global test environment set-up. [00163.332] 152249.152278> [----------] 9 tests from ProxyScriptFetcherImplTest [00163.333] 152249.152278> [ RUN ] ProxyScriptFetcherImplTest.ContentDisposition [00163.351] 152249.152278> [ OK ] ProxyScriptFetcherImplTest.ContentDisposition (18 ms) [00163.351] 152249.152278> [ RUN ] ProxyScriptFetcherImplTest.NoCache [00163.357] 152249.152278> [3:260892393:1018/171749.658101:163357114:WARNING:net_errors_posix.cc(124)] Unknown error No error information (-74) mapped to net::ERR_FAILED [00163.357] 152249.152278> [ OK ] ProxyScriptFetcherImplTest.NoCache (6 ms) [00163.357] 152249.152278> [ RUN ] ProxyScriptFetcherImplTest.TooLarge [00163.368] 152249.152278> [ OK ] ProxyScriptFetcherImplTest.TooLarge (11 ms) [00163.368] 152249.152278> [ RUN ] ProxyScriptFetcherImplTest.Hang [00163.872] 152249.152278> [ OK ] ProxyScriptFetcherImplTest.Hang (504 ms) [00163.872] 152249.152278> [ RUN ] ProxyScriptFetcherImplTest.Encodings [00163.885] 152249.152278> [ OK ] ProxyScriptFetcherImplTest.Encodings (12 ms) [00163.885] 152249.152278> [ RUN ] ProxyScriptFetcherImplTest.DataURLs [00163.888] 152249.152278> [ OK ] ProxyScriptFetcherImplTest.DataURLs (0 ms) [00163.888] 152249.152278> [ RUN ] ProxyScriptFetcherImplTest.Priority [00164.003] 152249.152278> [3:260892393:1018/171750.304096:164003110:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.005] 152249.152278> [3:260892393:1018/171750.306937:164005950:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.008] 152249.152278> [3:260892393:1018/171750.309650:164008662:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.008] 152249.152278> [3:260892393:1018/171750.309937:164008949:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.012] 152249.152278> [3:260892393:1018/171750.313904:164012917:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.015] 152249.152278> [3:260892393:1018/171750.316390:164015403:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.015] 152249.152278> [3:260892393:1018/171750.316721:164015733:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.019] 152249.152278> [3:260892393:1018/171750.320193:164019206:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.019] 152249.152278> [3:260892393:1018/171750.320595:164019608:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.024] 152249.152278> [3:260892393:1018/171750.325318:164024331:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.024] 152249.152278> [3:260892393:1018/171750.325642:164024656:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.031] 152249.152278> [3:260892393:1018/171750.332312:164031327:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.035] 152249.152278> [3:260892393:1018/171750.336738:164035774:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.040] 152249.152278> [3:260892393:1018/171750.341068:164040081:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.040] 152249.152278> [3:260892393:1018/171750.341429:164040441:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.040] 152249.152278> [3:260892393:1018/171750.341744:164040757:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.041] 152249.152278> [3:260892393:1018/171750.342063:164041076:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.054] 152249.152278> [3:260892393:1018/171750.355376:164054388:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.054] 152249.152278> [3:260892393:1018/171750.355784:164054796:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.055] 152249.152278> [3:260892393:1018/171750.356335:164055352:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.055] 152249.152278> [3:260892393:1018/171750.356881:164055894:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.066] 152249.152278> [3:260892393:1018/171750.367472:164066485:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.066] 152249.152278> [3:260892393:1018/171750.367933:164066950:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.074] 152249.152278> [3:260892393:1018/171750.375330:164074346:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.078] 152249.152278> [3:260892393:1018/171750.379059:164078072:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.081] 152249.152278> [3:260892393:1018/171750.382785:164081799:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.085] 152249.152278> [3:260892393:1018/171750.386353:164085367:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.086] 152249.152278> [3:260892393:1018/171750.386999:164086014:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.091] 152249.152278> [3:260892393:1018/171750.392468:164091483:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.092] 152249.152278> [3:260892393:1018/171750.393180:164092195:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.092] 152249.152278> [3:260892393:1018/171750.393816:164092831:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.100] 152249.152278> [3:260892393:1018/171750.401382:164100397:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.100] 152249.152278> [3:260892393:1018/171750.401946:164100960:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.101] 152249.152278> [3:260892393:1018/171750.402441:164101455:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.108] 152249.152278> [3:260892393:1018/171750.409858:164108871:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.109] 152249.152278> [3:260892393:1018/171750.410531:164109546:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.110] 152249.152278> [3:260892393:1018/171750.411075:164110092:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.110] 152249.152278> [3:260892393:1018/171750.411642:164110657:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.119] 152249.152278> [3:260892393:1018/171750.420608:164119624:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.120] 152249.152278> [3:260892393:1018/171750.421253:164120270:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.124] 152249.152278> [3:260892393:1018/171750.425707:164124719:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.125] 152249.152278> [3:260892393:1018/171750.426350:164125364:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.130] 152249.152278> [3:260892393:1018/171750.431160:164130175:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.130] 152249.152278> [3:260892393:1018/171750.431742:164130756:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.135] 152249.152278> [3:260892393:1018/171750.436260:164135273:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.135] 152249.152278> [3:260892393:1018/171750.436806:164135821:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.140] 152249.152278> [3:260892393:1018/171750.441076:164140088:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.140] 152249.152278> [3:260892393:1018/171750.441666:164140681:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.144] 152249.152278> [3:260892393:1018/171750.445947:164144959:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.145] 152249.152278> [3:260892393:1018/171750.446457:164145471:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.146] 152249.152278> [3:260892393:1018/171750.446986:164146002:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.152] 152249.152278> [3:260892393:1018/171750.453045:164152060:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.154] 152249.152278> [3:260892393:1018/171750.455370:164154383:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.155] 152249.152278> [3:260892393:1018/171750.455987:164155001:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.155] 152249.152278> [3:260892393:1018/171750.456477:164155493:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.161] 152249.152278> [3:260892393:1018/171750.462309:164161324:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.161] 152249.152278> [3:260892393:1018/171750.462905:164161920:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.162] 152249.152278> [3:260892393:1018/171750.463466:164162481:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.167] 152249.152278> [3:260892393:1018/171750.468810:164167822:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.170] 152249.152278> [3:260892393:1018/171750.471463:164170475:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.173] 152249.152278> [3:260892393:1018/171750.474047:164173062:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.177] 152249.152278> [3:260892393:1018/171750.478316:164177331:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.177] 152249.152278> [3:260892393:1018/171750.478892:164177907:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.178] 152249.152278> [3:260892393:1018/171750.479380:164178394:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.178] 152249.152278> [3:260892393:1018/171750.479969:164178982:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.179] 152249.152278> [3:260892393:1018/171750.480405:164179419:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.179] 152249.152278> [3:260892393:1018/171750.480826:164179838:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.180] 152249.152278> [3:260892393:1018/171750.481163:164180175:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.201] 152249.152278> [3:260892393:1018/171750.502183:164201195:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.201] 152249.152278> [3:260892393:1018/171750.502934:164201946:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.207] 152249.152278> [3:260892393:1018/171750.508877:164207893:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.208] 152249.152278> [3:260892393:1018/171750.509555:164208570:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.214] 152249.152278> [3:260892393:1018/171750.515344:164214358:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.218] 152249.152278> [3:260892393:1018/171750.519011:164218026:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.218] 152249.152278> [3:260892393:1018/171750.519542:164218556:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.219] 152249.152278> [3:260892393:1018/171750.520049:164219065:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.219] 152249.152278> [3:260892393:1018/171750.520559:164219574:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.220] 152249.152278> [3:260892393:1018/171750.521068:164220083:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.226] 152249.152278> [3:260892393:1018/171750.527039:164226053:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.236] 152249.152278> [3:260892393:1018/171750.537005:164236019:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.236] 152249.152278> [3:260892393:1018/171750.537662:164236676:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.237] 152249.152278> [3:260892393:1018/171750.538172:164237187:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.237] 152249.152278> [3:260892393:1018/171750.538760:164237775:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.238] 152249.152278> [3:260892393:1018/171750.539315:164238328:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.239] 152249.152278> [3:260892393:1018/171750.539964:164238977:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.252] 152249.152278> [3:260892393:1018/171750.553495:164252511:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.257] 152249.152278> [3:260892393:1018/171750.558566:164257581:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.262] 152249.152278> [3:260892393:1018/171750.563001:164262016:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.266] 152249.152278> [3:260892393:1018/171750.567347:164266360:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.266] 152249.152278> [3:260892393:1018/171750.567852:164266867:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.267] 152249.152278> [3:260892393:1018/171750.568520:164267533:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.278] 152249.152278> [3:260892393:1018/171750.579372:164278387:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.279] 152249.152278> [3:260892393:1018/171750.580035:164279049:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.279] 152249.152278> [3:260892393:1018/171750.580628:164279643:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.280] 152249.152278> [3:260892393:1018/171750.581204:164280217:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.293] 152249.152278> [3:260892393:1018/171750.594655:164293670:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.297] 152249.152278> [3:260892393:1018/171750.598280:164297295:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.301] 152249.152278> [3:260892393:1018/171750.602476:164301491:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.302] 152249.152278> [3:260892393:1018/171750.603128:164302142:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.308] 152249.152278> [3:260892393:1018/171750.609197:164308212:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.308] 152249.152278> [3:260892393:1018/171750.609835:164308849:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.309] 152249.152278> [3:260892393:1018/171750.610366:164309381:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.318] 152249.152278> [3:260892393:1018/171750.619626:164318640:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.322] 152249.152278> [3:260892393:1018/171750.623592:164322607:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.323] 152249.152278> [3:260892393:1018/171750.624083:164323097:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.331] 152249.152278> [3:260892393:1018/171750.632033:164331048:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.335] 152249.152278> [3:260892393:1018/171750.636453:164335467:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.336] 152249.152278> [3:260892393:1018/171750.637137:164336151:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.336] 152249.152278> [3:260892393:1018/171750.637673:164336687:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.347] 152249.152278> [3:260892393:1018/171750.648127:164347140:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.347] 152249.152278> [3:260892393:1018/171750.648654:164347667:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.348] 152249.152278> [3:260892393:1018/171750.649131:164348146:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.348] 152249.152278> [3:260892393:1018/171750.649613:164348626:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.349] 152249.152278> [3:260892393:1018/171750.650088:164349102:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.349] 152249.152278> [3:260892393:1018/171750.650622:164349635:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.350] 152249.152278> [3:260892393:1018/171750.651073:164350086:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.350] 152249.152278> [3:260892393:1018/171750.651529:164350542:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.350] 152249.152278> [3:260892393:1018/171750.651971:164350984:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.351] 152249.152278> [3:260892393:1018/171750.652439:164351454:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.351] 152249.152278> [3:260892393:1018/171750.652895:164351909:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.352] 152249.152278> [3:260892393:1018/171750.653382:164352397:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.391] 152249.152278> [3:260892393:1018/171750.692539:164391554:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.395] 152249.152278> [3:260892393:1018/171750.696668:164395682:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.396] 152249.152278> [3:260892393:1018/171750.697176:164396190:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.400] 152249.152278> [3:260892393:1018/171750.701803:164400817:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.404] 152249.152278> [3:260892393:1018/171750.705499:164404513:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.405] 152249.152278> [3:260892393:1018/171750.706017:164405031:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.405] 152249.152278> [3:260892393:1018/171750.706465:164405480:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.414] 152249.152278> [3:260892393:1018/171750.715636:164414650:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.415] 152249.152278> [3:260892393:1018/171750.716208:164415221:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.415] 152249.152278> [3:260892393:1018/171750.716678:164415690:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.416] 152249.152278> [3:260892393:1018/171750.717037:164416050:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.427] 152249.152278> [3:260892393:1018/171750.728485:164427498:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.431] 152249.152278> [3:260892393:1018/171750.732043:164431056:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.431] 152249.152278> [3:260892393:1018/171750.732588:164431601:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.432] 152249.152278> [3:260892393:1018/171750.733149:164432161:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.432] 152249.152278> [3:260892393:1018/171750.733660:164432674:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.433] 152249.152278> [3:260892393:1018/171750.734147:164433161:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.433] 152249.152278> [3:260892393:1018/171750.734540:164433555:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.452] 152249.152278> [3:260892393:1018/171750.753703:164452717:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.457] 152249.152278> [3:260892393:1018/171750.758765:164457781:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00164.458] 152249.152278> [3:260892393:1018/171750.759412:164458425:ERROR:socket_posix.cc(91)] CreatePlatformSocket() returned an error, errno=24: No file descriptors available (24) [00178.114] 02402.02448> Still waiting for the following processes to finish: [00178.114] 02402.02448> net_unittests --gtest_flagfile=/tmp/.org.chromium.Chromium.GCogPp/.org.chromium.Chromium.mfgppa --single-process-tests --test-launcher-jobs=1 --test-launcher-output=/tmp/.org.chromium.Chromium.FIiKLK/test_results.xml --test- [00178.114] 02402.02448> launcher-retry-limit=0 [00178.114] 02402.02448> return_code=0 [00178.114] 02402.02448> started=1 [00178.114] 02402.02448> exited=0 [00178.114] 02402.02448> debugger_attached=0
,
Oct 18 2017
Thanks, will try https://chromium-review.googlesource.com/c/chromium/src/+/724111.
,
Oct 18 2017
Unfortunately, still crashing at the same spot after cherry-picking 724111 [00055.537] 02730.02826> [3817/18641] ThroughputAnalyzerTest.TestThroughputWithMultipleRequestsOverlap (13 ms) [00055.537] 02730.02826> [3818/18641] ThroughputAnalyzerTest.TestThroughputWithNetworkRequestsOverlap (7 ms) [00055.537] 02730.02826> [3819/18641] ThroughputAnalyzerTest.TestThroughputWithMultipleNetworkRequests (1 ms) [00055.537] 02730.02826> [3820/18641] NtlmBufferReaderTest.Initialization (0 ms) [00056.126] 161747.161780> PageFault:577: ERROR: failed to fault in or grab existing page [00056.126] 161747.161780> PageFault:578: 0xffffff805d78fe70 vmo_offset 0x8f7a010000, pf_flags 0x33 [00056.126] 161881.161918> PageFault:577: ERROR: failed to fault in or grab existing page [00056.126] 161881.161918> PageFault:578: 0xffffff807882e200 vmo_offset 0x7cdde2000, pf_flags 0x33 [00056.126] 162209.162246> PageFault:577: ERROR: failed to fault in or grab existing page [00056.126] 162209.162246> PageFault:578: 0xffffff8078853ce0 vmo_offset 0x51b2439000, pf_flags 0x33 [00056.126] 01212.01257> <== fatal exception: process net_unittests[161881] thread initial-thread[161918] [00056.126] 01212.01257> <== fatal page fault, PC at 0x4a0a0a8f43a3 [00056.126] 01212.01257> CS: 0 RIP: 0x4a0a0a8f43a3 EFL: 0x10202 CR2: 0x6ca57f519000 [00056.126] 01212.01257> RAX: 0x6ca57f519000 RBX: 0x1000 RCX: 0x1000 RDX: 0x1000 [00056.126] 01212.01257> RSI: 0x6d44a8d4d880 RDI: 0x6ca57f519000 RBP: 0x1 RSP: 0x171084c30c18 [00056.126] 01212.01257> R8: 0xffffffffffffffff R9: 0x6cadfb9ebf40 R10: 0x1000 R11: 0x6cadfb9ebed8 [00056.126] 01212.01257> R12: 0x1000 R13: 0 R14: 0x6ca57f519000 R15: 0 [00056.126] 01212.01257> errc: 0x6 [00056.126] 01212.01257> bottom of user stack: [00056.126] 01212.01257> 0x0000171084c30c18: 3ea29175 00003eb5 2b96a008 00006cd4 |u..>.>.....+.l..| [00056.126] 01212.01257> 0x0000171084c30c28: 00010000 00000000 51515151 51515151 |........QQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30c38: 0000001c 00000001 84c31048 00001710 |........H.......| [00056.126] 01212.01257> 0x0000171084c30c48: 00000000 00004a0a 51515151 51515151 |.....J..QQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30c58: 0a8d6dce 00004a0a 51515151 51515151 |.m...J..QQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30c68: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30c78: 51515151 51515151 cd857e6d 785dc8fa |QQQQQQQQm~....]x| [00056.126] 01212.01257> 0x0000171084c30c88: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30c98: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30ca8: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30cb8: 51515151 51515151 cd857e6d 785dc8fa |QQQQQQQQm~....]x| [00056.126] 01212.01257> 0x0000171084c30cc8: 84c30fd0 00001710 00000000 00000000 |................| [00056.126] 01212.01257> 0x0000171084c30cd8: 3e9f2244 00003eb5 51515101 51515151 |D".>.>...QQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30ce8: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00056.126] 01212.01257> 0x0000171084c30cf8: a8cf5e80 00006d44 a8cf5601 00006d44 |.^..Dm...V..Dm..| [00056.126] 01212.01257> 0x0000171084c30d08: 2b96a1c0 00006cd4 a8cf5e80 00006d44 |...+.l...^..Dm..| [00056.126] 01212.01257> arch: x86_64 [00056.126] 01212.01257> PageFault:577: ERROR: failed to fault in or grab existing page [00056.126] 01212.01257> PageFault:578: 0xffffff80118d25b0 vmo_offset 0xae04782000, pf_flags 0x33 [00056.126] 01212.01528> PageFault:577: ERROR: failed to fault in or grab existing page [00056.126] 01212.01528> PageFault:578: 0xffffff807885d980 vmo_offset 0x8b3797f000, pf_flags 0x33 [00056.365] 00000.00000> OOM: 0M free (-612.1M) / 2046.9M total [00056.365] 00000.00000> OOM: oom_lowmem(shortfall_bytes=52428800) called [00056.365] 00000.00000> OOM: Process mapped committed bytes: [00056.365] 00000.00000> OOM: proc 2672 24M 'devhost:pci#4:8086:2922' [00056.365] 00000.00000> OOM: proc 1101 184M 'fshost' [00056.367] 00000.00000> OOM: proc 2629 15M 'netstack' [00056.368] 00000.00000> OOM: proc 2730 29M './net_unittests' [00056.369] 00000.00000> OOM: proc 155369 15M 'net_unittests' [00056.423] 00000.00000> OOM: proc 161747 747M 'net_unittests' [00056.467] 00000.00000> OOM: proc 161881 758M 'net_unittests' [00056.470] 00000.00000> OOM: proc 162209 86M 'net_unittests' [00056.470] 00000.00000> OOM: Finding a job to kill... [00056.470] 00000.00000> OOM: (skip) job 3215 'tcp:22' [00056.470] 00000.00000> OOM: *KILL* job 2320 'boot' [00056.470] 00000.00000> OOM: + proc 2466 run 'sh' [00056.470] 00000.00000> OOM: + proc 2540 run 'listen' [00056.470] 00000.00000> OOM: + proc 2629 run 'netstack' [00056.470] 00000.00000> OOM: + proc 2730 run './net_unittests' [00056.470] 00000.00000> OOM: + proc 155369 run 'net_unittests' [00056.470] 00000.00000> OOM: + proc 161747 run 'net_unittests' [00056.470] 00000.00000> OOM: + proc 161881 run 'net_unittests' [00056.470] 00000.00000> OOM: + proc 162209 run 'net_unittests' [00056.470] 00000.00000> OOM: + job 3215 'tcp:22' [00056.470] 00000.00000> OOM: = 8 running procs (8 total), 1 jobs [00056.471] 00000.00000> OOM: (next) job 1915 'root' [00056.472] 00000.00000> OOM: (next) job 1080 'fuchsia' [00056.472] 00000.00000> OOM: (next) job 1079 'zircon-services' [00056.543] 01953.02011> [ERROR:garnet/bin/bootstrap/app.cc(127)] Singleton /system/bin/netstack died [00056.685] 02505.03234> eth [netstack]: tx_thread: exit: 0
,
Oct 18 2017
Ran it again with --test-launcher-jobs=1 and this time it crashed slightly later [00175.074] 02638.02701> [3847/18641] BbrSenderTest.Drain (27 ms) [00175.303] 156643.156672> Note: Google Test filter = BbrSenderTest.ProbeRtt:BbrSenderTest.InFlightAwareGainCycling:BbrSenderTest.NoBandwidthDropOnStartup:BbrSenderTest.SimpleTransfer1RTTStartup:BbrSenderTest.SimpleTransfer2RTTStartup:BbrSenderTest.Si [00175.303] 156643.156672> mpleTransferLRTTStartup:BbrSenderTest.SimpleTransferLRTTStartupSmallBuffer:BbrSenderTest.SimpleCompetition:BbrSenderTest.ResumeConnectionState:GeneralLossAlgorithmTest.NackRetransmit1Packet [00175.304] 156643.156672> [==========] Running 10 tests from 2 test cases. [00175.304] 156643.156672> [----------] Global test environment set-up. [00175.304] 156643.156672> [----------] 9 tests from BbrSenderTest [00175.304] 156643.156672> [ RUN ] BbrSenderTest.ProbeRtt [00175.947] 156643.156672> [ OK ] BbrSenderTest.ProbeRtt (643 ms) [00175.947] 156643.156672> [ RUN ] BbrSenderTest.InFlightAwareGainCycling [00175.994] 156643.156672> [ OK ] BbrSenderTest.InFlightAwareGainCycling (48 ms) [00175.994] 156643.156672> [ RUN ] BbrSenderTest.NoBandwidthDropOnStartup [00175.995] 156643.156672> [ OK ] BbrSenderTest.NoBandwidthDropOnStartup (0 ms) [00175.995] 156643.156672> [ RUN ] BbrSenderTest.SimpleTransfer1RTTStartup [00176.004] 156643.156672> [ OK ] BbrSenderTest.SimpleTransfer1RTTStartup (9 ms) [00176.004] 156643.156672> [ RUN ] BbrSenderTest.SimpleTransfer2RTTStartup [00176.014] 156643.156672> [ OK ] BbrSenderTest.SimpleTransfer2RTTStartup (11 ms) [00176.014] 156643.156672> [ RUN ] BbrSenderTest.SimpleTransferLRTTStartup [00176.027] 156643.156672> [ OK ] BbrSenderTest.SimpleTransferLRTTStartup (12 ms) [00176.027] 156643.156672> [ RUN ] BbrSenderTest.SimpleTransferLRTTStartupSmallBuffer [00176.045] 156643.156672> [ OK ] BbrSenderTest.SimpleTransferLRTTStartupSmallBuffer (19 ms) [00176.045] 156643.156672> [ RUN ] BbrSenderTest.SimpleCompetition [00177.710] 156643.156672> PageFault:577: ERROR: failed to fault in or grab existing page [00177.710] 156643.156672> PageFault:578: 0xffffff8019454d18 vmo_offset 0xd4113ee000, pf_flags 0x33 [00177.710] 01210.01272> <== fatal exception: process net_unittests[156643] thread initial-thread[156672] [00177.710] 01210.01272> <== fatal page fault, PC at 0x51ca94e893a3 [00177.710] 01210.01272> CS: 0 RIP: 0x51ca94e893a3 EFL: 0x10206 CR2: 0x2d8914622000 [00177.710] 01210.01272> RAX: 0x2d8914622000 RBX: 0x1000 RCX: 0x1000 RDX: 0x1000 [00177.710] 01210.01272> RSI: 0x2daa3d2f75c0 RDI: 0x2d8914622000 RBP: 0x1 RSP: 0x7526e49ddc98 [00177.710] 01210.01272> R8: 0xffffffffffffffff R9: 0x2d208eb40820 R10: 0x1000 R11: 0x2d208eb407b8 [00177.710] 01210.01272> R12: 0x1000 R13: 0 R14: 0x2d8914622000 R15: 0 [00177.710] 01210.01272> errc: 0x6 [00177.710] 01210.01272> bottom of user stack: [00177.710] 01210.01272> 0x00007526e49ddc98: 427aa175 000019d8 e0af2008 00002d1a |u.zB..... ...-..| [00177.710] 01210.01272> 0x00007526e49ddca8: 0001c000 00000000 e0af2008 00002d1a |......... ...-..| [00177.710] 01210.01272> 0x00007526e49ddcb8: 0000001c 00000001 e49de0c8 00007526 |............&u..| [00177.710] 01210.01272> 0x00007526e49ddcc8: 00000000 000051ca 00000040 00000000 |.....Q..@.......| [00177.710] 01210.01272> 0x00007526e49ddcd8: 94e6bdce 000051ca 00000000 00000000 |.....Q..........| [00177.710] 01210.01272> 0x00007526e49ddce8: 00000000 00000000 00000000 00000000 |................| [00177.710] 01210.01272> 0x00007526e49ddcf8: 00000000 00000000 d71902a4 011e0000 |................| [00177.710] 01210.01272> 0x00007526e49ddd08: 0a650502 068c1200 0003c40e 51ce3474 |..e.........t4.Q| [00177.710] 01210.01272> 0x00007526e49ddd18: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00177.710] 01210.01272> 0x00007526e49ddd28: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00177.710] 01210.01272> 0x00007526e49ddd38: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00177.710] 01210.01272> 0x00007526e49ddd48: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00177.710] 01210.01272> 0x00007526e49ddd58: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00177.710] 01210.01272> 0x00007526e49ddd68: 51515151 51515151 51515151 51515151 |QQQQQQQQQQQQQQQQ| [00177.710] 01210.01272> 0x00007526e49ddd78: 51515151 51515151 51515101 51515151 |QQQQQQQQ.QQQQQQQ| [00177.710] 01210.01272> 0x00007526e49ddd88: 51515151 51515151 51515101 51515151 |QQQQQQQQ.QQQQQQQ| [00177.710] 01210.01272> arch: x86_64 [00177.711] 01210.01272> dso: id=e8fd5b80b15c4b7e054a70e536f5c03c57606ee3 base=0x71431ddaa000 name=<vDSO> [00177.711] 01210.01272> dso: id=2cf38f18909b486a53fc6d7eb6c9a70de715f12a base=0x51ca94df9000 name=libc.so [00177.711] 01210.01272> dso: id=bd399512aedf1b853802fc812ff91cf7c3fa91d8 base=0x30beebf59000 name=libfdio.so [00177.711] 01210.01272> dso: id=9fe16781abc05309563e44deb8ea7c98d88a27d2 base=0x1c865d032000 name=liblaunchpad.so [00177.711] 01210.01272> dso: id=fe305415539c45ca base=0x19d83f518000 name=app:net_unittests [00177.711] 01210.01272> PageFault:577: ERROR: failed to fault in or grab existing page [00177.711] 01210.01272> PageFault:578: 0xffffff80118ab8b8 vmo_offset 0x1808313000, pf_flags 0x33 [00177.711] 01210.01582> PageFault:577: ERROR: failed to fault in or grab existing page [00177.711] 01210.01582> PageFault:578: 0xffffff8019475188 vmo_offset 0x7f9ae2d000, pf_flags 0x33 [00178.378] 00000.00000> OOM: 0M free (-203.9M) / 2046.9M total [00178.378] 00000.00000> OOM: oom_lowmem(shortfall_bytes=52428800) called [00178.378] 00000.00000> OOM: Process mapped committed bytes: [00178.379] 00000.00000> OOM: proc 2479 24M 'devhost:pci#4:8086:2922' [00178.379] 00000.00000> OOM: proc 1101 184M 'fshost' [00178.381] 00000.00000> OOM: proc 2594 12M 'netstack' [00178.381] 00000.00000> OOM: proc 2638 27M './net_unittests' [00178.487] 00000.00000> OOM: proc 156643 1619M 'net_unittests' [00178.487] 00000.00000> OOM: Finding a job to kill... [00178.487] 00000.00000> OOM: (skip) job 3090 'tcp:22' [00178.487] 00000.00000> OOM: *KILL* job 2364 'boot' [00178.487] 00000.00000> OOM: + proc 2509 run 'sh' [00178.487] 00000.00000> OOM: + proc 2549 run 'listen' [00178.487] 00000.00000> OOM: + proc 2594 run 'netstack' [00178.487] 00000.00000> OOM: + proc 2638 run './net_unittests' [00178.487] 00000.00000> OOM: + proc 156643 run 'net_unittests' [00178.487] 00000.00000> OOM: + job 3090 'tcp:22' [00178.487] 00000.00000> OOM: = 5 running procs (5 total), 1 jobs [00178.488] 00000.00000> OOM: (next) job 1990 'root' [00178.488] 00000.00000> OOM: (next) job 1080 'fuchsia' [00178.488] 00000.00000> OOM: (next) job 1079 'zircon-services' [00178.520] 02013.02073> [ERROR:garnet/bin/bootstrap/app.cc(127)] Singleton /system/bin/netstack died [00178.528] 02318.03125> eth [netstack]: tx_thread: exit: 0 I'm guessing the net_unittests subprocess shouldn't be 1619M.
,
Oct 18 2017
Let's try this again. https://chromium-review.googlesource.com/c/chromium/src/+/726243 -- does this solve the problem?
,
Oct 18 2017
Confirmed running locally as: out/fuch/bin/run_net_unittests --test-launcher-filter-file=../../testing/buildbot/filters/fuchsia.net_unittests.filter At HEAD (af61c6c60f76...), crashes similar to above. With 1229670f090af8379a05114ba1e0954acb51dde7 reverted, everything succeeds.
,
Oct 18 2017
726243 seems to fix. (You can add the "fuchsia" tryjob from CHOOSE TRYJOBS on the gerrit page.)
,
Oct 18 2017
Thanks for confirming! I'll try to land this ASAP.
,
Oct 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e33bfa66427111cab769ebefa00b0dd930858f3c commit e33bfa66427111cab769ebefa00b0dd930858f3c Author: Victor Vasiliev <vasilvv@chromium.org> Date: Wed Oct 18 19:01:47 2017 In QUIC simulated endpoint, generate the data on the fly instead of buffering. Merge internal change: 172619825 R=rch@chromium.org Bug: 775492 , 775987 Change-Id: If2ee82cfa44cc3f4f3c31d41bceea74584445ea4 Reviewed-on: https://chromium-review.googlesource.com/726243 Reviewed-by: Ryan Hamilton <rch@chromium.org> Commit-Queue: Ryan Hamilton <rch@chromium.org> Cr-Commit-Position: refs/heads/master@{#509823} [modify] https://crrev.com/e33bfa66427111cab769ebefa00b0dd930858f3c/net/quic/core/congestion_control/bbr_sender_test.cc [modify] https://crrev.com/e33bfa66427111cab769ebefa00b0dd930858f3c/net/quic/core/congestion_control/send_algorithm_test.cc [modify] https://crrev.com/e33bfa66427111cab769ebefa00b0dd930858f3c/net/quic/test_tools/simulator/quic_endpoint.cc [modify] https://crrev.com/e33bfa66427111cab769ebefa00b0dd930858f3c/net/quic/test_tools/simulator/quic_endpoint.h
,
Oct 18 2017
Bot is green again at https://build.chromium.org/p/chromium.fyi/builders/Fuchsia/builds/10301 . Thanks! |
||||
►
Sign in to add a comment |
||||
Comment 1 by scottmg@chromium.org
, Oct 18 2017