Build failure on M59 |
||||||||
Issue descriptionBuilds for moblab have been failing since 4/13. Culprit: HWTest is failing Example log https://uberchromegw.corp.google.com/i/chromeos_release/builders/guado_moblab-release%20release-R59-9460.B/builds/9/steps/HWTest%20%5Bmoblab%5D/logs/stdio
,
Apr 28 2017
Adding ayatane@chromium.org since the CL's related to suite refactor look the most suspicious. Any insight as to why run_suite might be crashing on moblab 59 ?
,
Apr 28 2017
It looks like the suite job finished normally. The test result collection failed 04-28-2017 [02:36:29] Suite job is finished. 04-28-2017 [02:36:29] Start collectint test results and dump them to json. Write failed: Broken pipe. Exception log follows the after_iteration_hooks. Or rather, wherever the JSON output was being written was closed.
,
Apr 28 2017
Moved over info from incorrect bug #
,
Apr 28 2017
Best I can tell it has something to do with the dbus test - it is crashing the moblab with OOM This test causes the issue platform_CrosDisksDBus This CL is now the most likely to be the culprit https://chromium-review.googlesource.com/#/c/465546/ Adding the owner of that CL to see if there is any insight /var/log/messages for just before the crash. 2017-04-28T15:10:35.884738-07:00 INFO dhcpd[1815]: DHCPREQUEST for 192.168.231.100 (192.168.231.1) from 14:91:82:3c:0b:6e via lxcbr0 2017-04-28T15:10:35.884787-07:00 INFO dhcpd[1815]: DHCPACK on 192.168.231.100 to 14:91:82:3c:0b:6e via lxcbr0 2017-04-28T15:12:43.728846-07:00 WARNING kernel: [ 380.179757] init: whining-update-db main process ended, respawning 2017-04-28T15:12:51.679039-07:00 INFO kernel: [ 388.135228] entering low_mem (avail RAM = 205768 kB, avail swap 662176 kB) with lowest seen anon mem: 1861788 kB 2017-04-28T15:12:51.872833-07:00 INFO kernel: [ 388.329096] entering low_mem (avail RAM = 205792 kB, avail swap 634992 kB) with lowest seen anon mem: 1850216 kB 2017-04-28T15:12:58.512554-07:00 INFO sshd[11036]: Accepted publickey for moblab from 172.22.64.195 port 44384 ssh2: RSA SHA256:Fp1qWjFLyK1cTpiI5rdk7iEJwoK9lcnYAgbQtGC3jzU 2017-04-28T15:13:27.843722-07:00 INFO kernel: [ 424.318097] entering low_mem (avail RAM = 205816 kB, avail swap 637888 kB) with lowest seen anon mem: 1825488 kB 2017-04-28T15:13:29.133678-07:00 INFO kernel: [ 425.609900] entering low_mem (avail RAM = 205728 kB, avail swap 561932 kB) with lowest seen anon mem: 1804216 kB 2017-04-28T15:13:36.489276-07:00 INFO kernel: [ 432.968366] entering low_mem (avail RAM = 205732 kB, avail swap 275648 kB) with lowest seen anon mem: 1634576 kB 2017-04-28T15:13:40.488680-07:00 INFO kernel: [ 436.970653] entering low_mem (avail RAM = 205780 kB, avail swap 162620 kB) with lowest seen anon mem: 1582132 kB 2017-04-28T15:13:44.050678-07:00 INFO kernel: [ 440.533634] entering low_mem (avail RAM = 205800 kB, avail swap 112228 kB) with lowest seen anon mem: 1551832 kB 2017-04-28T15:13:49.435578-07:00 INFO kernel: [ 445.919897] entering low_mem (avail RAM = 205760 kB, avail swap 57284 kB) with lowest seen anon mem: 1517828 kB 2017-04-28T15:13:49.436233-07:00 INFO kernel: [ 445.919912] entering low_mem (avail RAM = 205760 kB, avail swap 57284 kB) with lowest seen anon mem: 1517828 kB 2017-04-28T15:13:50.828681-07:00 WARNING kernel: [ 447.315644] perf samples too long (2509 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 2017-04-28T15:13:57.480289-07:00 INFO kernel: [ 453.969603] entering low_mem (avail RAM = 205744 kB, avail swap 15224 kB) with lowest seen anon mem: 1503248 kB 2017-04-28T15:13:57.480324-07:00 INFO kernel: [ 453.969614] entering low_mem (avail RAM = 205744 kB, avail swap 15224 kB) with lowest seen anon mem: 1503248 kB 2017-04-28T15:14:06.824011-07:00 WARNING kernel: [ 463.317592] gsutil invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=-1000 2017-04-28T15:14:06.824026-07:00 INFO kernel: [ 463.317605] gsutil cpuset=/ mems_allowed=0 2017-04-28T15:14:06.824031-07:00 NOTICE kernel: [ 463.317616] CPU: 3 PID: 20144 Comm: gsutil Not tainted 3.14.0 #1 2017-04-28T15:14:06.824032-07:00 NOTICE kernel: [ 463.317623] Hardware name: GOOGLE Guado, BIOS Google_Guado.6301.108.4 04/01/2015 2017-04-28T15:14:06.824033-07:00 NOTICE kernel: [ 463.317631] 0000000000000000 000000006f4768f9 ffff880013fd7b70 ffffffff8fe1c86c 2017-04-28T15:14:06.824034-07:00 NOTICE kernel: [ 463.317643] ffff880077999240 ffff880013fd7be0 ffffffff8fe1ba1a ffffffff8f96c1e9 2017-04-28T15:14:06.824036-07:00 NOTICE kernel: [ 463.317654] ffffffff90448b10 000000006f4768f9 0000000000000206 ffff880013fd7bb8 2017-04-28T15:14:06.824058-07:00 NOTICE kernel: [ 463.317665] Call Trace: 2017-04-28T15:14:06.824060-07:00 NOTICE kernel: [ 463.317674] [<ffffffff8fe1c86c>] dump_stack+0x4d/0x6f 2017-04-28T15:14:06.824061-07:00 NOTICE kernel: [ 463.317684] [<ffffffff8fe1ba1a>] dump_header.isra.9+0xaa/0x24e 2017-04-28T15:14:06.824062-07:00 NOTICE kernel: [ 463.317694] [<ffffffff8f96c1e9>] ? wakeup_flusher_threads+0x77/0x94 2017-04-28T15:14:06.824063-07:00 NOTICE kernel: [ 463.317704] [<ffffffff8f903ffd>] oom_kill_process+0x92/0x375 2017-04-28T15:14:06.824064-07:00 NOTICE kernel: [ 463.317713] [<ffffffff8f9046e4>] out_of_memory+0x250/0x2d9 2017-04-28T15:14:06.824065-07:00 NOTICE kernel: [ 463.317722] [<ffffffff8f9083cd>] __alloc_pages_nodemask+0x73d/0x895 2017-04-28T15:14:06.824076-07:00 NOTICE kernel: [ 463.317732] [<ffffffff8f923e3a>] handle_mm_fault+0x339/0x859 2017-04-28T15:14:06.824078-07:00 NOTICE kernel: [ 463.317742] [<ffffffff8f833ca4>] __do_page_fault+0x290/0x383 2017-04-28T15:14:06.824079-07:00 NOTICE kernel: [ 463.317751] [<ffffffff8fd27be2>] ? fput_light+0xd/0xf 2017-04-28T15:14:06.824080-07:00 NOTICE kernel: [ 463.317759] [<ffffffff8fd29a58>] ? SYSC_accept4+0x1b8/0x1e5 2017-04-28T15:14:06.824081-07:00 NOTICE kernel: [ 463.317768] [<ffffffff8f85cb01>] ? do_fork+0xca/0x2c4 2017-04-28T15:14:06.824082-07:00 NOTICE kernel: [ 463.317777] [<ffffffff8f833dc7>] do_page_fault+0xc/0xe 2017-04-28T15:14:06.824093-07:00 NOTICE kernel: [ 463.317786] [<ffffffff8fe24812>] page_fault+0x22/0x30 2017-04-28T15:14:06.824094-07:00 NOTICE kernel: [ 463.317792] Mem-Info: 2017-04-28T15:14:06.824095-07:00 NOTICE kernel: [ 463.317796] DMA per-cpu: 2017-04-28T15:14:06.824096-07:00 NOTICE kernel: [ 463.317801] CPU 0: hi: 0, btch: 1 usd: 0 2017-04-28T15:14:06.824097-07:00 NOTICE kernel: [ 463.317807] CPU 1: hi: 0, btch: 1 usd: 0 2017-04-28T15:14:06.824098-07:00 NOTICE kernel: [ 463.317812] CPU 2: hi: 0, btch: 1 usd: 0 2017-04-28T15:14:06.824100-07:00 NOTICE kernel: [ 463.317818] CPU 3: hi: 0, btch: 1 usd: 0 2017-04-28T15:14:06.824114-07:00 NOTICE kernel: [ 463.317824] DMA32 per-cpu: 2017-04-28T15:14:06.824116-07:00 NOTICE kernel: [ 463.317828] CPU 0: hi: 186, btch: 31 usd: 0 2017-04-28T15:14:06.824117-07:00 NOTICE kernel: [ 463.317834] CPU 1: hi: 186, btch: 31 usd: 36 2017-04-28T15:14:06.824118-07:00 NOTICE kernel: [ 463.317840] CPU 2: hi: 186, btch: 31 usd: 19 2017-04-28T15:14:06.824119-07:00 NOTICE kernel: [ 463.317846] CPU 3: hi: 186, btch: 31 usd: 156 2017-04-28T15:14:06.824120-07:00 NOTICE kernel: [ 463.317851] Normal per-cpu: 2017-04-28T15:14:06.824130-07:00 NOTICE kernel: [ 463.317856] CPU 0: hi: 186, btch: 31 usd: 38 2017-04-28T15:14:06.824131-07:00 NOTICE kernel: [ 463.317862] CPU 1: hi: 186, btch: 31 usd: 118 2017-04-28T15:14:06.824132-07:00 NOTICE kernel: [ 463.317868] CPU 2: hi: 186, btch: 31 usd: 32 2017-04-28T15:14:06.824133-07:00 NOTICE kernel: [ 463.317874] CPU 3: hi: 186, btch: 31 usd: 176 2017-04-28T15:14:06.824134-07:00 NOTICE kernel: [ 463.317883] active_anon:279443 inactive_anon:142288 isolated_anon:32 2017-04-28T15:14:06.824135-07:00 NOTICE kernel: [ 463.317883] active_file:4774 inactive_file:6581 isolated_file:0 2017-04-28T15:14:06.824136-07:00 NOTICE kernel: [ 463.317883] unevictable:0 dirty:0 writeback:1 unstable:0 2017-04-28T15:14:06.824147-07:00 NOTICE kernel: [ 463.317883] free:39712 slab_reclaimable:5773 slab_unreclaimable:9620 2017-04-28T15:14:06.824149-07:00 NOTICE kernel: [ 463.317883] mapped:4124 shmem:8969 pagetables:8391 bounce:0 2017-04-28T15:14:06.824150-07:00 NOTICE kernel: [ 463.317883] free_cma:0 2017-04-28T15:14:06.824151-07:00 NOTICE kernel: [ 463.317917] DMA free:15788kB min:540kB low:672kB high:808kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2017-04-28T15:14:06.824153-07:00 NOTICE kernel: [ 463.317945] lowmem_reserve[]: 0 1903 3863 3863 2017-04-28T15:14:06.824154-07:00 NOTICE kernel: [ 463.317962] DMA32 free:74768kB min:66304kB low:82880kB high:99456kB active_anon:822840kB inactive_anon:274408kB active_file:2808kB inactive_file:13800kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2029812kB managed:1953336kB mlocked:0kB dirty:0kB writeback:0kB mapped:3176kB shmem:10532kB slab_reclaimable:8376kB slab_unreclaimable:16988kB kernel_stack:3424kB pagetables:15440kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 2017-04-28T15:14:06.824164-07:00 NOTICE kernel: [ 463.317994] lowmem_reserve[]: 0 0 1960 1960 2017-04-28T15:14:06.824166-07:00 NOTICE kernel: [ 463.318009] Normal free:68292kB min:68320kB low:85400kB high:102480kB active_anon:294932kB inactive_anon:294744kB active_file:16288kB inactive_file:12524kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:2080768kB managed:2007960kB mlocked:0kB dirty:36kB writeback:4kB mapped:13320kB shmem:25344kB slab_reclaimable:14716kB slab_unreclaimable:21492kB kernel_stack:2552kB pagetables:18124kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2500 all_unreclaimable? yes 2017-04-28T15:14:06.824167-07:00 NOTICE kernel: [ 463.318041] lowmem_reserve[]: 0 0 0 0 2017-04-28T15:14:06.824168-07:00 NOTICE kernel: [ 463.318051] DMA: 1*4kB (U) 1*8kB (U) 0*16kB 1*32kB (U) 0*64kB 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (MR) = 15788kB 2017-04-28T15:14:06.824170-07:00 NOTICE kernel: [ 463.318086] DMA32: 602*4kB (UEM) 2492*8kB (UEM) 2172*16kB (UEM) 108*32kB (UEM) 22*64kB (UE) 6*128kB (UE) 3*256kB (UE) 0*512kB 1*1024kB (E) 1*2048kB (E) 2*4096kB (R) = 74760kB 2017-04-28T15:14:06.824179-07:00 NOTICE kernel: [ 463.318126] Normal: 537*4kB (UEM) 6307*8kB (UEM) 271*16kB (UEM) 74*32kB (UEMR) 20*64kB (UER) 4*128kB (UER) 4*256kB (R) 0*512kB 0*1024kB 1*2048kB (R) 1*4096kB (R) = 68268kB 2017-04-28T15:14:06.824181-07:00 NOTICE kernel: [ 463.318164] 25166 total pagecache pages 2017-04-28T15:14:06.824182-07:00 NOTICE kernel: [ 463.318169] 4721 pages in swap cache 2017-04-28T15:14:06.824183-07:00 NOTICE kernel: [ 463.318174] Swap cache stats: add 3349978, delete 3345257, find 97086/1053114 2017-04-28T15:14:06.824184-07:00 NOTICE kernel: [ 463.318181] Free swap = 0kB 2017-04-28T15:14:06.824185-07:00 NOTICE kernel: [ 463.318185] Total swap = 5825980kB 2017-04-28T15:14:06.824196-07:00 NOTICE kernel: [ 463.318190] 1031644 pages RAM 2017-04-28T15:14:06.824197-07:00 NOTICE kernel: [ 463.318194] 0 pages HighMem/MovableOnly 2017-04-28T15:14:06.824198-07:00 NOTICE kernel: [ 463.318198] 18202 pages reserved 2017-04-28T15:14:06.824199-07:00 INFO kernel: [ 463.318202] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 2017-04-28T15:14:06.824201-07:00 INFO kernel: [ 463.318218] [ 126] 0 126 3179 144 11 211 -1000 udevd 2017-04-28T15:14:06.824202-07:00 INFO kernel: [ 463.318227] [ 180] 0 180 4348 104 12 208 -1000 frecon 2017-04-28T15:14:06.824203-07:00 INFO kernel: [ 463.318238] [ 337] 0 337 23215 111 12 69 -1000 adb 2017-04-28T15:14:06.824214-07:00 INFO kernel: [ 463.318247] [ 371] 202 371 74263 193 18 139 -1000 rsyslogd 2017-04-28T15:14:06.824215-07:00 INFO kernel: [ 463.318257] [ 393] 201 393 2944 166 9 57 -1000 dbus-daemon 2017-04-28T15:14:06.824216-07:00 INFO kernel: [ 463.318266] [ 528] 0 528 276645 3677 68 10090 -1000 python2 2017-04-28T15:14:06.824217-07:00 INFO kernel: [ 463.318275] [ 532] 0 532 5325 141 15 117 -1000 firewalld 2017-04-28T15:14:06.824218-07:00 INFO kernel: [ 463.318284] [ 534] 219 534 4541 162 14 182 -1000 wpa_supplicant 2017-04-28T15:14:06.824219-07:00 INFO kernel: [ 463.318294] [ 564] 230 564 7951 194 20 509 -1000 permission_brok 2017-04-28T15:14:06.824228-07:00 INFO kernel: [ 463.318303] [ 705] 228 705 26270 218 20 154 -1000 powerd 2017-04-28T15:14:06.824229-07:00 INFO kernel: [ 463.318312] [ 722] 0 722 9150 203 23 320 -1000 session_manager 2017-04-28T15:14:06.824230-07:00 INFO kernel: [ 463.318322] [ 747] 1000 747 133665 2876 160 5872 -1000 chrome 2017-04-28T15:14:06.824231-07:00 INFO kernel: [ 463.318331] [ 781] 0 781 5884 160 16 134 -1000 debugd 2017-04-28T15:14:06.824233-07:00 INFO kernel: [ 463.318340] [ 806] 207 806 64112 34 22 163 -1000 tcsd 2017-04-28T15:14:06.824234-07:00 INFO kernel: [ 463.318350] [ 811] 223 811 26002 138 19 235 -1000 chapsd 2017-04-28T15:14:06.824235-07:00 INFO kernel: [ 463.318359] [ 821] 1000 821 55756 4936 72 4828 -1000 chrome 2017-04-28T15:14:06.828451-07:00 INFO kernel: [ 463.319310] [ 824] 1000 824 7657 150 19 240 -1000 nacl_helper 2017-04-28T15:14:06.828459-07:00 INFO kernel: [ 463.319323] [ 827] 1000 827 947 0 5 31 -1000 nacl_helper_non 2017-04-28T15:14:06.828461-07:00 INFO kernel: [ 463.319333] [ 830] 1000 830 55756 4760 45 4829 -1000 chrome 2017-04-28T15:14:06.828462-07:00 INFO kernel: [ 463.319343] [ 856] 1000 856 68027 1229 106 4796 -1000 chrome 2017-04-28T15:14:06.828463-07:00 INFO kernel: [ 463.319351] [ 874] 1000 874 49821 40 39 2294 -1000 chrome 2017-04-28T15:14:06.828465-07:00 INFO kernel: [ 463.319361] [ 993] 245 993 338602 5669 88 10611 -1000 mysqld 2017-04-28T15:14:06.828478-07:00 INFO kernel: [ 463.319405] [ 1016] 0 1016 28075 132 24 356 -1000 cryptohomed 2017-04-28T15:14:06.828480-07:00 INFO kernel: [ 463.319415] [ 1022] 1000 1022 190328 10017 229 18840 300 chrome 2017-04-28T15:14:06.828481-07:00 INFO kernel: [ 463.319425] [ 1200] 0 1200 1081 117 7 29 -1000 periodic_schedu 2017-04-28T15:14:06.828482-07:00 INFO kernel: [ 463.319435] [ 1208] 0 1208 1081 117 7 29 -1000 periodic_schedu 2017-04-28T15:14:06.828483-07:00 INFO kernel: [ 463.319444] [ 1216] 0 1216 2152 118 10 43 -1000 minijail0 2017-04-28T15:14:06.828484-07:00 INFO kernel: [ 463.319453] [ 1225] 0 1225 1081 117 6 27 -1000 periodic_schedu 2017-04-28T15:14:06.828485-07:00 INFO kernel: [ 463.319463] [ 1230] 226 1230 9431 131 18 168 -1000 mtpd 2017-04-28T15:14:06.828496-07:00 INFO kernel: [ 463.319472] [ 1261] 0 1261 2152 106 9 42 -1000 minijail0 2017-04-28T15:14:06.828497-07:00 INFO kernel: [ 463.319481] [ 1269] 0 1269 2152 106 9 41 -1000 minijail0 2017-04-28T15:14:06.828498-07:00 INFO kernel: [ 463.319491] [ 1273] 600 1273 23675 178 16 328 -1000 cras 2017-04-28T15:14:06.828500-07:00 INFO kernel: [ 463.319499] [ 1280] 241 1280 60726 151 20 251 -1000 ModemManager 2017-04-28T15:14:06.828501-07:00 INFO kernel: [ 463.319509] [ 1282] 218 1282 5161 139 15 135 -1000 bluetoothd 2017-04-28T15:14:06.828502-07:00 INFO kernel: [ 463.319519] [ 1286] 0 1286 2152 119 10 42 -1000 minijail0 2017-04-28T15:14:06.828513-07:00 INFO kernel: [ 463.319528] [ 1289] 0 1289 6837 125 17 154 -1000 metrics_daemon 2017-04-28T15:14:06.828515-07:00 INFO kernel: [ 463.319539] [ 1292] 232 1292 1741 108 8 60 -1000 conntrackd 2017-04-28T15:14:06.828516-07:00 INFO kernel: [ 463.319549] [ 1297] 0 1297 7053 96 17 217 -1000 update_engine 2017-04-28T15:14:06.828517-07:00 INFO kernel: [ 463.319558] [ 1300] 238 1300 3403 140 10 43 -1000 avahi-daemon 2017-04-28T15:14:06.828518-07:00 INFO kernel: [ 463.319568] [ 1302] 0 1302 26318 146 19 241 -1000 disks 2017-04-28T15:14:06.828519-07:00 INFO kernel: [ 463.319577] [ 1305] 0 1305 3945 62 11 106 -1000 sshd 2017-04-28T15:14:06.828520-07:00 INFO kernel: [ 463.319586] [ 1307] 238 1307 3403 37 9 57 -1000 avahi-daemon 2017-04-28T15:14:06.828531-07:00 INFO kernel: [ 463.319596] [ 1360] 0 1360 2261 17 9 61 -1000 upstart-socket- 2017-04-28T15:14:06.828533-07:00 INFO kernel: [ 463.319607] [ 1815] 224 1815 3246 98 10 1219 -1000 dhcpd 2017-04-28T15:14:06.828534-07:00 INFO kernel: [ 463.319616] [ 2020] 0 2020 6062 443 16 1121 -1000 python 2017-04-28T15:14:06.828535-07:00 INFO kernel: [ 463.319625] [ 2022] 0 2022 6193 123 15 125 -1000 warn_collector 2017-04-28T15:14:06.828536-07:00 INFO kernel: [ 463.319634] [ 2033] 0 2033 1081 140 6 13 -1000 sh 2017-04-28T15:14:06.828537-07:00 INFO kernel: [ 463.319643] [ 2051] 0 2051 1081 107 7 24 -1000 sh 2017-04-28T15:14:06.828548-07:00 INFO kernel: [ 463.319652] [ 2087] 0 2087 1081 117 7 29 -1000 periodic_schedu 2017-04-28T15:14:06.828550-07:00 INFO kernel: [ 463.319662] [ 2089] 234 2089 3859 114 12 105 -1000 tlsdated 2017-04-28T15:14:06.828551-07:00 INFO kernel: [ 463.319671] [ 2090] 0 2090 1060 108 7 31 -1000 logger 2017-04-28T15:14:06.828552-07:00 INFO kernel: [ 463.319680] [ 2109] 0 2109 3336 1 10 92 -1000 tlsdated-setter 2017-04-28T15:14:06.828553-07:00 INFO kernel: [ 463.319690] [ 2450] 0 2450 8502 300 20 186 -1000 shill 2017-04-28T15:14:06.828554-07:00 INFO kernel: [ 463.319699] [ 2456] 202 2456 1613 110 8 34 -1000 logger 2017-04-28T15:14:06.828555-07:00 INFO kernel: [ 463.319708] [ 2485] 224 2485 2799 132 10 74 -1000 dhcpcd 2017-04-28T15:14:06.828566-07:00 INFO kernel: [ 463.323698] [ 3820] 0 3820 3909 123 11 87 -1000 sudo 2017-04-28T15:14:06.828568-07:00 INFO kernel: [ 463.323712] [ 3822] 0 3822 1081 110 7 24 -1000 sh 2017-04-28T15:14:06.828569-07:00 INFO kernel: [ 463.323722] [ 3825] 246 3825 844132 479 227 14614 -1000 devserver.py 2017-04-28T15:14:06.828570-07:00 INFO kernel: [ 463.323731] [ 3828] 0 3828 2361 67 9 38 -1000 sleep 2017-04-28T15:14:06.828571-07:00 INFO kernel: [ 463.323741] [ 3830] 0 3830 6352 194 16 333 -1000 apache2 2017-04-28T15:14:06.828572-07:00 INFO kernel: [ 463.323751] [ 3831] 0 3831 3337 92 11 51 -1000 rotatelogs 2017-04-28T15:14:06.828573-07:00 INFO kernel: [ 463.323760] [ 3832] 0 3832 3337 98 10 41 -1000 rotatelogs 2017-04-28T15:14:06.828584-07:00 INFO kernel: [ 463.323771] [ 3833] 244 3833 6334 128 16 351 -1000 apache2 2017-04-28T15:14:06.828586-07:00 INFO kernel: [ 463.323781] [ 4118] 244 4118 79894 2844 68 15504 -1000 apache2 2017-04-28T15:14:06.828587-07:00 INFO kernel: [ 463.323791] [ 4119] 244 4119 88342 4169 97 29262 -1000 apache2 2017-04-28T15:14:06.828588-07:00 INFO kernel: [ 463.323800] [ 4120] 244 4120 88406 6309 97 27385 -1000 apache2 2017-04-28T15:14:06.828589-07:00 INFO kernel: [ 463.323809] [ 4121] 244 4121 88342 2794 97 30595 -1000 apache2 2017-04-28T15:14:06.828590-07:00 INFO kernel: [ 463.323818] [ 4122] 244 4122 88406 2294 97 31292 -1000 apache2 2017-04-28T15:14:06.828601-07:00 INFO kernel: [ 463.323828] [ 4123] 244 4123 79894 525 69 18325 -1000 apache2 2017-04-28T15:14:06.828603-07:00 INFO kernel: [ 463.323837] [ 4124] 244 4124 79894 3351 69 15658 -1000 apache2 2017-04-28T15:14:06.828604-07:00 INFO kernel: [ 463.323847] [ 4125] 244 4125 79894 2230 69 16633 -1000 apache2 2017-04-28T15:14:06.828605-07:00 INFO kernel: [ 463.323857] [ 4126] 244 4126 79894 1194 69 17674 -1000 apache2 2017-04-28T15:14:06.828606-07:00 INFO kernel: [ 463.323866] [ 4127] 244 4127 88342 31611 97 2264 -1000 apache2 2017-04-28T15:14:06.828607-07:00 INFO kernel: [ 463.323876] [ 4128] 244 4128 79894 1309 69 17560 -1000 apache2 2017-04-28T15:14:06.828609-07:00 INFO kernel: [ 463.323885] [ 4129] 244 4129 88342 4676 97 28830 -1000 apache2 2017-04-28T15:14:06.828619-07:00 INFO kernel: [ 463.323895] [ 4130] 244 4130 88342 6258 97 27174 -1000 apache2 2017-04-28T15:14:06.835707-07:00 INFO kernel: [ 463.331022] [ 4131] 244 4131 88342 5451 97 28049 -1000 apache2 2017-04-28T15:14:06.835720-07:00 INFO kernel: [ 463.331035] [ 4132] 244 4132 88406 3241 97 30354 -1000 apache2 2017-04-28T15:14:06.835722-07:00 INFO kernel: [ 463.331045] [ 4133] 244 4133 88342 5195 97 28368 -1000 apache2 2017-04-28T15:14:06.835723-07:00 INFO kernel: [ 463.331054] [ 4134] 244 4134 88342 29423 97 4000 -1000 apache2 2017-04-28T15:14:06.835724-07:00 INFO kernel: [ 463.331063] [ 4135] 244 4135 88342 6928 97 26492 -1000 apache2 2017-04-28T15:14:06.835725-07:00 INFO kernel: [ 463.331071] [ 4136] 244 4136 88342 29426 97 3944 -1000 apache2 2017-04-28T15:14:06.835727-07:00 INFO kernel: [ 463.331080] [ 4137] 244 4137 79894 3439 69 15505 -1000 apache2 2017-04-28T15:14:06.835729-07:00 INFO kernel: [ 463.331089] [ 4138] 244 4138 88342 11599 97 21838 -1000 apache2 2017-04-28T15:14:06.835730-07:00 INFO kernel: [ 463.331098] [ 4139] 244 4139 79894 3490 69 15457 -1000 apache2 2017-04-28T15:14:06.835731-07:00 INFO kernel: [ 463.331107] [ 4140] 244 4140 79894 3307 69 15652 -1000 apache2 2017-04-28T15:14:06.835732-07:00 INFO kernel: [ 463.331116] [ 4141] 244 4141 88342 18168 97 15350 -1000 apache2 2017-04-28T15:14:06.835734-07:00 INFO kernel: [ 463.331125] [ 4142] 244 4142 79894 1867 69 16982 -1000 apache2 2017-04-28T15:14:06.835735-07:00 INFO kernel: [ 463.331134] [ 4143] 244 4143 88342 1385 97 32002 -1000 apache2 2017-04-28T15:14:06.835736-07:00 INFO kernel: [ 463.331144] [ 4144] 244 4144 79894 3035 69 15833 -1000 apache2 2017-04-28T15:14:06.835737-07:00 INFO kernel: [ 463.331153] [ 4145] 244 4145 79958 2855 69 16081 -1000 apache2 2017-04-28T15:14:06.835738-07:00 INFO kernel: [ 463.331162] [ 4146] 244 4146 88406 5192 97 28425 -1000 apache2 2017-04-28T15:14:06.835740-07:00 INFO kernel: [ 463.331171] [ 4147] 244 4147 79894 3264 69 15636 -1000 apache2 2017-04-28T15:14:06.835741-07:00 INFO kernel: [ 463.331180] [ 4148] 244 4148 79894 1480 69 17380 -1000 apache2 2017-04-28T15:14:06.835742-07:00 INFO kernel: [ 463.331189] [ 4149] 244 4149 79894 2972 69 15878 -1000 apache2 2017-04-28T15:14:06.835743-07:00 INFO kernel: [ 463.331198] [ 4150] 244 4150 79894 3413 69 15607 -1000 apache2 2017-04-28T15:14:06.835745-07:00 INFO kernel: [ 463.331207] [ 4151] 244 4151 79893 3304 69 15656 -1000 apache2 2017-04-28T15:14:06.835746-07:00 INFO kernel: [ 463.331216] [ 4152] 244 4152 79893 1791 69 17067 -1000 apache2 2017-04-28T15:14:06.835747-07:00 INFO kernel: [ 463.331225] [ 4153] 244 4153 88341 13744 97 19688 -1000 apache2 2017-04-28T15:14:06.835748-07:00 INFO kernel: [ 463.331234] [ 4154] 244 4154 88341 10684 97 22740 -1000 apache2 2017-04-28T15:14:06.835749-07:00 INFO kernel: [ 463.331244] [ 4155] 244 4155 79893 1529 69 17336 -1000 apache2 2017-04-28T15:14:06.835751-07:00 INFO kernel: [ 463.331253] [ 4156] 244 4156 79893 3067 69 15800 -1000 apache2 2017-04-28T15:14:06.835752-07:00 INFO kernel: [ 463.331262] [ 4157] 244 4157 88344 4922 97 28478 -1000 apache2 2017-04-28T15:14:06.835753-07:00 INFO kernel: [ 463.331270] [ 4158] 244 4158 88344 2753 97 30657 -1000 apache2 2017-04-28T15:14:06.835755-07:00 INFO kernel: [ 463.331280] [ 4159] 244 4159 88344 31478 97 2265 -1000 apache2 2017-04-28T15:14:06.835756-07:00 INFO kernel: [ 463.331289] [ 4160] 244 4160 88344 3334 97 30089 -1000 apache2 2017-04-28T15:14:06.835757-07:00 INFO kernel: [ 463.331298] [ 4161] 244 4161 79896 1026 69 17849 -1000 apache2 2017-04-28T15:14:06.835758-07:00 INFO kernel: [ 463.331307] [ 4162] 244 4162 79896 3321 69 15643 -1000 apache2 2017-04-28T15:14:06.835759-07:00 INFO kernel: [ 463.331316] [ 4163] 244 4163 88344 3540 97 29966 -1000 apache2 2017-04-28T15:14:06.835760-07:00 INFO kernel: [ 463.331326] [ 4164] 244 4164 79896 3285 69 15677 -1000 apache2 2017-04-28T15:14:06.835762-07:00 INFO kernel: [ 463.331335] [ 4165] 244 4165 79896 3248 69 15688 -1000 apache2 2017-04-28T15:14:06.835763-07:00 INFO kernel: [ 463.331344] [ 4166] 244 4166 88344 4805 97 28641 -1000 apache2 2017-04-28T15:14:06.835764-07:00 INFO kernel: [ 463.331353] [ 4167] 244 4167 88344 3603 97 29815 -1000 apache2 2017-04-28T15:14:06.835765-07:00 INFO kernel: [ 463.331362] [ 4168] 244 4168 79896 3464 69 15496 -1000 apache2 2017-04-28T15:14:06.839667-07:00 INFO kernel: [ 463.332242] [ 4169] 244 4169 88344 2300 97 31119 -1000 apache2 2017-04-28T15:14:06.839681-07:00 INFO kernel: [ 463.332253] [ 4170] 244 4170 79896 3377 69 15633 -1000 apache2 2017-04-28T15:14:06.839683-07:00 INFO kernel: [ 463.332263] [ 4171] 244 4171 79896 698 69 18165 -1000 apache2 2017-04-28T15:14:06.845680-07:00 INFO kernel: [ 463.340535] [ 4172] 244 4172 79896 3526 69 15425 -1000 apache2 2017-04-28T15:14:06.845693-07:00 INFO kernel: [ 463.340550] [ 4173] 244 4173 79896 3315 69 15605 -1000 apache2 2017-04-28T15:14:06.845695-07:00 INFO kernel: [ 463.340560] [ 4204] 244 4204 79896 1966 69 16906 -1000 apache2 2017-04-28T15:14:06.845696-07:00 INFO kernel: [ 463.340569] [ 4205] 244 4205 89304 5791 102 29836 -1000 apache2 2017-04-28T15:14:06.845697-07:00 INFO kernel: [ 463.340578] [ 4206] 244 4206 88344 7222 97 26279 -1000 apache2 2017-04-28T15:14:06.845699-07:00 INFO kernel: [ 463.340587] [ 4207] 244 4207 79896 1810 69 17052 -1000 apache2 2017-04-28T15:14:06.845700-07:00 INFO kernel: [ 463.340613] [ 4208] 244 4208 79896 1213 69 17646 -1000 apache2 2017-04-28T15:14:06.845702-07:00 INFO kernel: [ 463.340622] [ 4209] 244 4209 88344 23889 97 9624 -1000 apache2 2017-04-28T15:14:06.845703-07:00 INFO kernel: [ 463.340632] [ 4267] 244 4267 88344 4631 97 28795 -1000 apache2 2017-04-28T15:14:06.845704-07:00 INFO kernel: [ 463.340930] [ 4268] 244 4268 88408 4081 97 29614 -1000 apache2 2017-04-28T15:14:06.845705-07:00 INFO kernel: [ 463.340940] [ 4269] 244 4269 88408 3964 97 29773 -1000 apache2 2017-04-28T15:14:06.845706-07:00 INFO kernel: [ 463.340949] [ 4270] 246 4270 54013 406 26 909 -1000 apache2 2017-04-28T15:14:06.845708-07:00 INFO kernel: [ 463.340958] [ 4271] 244 4271 74503 439 36 940 -1000 apache2 2017-04-28T15:14:06.845709-07:00 INFO kernel: [ 463.341252] [ 4433] 0 4433 3909 123 12 87 -1000 sudo 2017-04-28T15:14:06.845710-07:00 INFO kernel: [ 463.341262] [ 4436] 246 4436 75387 4202 50 9425 -1000 monitor_db.py 2017-04-28T15:14:06.845711-07:00 INFO kernel: [ 463.341272] [ 4437] 0 4437 3909 123 11 88 -1000 sudo 2017-04-28T15:14:06.850714-07:00 INFO kernel: [ 463.345391] [ 4438] 0 4438 3909 123 13 87 -1000 sudo 2017-04-28T15:14:06.850725-07:00 INFO kernel: [ 463.345402] [ 4439] 246 4439 14579 2427 32 6940 -1000 gs_offloader.py 2017-04-28T15:14:06.850727-07:00 INFO kernel: [ 463.345410] [ 4440] 246 4440 14581 875 32 8375 -1000 gs_offloader.py 2017-04-28T15:14:06.850728-07:00 INFO kernel: [ 463.345418] [ 4456] 246 4456 69215 778 36 8082 -1000 gs_offloader.py 2017-04-28T15:14:06.850729-07:00 INFO kernel: [ 463.345426] [ 4457] 246 4457 87648 609 38 8260 -1000 gs_offloader.py 2017-04-28T15:14:06.850730-07:00 INFO kernel: [ 463.345435] [ 4461] 246 4461 15526 1220 34 7840 -1000 gs_offloader.py 2017-04-28T15:14:06.850731-07:00 INFO kernel: [ 463.345443] [ 4470] 246 4470 15528 1306 34 7788 -1000 gs_offloader.py 2017-04-28T15:14:06.850732-07:00 INFO kernel: [ 463.345451] [ 4475] 246 4475 54732 1026 41 10516 -1000 monitor_db.py 2017-04-28T15:14:06.850734-07:00 INFO kernel: [ 463.345459] [ 4486] 246 4486 56325 2540 45 9719 -1000 monitor_db.py 2017-04-28T15:14:06.850735-07:00 INFO kernel: [ 463.345469] [ 5335] 246 5335 44249 2596 57 15676 -1000 autoserv 2017-04-28T15:14:06.850736-07:00 INFO kernel: [ 463.345478] [ 5376] 246 5376 16531 87 35 10444 -1000 autoserv 2017-04-28T15:14:06.850737-07:00 INFO kernel: [ 463.345487] [ 5377] 246 5377 16531 87 35 10444 -1000 autoserv 2017-04-28T15:14:06.850738-07:00 INFO kernel: [ 463.345497] [ 704] 0 704 2361 92 8 38 -1000 sleep 2017-04-28T15:14:06.850739-07:00 INFO kernel: [ 463.345506] [ 706] 0 706 2361 92 9 37 -1000 sleep 2017-04-28T15:14:06.850740-07:00 INFO kernel: [ 463.345515] [ 710] 0 710 2361 92 9 37 -1000 sleep 2017-04-28T15:14:06.850741-07:00 INFO kernel: [ 463.345524] [ 735] 0 735 2361 92 9 38 -1000 sleep 2017-04-28T15:14:06.850742-07:00 INFO kernel: [ 463.345534] [ 9371] 0 9371 1081 112 6 23 -1000 sh 2017-04-28T15:14:06.850743-07:00 INFO kernel: [ 463.345543] [ 9373] 0 9373 2361 92 9 37 -1000 sleep 2017-04-28T15:14:06.850745-07:00 INFO kernel: [ 463.345552] [11036] 0 11036 4687 236 14 145 -1000 sshd 2017-04-28T15:14:06.850746-07:00 INFO kernel: [ 463.345560] [11441] 246 11441 4687 110 13 123 -1000 sshd 2017-04-28T15:14:06.850747-07:00 INFO kernel: [ 463.345569] [11445] 246 11445 1927 110 8 94 -1000 bash 2017-04-28T15:14:06.850748-07:00 INFO kernel: [ 463.345577] [13725] 246 13725 2394 116 9 41 -1000 tail 2017-04-28T15:14:06.850749-07:00 INFO kernel: [ 463.345586] [14708] 244 14708 6923 722 18 779 -1000 apache2 2017-04-28T15:14:06.850750-07:00 INFO kernel: [ 463.345595] [14718] 244 14718 6923 716 18 752 -1000 apache2 2017-04-28T15:14:06.850751-07:00 INFO kernel: [ 463.345604] [14892] 246 14892 51437 493 34 8477 -1000 gs_offloader.py 2017-04-28T15:14:06.850753-07:00 INFO kernel: [ 463.345613] [14897] 246 14897 33004 535 33 8439 -1000 gs_offloader.py 2017-04-28T15:14:06.850754-07:00 INFO kernel: [ 463.345623] [14903] 246 14903 15165 1990 33 7695 -1000 gs_offloader.py 2017-04-28T15:14:06.850755-07:00 INFO kernel: [ 463.345633] [15524] 244 15524 6923 907 18 565 -1000 apache2 2017-04-28T15:14:06.850756-07:00 INFO kernel: [ 463.345642] [18162] 244 18162 6923 1095 18 407 -1000 apache2 2017-04-28T15:14:06.850757-07:00 INFO kernel: [ 463.345651] [19292] 244 19292 6923 1393 18 183 -1000 apache2 2017-04-28T15:14:06.850758-07:00 INFO kernel: [ 463.345660] [20092] 246 20092 13546 8785 31 0 -1000 autoserv 2017-04-28T15:14:06.850759-07:00 INFO kernel: [ 463.345668] [20097] 246 20097 48947 8211 32 0 -1000 gsutil 2017-04-28T15:14:06.850761-07:00 INFO kernel: [ 463.345675] [20113] 246 20113 33002 2214 33 6832 -1000 gs_offloader.py 2017-04-28T15:14:06.850762-07:00 INFO kernel: [ 463.345683] [20118] 246 20118 33002 2205 33 6831 -1000 gs_offloader.py 2017-04-28T15:14:06.850763-07:00 INFO kernel: [ 463.345692] [20124] 246 20124 14579 2182 32 6853 -1000 gs_offloader.py 2017-04-28T15:14:06.850764-07:00 INFO kernel: [ 463.345701] [20134] 246 20134 138478 6797 39 0 -1000 gsutil 2017-04-28T15:14:06.850765-07:00 INFO kernel: [ 463.345710] [20144] 246 20144 362051 7437 64 0 -1000 gsutil 2017-04-28T15:14:06.850767-07:00 INFO kernel: [ 463.345719] [20296] 0 20296 2361 132 9 0 -1000 sleep 2017-04-28T15:14:06.850768-07:00 INFO kernel: [ 463.345728] [20356] 246 20356 196444 7342 47 0 -1000 gsutil 2017-04-28T15:14:06.850768-07:00 INFO kernel: [ 463.345737] [20357] 246 20357 196444 7329 47 0 -1000 gsutil 2017-04-28T15:14:06.850769-07:00 INFO kernel: [ 463.345745] [20358] 246 20358 196636 7722 47 0 -1000 gsutil 2017-04-28T15:14:06.850770-07:00 INFO kernel: [ 463.345754] [20360] 246 20360 196573 7542 47 0 -1000 gsutil 2017-04-28T15:14:06.850771-07:00 ERR kernel: [ 463.345765] Out of memory: Kill process 1022 (chrome) score 311 or sacrifice child 2017-04-28T15:14:06.850772-07:00 ERR kernel: [ 463.345775] Killed process 1022 (chrome) total-vm:761312kB, anon-rss:36000kB, file-rss:4068kB 2017-04-28T15:14:07.049292-07:00 INFO session_manager[722]: [INFO:child_exit_handler.cc(73)] 874 is not a managed job. 2017-04-28T15:14:07.075929-07:00 INFO session_manager[722]: [INFO:child_exit_handler.cc(77)] Handling 747 exit. 2017-04-28T15:14:07.075945-07:00 ERR session_manager[722]: [ERROR:child_exit_handler.cc(79)] Exited with exit code 1 2017-04-28T15:14:07.076084-07:00 INFO session_manager[722]: [INFO:session_manager_service.cc(274)] Exiting process is chrome. 2017-04-28T15:14:07.076100-07:00 INFO session_manager[722]: [INFO:browser_job.cc(149)] Terminating process group: Ensuring browser processes are gone. 2017-04-28T15:14:07.076114-07:00 INFO session_manager[722]: [INFO:system_utils_impl.cc(110)] Sending 9 to -747 as 1000 2017-04-28T15:14:07.077250-07:00 INFO session_manager[722]: [INFO:browser_job.cc(173)] Cleaned up child 747 2017-04-28T15:14:07.079596-07:00 INFO session_manager[722]: [INFO:browser_job.cc(140)] Running child /opt/google/chrome/chrome --ui-prioritize-in-gpu-process --use-gl=egl --gpu-sandbox-failures-fatal=yes --enable-logging --log-level=1 --use-cras --enable-wayland-server --user-data-dir=/home/chronos --max-unused-resource-memory-usage-percentage=5 --system-developer-mode --login-profile=user --aura-legacy-power-button --disable-demo-mode --enable-prefixed-encrypted-media --enable-consumer-kiosk --enterprise-enrollment-initial-modulus=15 --enterprise-enrollment-modulus-limit=19 --login-manager --vmodule=*chromeos/login/*=1,auto_enrollment_controller=1,*plugin*=2,*zygote*=1,*/ui/ozone/*=1,*/ui/display/manager/chromeos/*=1,power_button_observer=2,webui_login_view=2,lock_state_controller=2,webui_screen_locker=2,screen_locker=2 2017-04-28T15:14:07.082580-07:00 INFO session_manager[722]: [INFO:session_manager_service.cc(225)] Browser is 20621
,
Apr 28 2017
Any logs to the failing platform_CrosDisksDBus test? /var/log/messages wasn't very helpful.
,
Apr 28 2017
From the first attempt nothing useful http://100.107.3.109/results/25-moblab/192.168.231.100/ From the second run - nothing too unusual - although I think the issue is parsing the result http://100.107.3.109/results/74-moblab/192.168.231.100/debug/
,
Apr 28 2017
> http://100.107.3.109/results/74-moblab/192.168.231.100/debug/ Is this the log for a failed run? Don't see anything fatal there. Even if there were any parsing errors, it was autotest failed parsing test results, not a particular test failed parsing dbus outputs. The CL you posted was about the latter.
,
Apr 28 2017
I will keep trying to figure out the issue - just wondered for context I am pretty certain that one of these CL's broke moblab 1a25ec40 474704 Tue Apr 11 02:56:50 2017 kcwu@chromium.org make video unittests support setting test filter from command line 087e008d 477171 708679 Thu Apr 13 17:40:10 2017 jrbarnette@chromium.org [autotest] Increase WSGI process limit to 65. 2b1a26af 465546 Fri Mar 31 18:51:41 2017 wonderfly@google.com dbus_send: correctly parse new dbus message header cec26f75 452950 672348 Fri Mar 10 00:39:09 2017 ayatane@chromium.org [autotest] Extract _ComposedPredicate 67c9af88 452949 672348 Fri Mar 10 00:33:24 2017 ayatane@chromium.org [autotest] Use parameters instead of implicit attributes 2887e33e 452948 672348 Fri Mar 10 00:30:36 2017 ayatane@chromium.org [autotest] Fix docstring typo 7f43ef9c 452947 672348 Fri Mar 10 00:29:48 2017 ayatane@chromium.org [autotest] Move test initialization to Suite subclass 25bb1c62 452946 672348 Fri Mar 10 00:27:00 2017 ayatane@chromium.org [autotest] Move extra constructors to Suite subclass 3b1d4e59 452945 672348 Fri Mar 10 00:23:06 2017 ayatane@chromium.org [autotest] Duplicate constructor in Suite subclass 4b5a24f9 452944 672348 Fri Mar 10 00:01:35 2017 ayatane@chromium.org [autotest] Split Suite into two classes Which are the CL's from 9459.0.0 to 9460.0.0 https://cros-goldeneye.corp.google.com/moblab/console/listBuild?milestone=59#%2F And the moblab box crashed and rebooted twice just when it was running the platform_CrosDisksDBus test, not conclusive proof, I am still working on it but I try to cc in owners of likely problem CL's since more often than not they know of a future fix that did not make it in to the branch. I will work on this more Monday and post any more conclusive logs I can get into the bug.
,
May 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/34a496f81cd7781705533d0ab25803455b745468 commit 34a496f81cd7781705533d0ab25803455b745468 Author: Keith Haddow <haddowk@chromium.org> Date: Thu May 04 17:45:11 2017 [autotest] Revert the change to apache process on M59 branch. THe change the lab made breaks moblab, reverting on the 59 branch as this does not affect the lab and will unblock moblab until a fix can be checked in to master. BUG= chromium:716215 TEST=manual testng on moblab Change-Id: I344941d0157113631ee863dd6d4e84f352c7e3c0 Reviewed-on: https://chromium-review.googlesource.com/495870 Reviewed-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Michael Tang <ntang@chromium.org> Commit-Queue: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> [modify] https://crrev.com/34a496f81cd7781705533d0ab25803455b745468/apache/conf/django-directives
,
May 4 2017
,
May 10 2017
,
Jan 22 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by haddowk@chromium.org
, Apr 28 2017From the logs run suite is crashing 04/28 02:52:44.659 DEBUG| base_utils:0185| Running 'rsync -L --timeout=1800 --rsh='/usr/bin/ssh -a -x -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22' -az --no-o --no-g root@chromeos2-row2-rack8-host7:"/usr/local/autotest/logs" "/usr/local/autotest/results/114461536-chromeos-test/moblab_RunSuite/results/moblab_logs"' 04/28 02:52:45.830 DEBUG| test:0389| Test failed due to command execution error * Command: /usr/bin/ssh -a -x -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos2-row2-rack8-host7 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::_call_run_once|run_once|run_as_moblab] -> ssh_run(su - moblab -c '/usr/local/autotest/site_utils/run_suite.py --pool='' --board=cyan --build=cyan-release/R55-8872.67.0 --suite_name=smoke')\";fi; su - moblab -c '/usr/local/autotest/site_utils/run_suite.py --pool='' --board=cyan --build=cyan-release/R55-8872.67.0 --suite_name=smoke'" Exit status: 255 Duration: 3646.11512089 stdout: [?25h[?0c stderr: ERROR:root:Could not import tzlocal. Autotest instance: localhost 04-28-2017 [01:51:45] Submitted create_suite_job rpc 04-28-2017 [01:51:53] Created suite job: http://localhost/afe/#tab_id=view_job&object_id=1 @@@STEP_LINK@Link to suite@http://localhost/afe/#tab_id=view_job&object_id=1@@@ The suite job has another 23:29:58.881610 till timeout. 04-28-2017 [02:36:29] Suite job is finished. 04-28-2017 [02:36:29] Start collectint test results and dump them to json. Write failed: Broken pipe. Exception log follows the after_iteration_hooks. 04/28 02:52:45.831 DEBUG| test:0392| starting after_iteration_hooks