kefka-release HWTests are very flaky |
|||
Issue descriptionErrors aren't always the same but seem to happen during provisioning: https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1967 " FAIL: Saw file system error: [ 1.767018] EXT4-fs error (device mmcblk0p1): ext4_lookup:1590: inode #147000: comm rm: deleted inode referenced: 260848, completed successfully " Same error on both https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1966 https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1968 " provision: FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row4-rack8-host11: 0) SSHConnectionError: None, 1) SSHConnectionError: ssh: connect to host 100.115.227.30 port 22: Connection timed out "
,
Mar 2 2018
I have essentially no kernel logs from these repair jobs for chromeos2-row4-rack8-host11: 2018-02-09-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-09 21:40:28 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/13343-repair/ 2018-02-10-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-10 22:07:30 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/26275-repair/ 2018-02-11-bvt.txt:chromeos2-row4-rack8-host11 OK 2018-02-11 21:06:58 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/35415-repair/ 2018-02-12-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-12 22:02:05 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/46949-repair/ 2018-02-13-bvt.txt:chromeos2-row4-rack8-host11 OK 2018-02-13 21:51:16 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/55640-repair/ 2018-02-14-bvt.txt:chromeos2-row4-rack8-host11 OK 2018-02-14 16:51:04 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/62852-reset/ 2018-02-15-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-15 22:38:57 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/72521-repair/ 2018-02-16-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-16 22:21:09 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/81226-repair/ 2018-02-17-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-17 21:52:16 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/89688-repair/ 2018-02-18-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-18 22:46:45 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/99736-repair/ 2018-02-19-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-19 22:01:47 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/110320-repair/ 2018-02-20-bvt.txt:chromeos2-row4-rack8-host11 OK 2018-02-20 22:56:09 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/118914-repair/ 2018-02-21-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-21 22:15:56 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/126014-repair/ 2018-02-22-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-22 22:46:07 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/132472-repair/ 2018-02-23-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-23 22:30:26 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/138229-repair/ 2018-02-24-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-24 21:40:08 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/147376-repair/ 2018-02-25-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-25 22:41:30 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/157841-repair/ 2018-02-26-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-26 21:48:50 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/168989-repair/ 2018-02-27-bvt.txt:chromeos2-row4-rack8-host11 OK 2018-02-27 22:02:05 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/180601-reset/ 2018-02-28-bvt.txt:chromeos2-row4-rack8-host11 NO 2018-02-28 23:04:22 http://cautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row4-rack8-host11/190571-repair/ No logs means there is likely something really wrong with the HW given we are getting logs from other machines. Same story for chromeos2-row4-rack8-host18 Given lack of data, I believe some TLC from a labtech should be the next step OR some one has to tell me where to find the system or kernel logs that correspond to those repair jobs.
,
Mar 2 2018
I'm able to get these logs from chromeos2-row4-rack8-host18 /var/log/messages before I'm disconnected from ssh: 2018-03-02T10:48:51.681051-08:00 INFO kernel: [ 448.157704] usb 1-2.2: USB disconnect, device number 6 2018-03-02T10:48:51.693735-08:00 ERR kernel: [ 448.169093] Aborting journal on device sda1-8. 2018-03-02T10:48:51.693775-08:00 ERR kernel: [ 448.169128] JBD2: Error -5 detected when updating journal superblock for sda1-8. 2018-03-02T10:48:55.361720-08:00 WARNING kernel: [ 451.839326] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.361753-08:00 WARNING kernel: [ 451.839371] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.361757-08:00 WARNING kernel: [ 451.839402] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.361759-08:00 WARNING kernel: [ 451.839430] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.361761-08:00 WARNING kernel: [ 451.839461] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.362735-08:00 WARNING kernel: [ 451.839595] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.362765-08:00 WARNING kernel: [ 451.839623] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.362768-08:00 WARNING kernel: [ 451.839771] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.364735-08:00 WARNING kernel: [ 451.841844] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0) 2018-03-02T10:48:55.364766-08:00 WARNING kernel: [ 451.841978] EXT4-fs warning (device sda1): __ext4_read_dirblock:734: error -5 reading directory block (ino 25925, block 0)
,
Mar 6 2018
,
Jun 8 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by davidri...@chromium.org
, Mar 1 2018