New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 700781 link

Starred by 1 user
Status: Archived
Owner: ----
Closed: Jun 2018
Cc:
kinaba@chromium.org
gwendal@chromium.org
█ choonc@google.com
katie@chromium.org
rohi...@chromium.org
dhadd...@chromium.org
harpreet@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Still losing CTS/GTS jobs due to provision errors

Project Member Reported by ihf@chromium.org, Mar 13 2017 
[mostly fyi for tracking]

Still losing CTS/GTS jobs due to provision errors
terra-release/R59-9355.0.0/arc-gts/cheets_GTS.4.1_r1.GtsCastHostTestCases
https://ubercautotest.corp.google.com/afe/#tab_id=view_job&object_id=105963853

START	----	provision	timestamp=1489193627	localtime=Mar 10 16:53:47	
	GOOD	----	verify.ssh	timestamp=1489193630	localtime=Mar 10 16:53:50	
	GOOD	----	verify.update	timestamp=1489193633	localtime=Mar 10 16:53:53	
	GOOD	----	verify.brd_config	timestamp=1489193634	localtime=Mar 10 16:53:54	
	GOOD	----	verify.ser_config	timestamp=1489193634	localtime=Mar 10 16:53:54	
	GOOD	----	verify.job	timestamp=1489193635	localtime=Mar 10 16:53:55	
	GOOD	----	verify.servod	timestamp=1489193640	localtime=Mar 10 16:54:00	
	GOOD	----	verify.pwr_button	timestamp=1489193640	localtime=Mar 10 16:54:00	
	GOOD	----	verify.lid_open	timestamp=1489193640	localtime=Mar 10 16:54:00	
	GOOD	----	verify.PASS	timestamp=1489193640	localtime=Mar 10 16:54:00	
	START	provision_AutoUpdate	provision_AutoUpdate	timestamp=1489193640	localtime=Mar 10 16:54:00	
		START	----	----	timestamp=1489193660	localtime=Mar 10 16:54:20	
			GOOD	----	sysinfo.before	timestamp=1489193660	localtime=Mar 10 16:54:20	
		END GOOD	----	----	timestamp=1489193660	localtime=Mar 10 16:54:20	
		GOOD	provision_AutoUpdate	provision_AutoUpdate	timestamp=1489194313	localtime=Mar 10 17:05:13	completed successfully
	END GOOD	provision_AutoUpdate	provision_AutoUpdate	timestamp=1489194313	localtime=Mar 10 17:05:13	
	GOOD	----	verify.ssh	timestamp=1489194314	localtime=Mar 10 17:05:14	
	GOOD	----	verify.devmode	timestamp=1489194315	localtime=Mar 10 17:05:15	
	GOOD	----	verify.power	timestamp=1489194315	localtime=Mar 10 17:05:15	
	FAIL	----	verify.ext4	timestamp=1489194315	localtime=Mar 10 17:05:15	Saw file system error: [    5.676788] EXT4-fs error (device mmcblk0p1): ext4_lookup:1591: inode #133760: comm find: deleted inode referenced: 133770
	GOOD	----	verify.writable	timestamp=1489194315	localtime=Mar 10 17:05:15	
	GOOD	----	verify.tpm	timestamp=1489194316	localtime=Mar 10 17:05:16	
	GOOD	----	verify.good_au	timestamp=1489194316	localtime=Mar 10 17:05:16	
	GOOD	----	verify.fwstatus	timestamp=1489194316	localtime=Mar 10 17:05:16	
	GOOD	----	verify.rwfw	timestamp=1489194317	localtime=Mar 10 17:05:17	
	GOOD	----	verify.python	timestamp=1489194317	localtime=Mar 10 17:05:17	
	GOOD	----	verify.cros	timestamp=1489194321	localtime=Mar 10 17:05:21	
END FAIL	----	provision	timestamp=1489194321	localtime=Mar 10 17:05:21	
INFO	----	----	timestamp=1489194321	job_abort_reason=	localtime=Mar 10 17:05:21

Comment 1 by mzhuo@chromium.org, Nov 19 2017

Cc: katie@chromium.org choonc@google.com harpreet@chromium.org frankhu@chromium.org
Labels: -Pri-3 M-62 Proj-Bluestreak Pri-2
Hit the similar issue on Guado running 9901.77.1

mzhuo@minzhuo2:~/chromiumos$ ssh root@100.107.146.3

localhost ~ # 
localhost ~ # tail -f /var/log/messages &
[1] 4064
localhost ~ # 2017-11-18T17:55:52.102423-08:00 CRIT kernel: [   53.681085] EXT4-fs error (device dm-1): ext4_lookup:1593: inode #24120: comm find: deleted inode referenced: 24272
2017-11-18T17:55:52.102427-08:00 CRIT kernel: [   53.681144] EXT4-fs error (device dm-1): ext4_lookup:1593: inode #24120: comm find: deleted inode referenced: 24264
2017-11-18T17:55:52.102428-08:00 CRIT kernel: [   53.681239] EXT4-fs error (device dm-1): ext4_lookup:1593: inode #24120: comm find: deleted inode referenced: 24259
2017-11-18T17:55:52.102429-08:00 CRIT kernel: [   53.681309] EXT4-fs error (device dm-1): ext4_lookup:1593: inode #24120: comm find: deleted inode referenced: 24258
2017-11-18T17:55:52.905430-08:00 INFO kernel: [   54.484650] tpm_tis tpm_tis: command 0x65 (size 22) returned code 0x0
2017-11-18T17:56:00.523432-08:00 WARNING kernel: [   62.108913] EXT4-fs error: 16 callbacks suppressed
2017-11-18T17:56:00.523444-08:00 CRIT kernel: [   62.108920] EXT4-fs error (device dm-1): ext4_validate_block_bitmap:381: comm kworker/u8:4: bg 1: block 32873: invalid block bitmap
2017-11-18T17:56:11.614762-08:00 INFO sshd[3628]: Accepted publickey for root from 100.123.202.88 port 50156 ssh2: RSA SHA256:Fp1qWjFLyK1cTpiI5rdk7iEJwoK9lcnYAgbQtGC3jzU
2017-11-18T17:56:14.905674-08:00 ERR chrome[950]: [950:950:1118/175614.905595:INFO:status_uploader.cc(222)] Starting status upload: have_device_status = 1
2017-11-18T17:56:15.144068-08:00 ERR chrome[950]: [950:950:1118/175615.143995:INFO:status_uploader.cc(235)] Status upload successful
2017-11-18T17:56:36.910066-08:00 INFO sshd[4067]: Accepted publickey for root from 100.123.202.88 port 50186 ssh2: RSA SHA256:Fp1qWjFLyK1cTpiI5rdk7iEJwoK9lcnYAgbQtGC3jzU

localhost ~ # localhost ~ # cat /etc/lsb-release 
CHROMEOS_RELEASE_APPID={8AA6D9AC-6EBC-4288-A615-171F56F66B4E}
CHROMEOS_BOARD_APPID={8AA6D9AC-6EBC-4288-A615-171F56F66B4E}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
DEVICETYPE=CHROMEBOX
CHROMEOS_RELEASE_BUILDER_PATH=guado-release/R62-9901.77.1
GOOGLE_RELEASE=9901.77.1
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_BOARD=guado
CHROMEOS_RELEASE_BUILD_NUMBER=9901
CHROMEOS_RELEASE_BRANCH_NUMBER=77
CHROMEOS_RELEASE_CHROME_MILESTONE=62
CHROMEOS_RELEASE_PATCH_NUMBER=1
CHROMEOS_RELEASE_TRACK=testimage-channel
CHROMEOS_RELEASE_DESCRIPTION=9901.77.1 (Official Build) dev-channel guado test
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_VERSION=9901.77.1
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
localhost ~ # lsof /dev/video^C
localhost ~ # tail -8 /var/log/eventlog.txt 
172 | 2017-11-18 17:40:26 | Wake Source | GPIO | 27
173 | 2017-11-18 17:40:27 | Chrome OS Developer Mode
174 | 2017-11-18 17:54:27 | Kernel Event | Clean Shutdown
175 | 2017-11-18 17:54:27 | System boot | 1964
176 | 2017-11-18 17:54:27 | System Reset
177 | 2017-11-18 17:54:27 | Log area cleared | 1032
178 | 2017-11-18 17:54:27 | Wake Source | GPIO | 27
179 | 2017-11-18 17:54:28 | Chrome OS Developer Mode
localhost ~ # uptime
 18:17:18 up 22 min,  0 users,  load average: 2.94, 2.71, 2.03
localhost ~ #
debug-logs_20171118-180420.tgz
8.3 MB Download

Comment 2 by jbudorick@chromium.org, Nov 19 2017

Components: -Infra>Client>Chrome Infra>Client>ChromeOS

Comment 3 by mzhuo@chromium.org, Nov 20 2017

Cc: -frankhu@chromium.org gwendal@chromium.org

Comment 4 by ihf@chromium.org, Nov 30 2017 

So, Guado doesn't run CTS/GTS. And not clear if this was a provision problem. The common issue posted is on the filesystem corruption. It is probably best to refile this against "OS>Kernel>IntStorage" and provide some data on how to repro it.

Comment 5 by cindyb@chromium.org, May 31 2018 

Hi, this bug has not been updated recently and remains untriaged. Please acknowledge the bug and provide status within two weeks (6/8/2018), or the bug will be closed. Thank you.

Comment 6 by jrbarnette@chromium.org, Jun 1 2018

Status: Archived (was: Untriaged)
►

Sign in to add a comment