New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 600452 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Oct 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

veyron_speedy-paladin: DUT rebooted during the test run

Project Member Reported by aaboagye@chromium.org, Apr 4 2016

Issue description

https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_speedy-paladin/builds/1541

On the most recent run, it appears that the DUT rebooted during a test. This test was logging_UserCrash.

Error snippet:
04/04 07:09:26.592 DEBUG|          ssh_host:0153| Running (ssh) 'nohup /usr/local/autotest/bin/autotestd /tmp/autoserv-6OY4D8 -H autoserv --verbose --hostname=chromeos4-row4-rack11-host10 --user=chromeos-test /usr/local/autotest/control.autoserv >/dev/null 2>/dev/null &'
04/04 07:09:26.934 DEBUG|          ssh_host:0153| Running (ssh) '/usr/local/autotest/bin/autotestd_monitor /tmp/autoserv-6OY4D8 0 0'
04/04 07:09:27.293 DEBUG|     site_autotest:0188| Entered autotestd_monitor.
04/04 07:09:27.294 INFO |          autotest:1113| Entered autotestd_monitor.
04/04 07:09:27.294 DEBUG|     site_autotest:0188| Finished launching tail subprocesses.
04/04 07:09:27.294 INFO |          autotest:1113| Finished launching tail subprocesses.
04/04 07:09:27.295 DEBUG|     site_autotest:0188| Finished waiting on autotestd to start.
04/04 07:09:27.295 INFO |          autotest:1113| Finished waiting on autotestd to start.
04/04 07:09:28.530 DEBUG|     site_autotest:0188| AUTOTEST_STATUS::START	----	----	timestamp=1459778967	localtime=Apr 04 07:09:27	
04/04 07:09:28.532 INFO |        server_job:0128| START	----	----	timestamp=1459778967	localtime=Apr 04 07:09:27	
04/04 07:09:28.676 DEBUG|     site_autotest:0188| AUTOTEST_STATUS::	START	logging_UserCrash	logging_UserCrash	timestamp=1459778967	localtime=Apr 04 07:09:27	
04/04 07:09:28.676 INFO |        server_job:0128| 	START	logging_UserCrash	logging_UserCrash	timestamp=1459778967	localtime=Apr 04 07:09:27	
04/04 07:24:40.060 DEBUG|          autotest:0777| Result exit status is 255.
04/04 07:24:40.061 DEBUG|        base_utils:0177| Running 'ping chromeos4-row4-rack11-host10 -w1 -c1'
04/04 07:24:40.184 DEBUG|        base_utils:0268| [stdout] PING chromeos4-row4-rack11-host10.cros.corp.google.com (100.107.190.2) 56(84) bytes of data.
04/04 07:24:40.185 DEBUG|        base_utils:0268| [stdout] 64 bytes from 100.107.190.2: icmp_seq=1 ttl=60 time=7.88 ms
04/04 07:24:40.185 DEBUG|        base_utils:0268| [stdout] 
04/04 07:24:40.185 DEBUG|        base_utils:0268| [stdout] --- chromeos4-row4-rack11-host10.cros.corp.google.com ping statistics ---
04/04 07:24:40.185 DEBUG|        base_utils:0268| [stdout] 1 packets transmitted, 1 received, 0% packet loss, time 0ms
04/04 07:24:40.185 DEBUG|        base_utils:0268| [stdout] rtt min/avg/max/mdev = 7.883/7.883/7.883/0.000 ms
04/04 07:24:40.186 DEBUG|          ssh_host:0153| Running (ssh) 'if [ -f '/proc/sys/kernel/random/boot_id' ]; then cat '/proc/sys/kernel/random/boot_id'; else echo 'no boot_id available'; fi'
04/04 07:24:40.186 INFO |      abstract_ssh:0735| Master ssh connection to chromeos4-row4-rack11-host10 is down.
04/04 07:24:40.186 DEBUG|      abstract_ssh:0696| Nuking master_ssh_job.
04/04 07:24:40.187 DEBUG|      abstract_ssh:0702| Cleaning master_ssh_tempdir.
04/04 07:24:40.187 INFO |      abstract_ssh:0749| Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_x8kPMAssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos4-row4-rack11-host10'
04/04 07:24:40.188 DEBUG|        base_utils:0177| Running '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_x8kPMAssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos4-row4-rack11-host10'
04/04 07:24:41.608 DEBUG|        base_utils:0268| [stdout] 8cf15219-f927-4760-95ed-1f7941c21aa9
04/04 07:24:41.611 INFO |        server_job:0128| 		FAIL	----	----	timestamp=1459779881	localtime=Apr 04 07:24:41	Autotest client terminated unexpectedly: DUT rebooted during the test run.
  
04/04 07:24:41.612 INFO |        server_job:0128| 	END FAIL	----	----	timestamp=1459779881	localtime=Apr 04 07:24:41	
04/04 07:24:41.612 INFO |        server_job:0128| END GOOD	----	----	timestamp=1459779881	localtime=Apr 04 07:24:41	
04/04 07:24:41.613 DEBUG|          ssh_host:0153| Running (ssh) 'true'
04/04 07:24:41.930 DEBUG|      abstract_ssh:0542| Host chromeos4-row4-rack11-host10 is now up

Assigning to current infra deputy.
 
Labels: -Infra-Labs
It helps to list the bot name, as well as builder name.

This is build234-m2.  Host seems fine.  I assume DUT is your host in "row4", "rack11", "host10"?  (Might I suggest you use an easier naming scheme going forward?  cros4-4k10)

Please re-apply label Infra-Labs if this is actually related to build234-m2.
Cc: d...@chromium.org
Labels: Infra-Labs
@dnj +Infra-Labs

veyron_speedy-paladin has this error again, 

build234-m2

chromeos4-row4-rack10-host13 
log: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/58975887-chromeos-test/
Labels: -Infra-Labs
Infra-Labs can only fix the build host, build234-m2.  We can't access "chromeos4-row4-rack10-host13".
Cc: jrbarnette@chromium.org
Seeing this issue on lumpy board. Lumpy-release/R50-7978.64.0/bvt-cq/logging_UserCrash. Details at

https://wmatrix.googleplex.com/testrun/bvt-cq?test_ids=292853105 . Who can take a look at this issue?
Status: WontFix (was: Untriaged)
seems no update for a long time. Mark it as won't fix. Re-open it if the bug happens again.

Sign in to add a comment