Guado VM test flake in 'crossystem hwid' |
|||
Issue descriptionKilled master-paladin build: https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/18576 Logs: https://logs.chromium.org/v/?s=chromiumos%2Fbb%2Fchromiumos%2Fmoblab-generic-vm-paladin%2F1567%2F%2B%2Frecipes%2Fsteps%2FMoblabVMTest%2F0%2Fstdout Relevant error: /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.533 ERROR| utils:0282| [stderr] INFO:root:checking label HWIDLabel /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.535 ERROR| utils:0282| [stderr] DEBUG:root:Running (ssh) 'crossystem hwid' from 'get_labels|get|generate_labels|run_output|run|run_very_slowly' /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.552 ERROR| utils:0282| [stderr] ERROR:root:error getting label HWIDLabel. /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.553 ERROR| utils:0282| [stderr] Traceback (most recent call last): /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.553 ERROR| utils:0282| [stderr] File "/usr/local/autotest/server/hosts/base_label.py", line 202, in get_labels /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.553 ERROR| utils:0282| [stderr] labels.extend(label.get(host)) /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.554 ERROR| utils:0282| [stderr] File "/usr/local/autotest/server/hosts/base_label.py", line 82, in get /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.554 ERROR| utils:0282| [stderr] return self.generate_labels(host) /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.554 ERROR| utils:0282| [stderr] File "/usr/local/autotest/server/hosts/cros_label.py", line 558, in generate_labels /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.554 ERROR| utils:0282| [stderr] hwid = host.run_output('crossystem hwid').strip() /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.555 ERROR| utils:0282| [stderr] File "/usr/local/autotest/client/common_lib/hosts/base_classes.py", line 131, in run_output /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.555 ERROR| utils:0282| [stderr] return self.run(command, *args, **dargs).stdout.rstrip() /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.555 ERROR| utils:0282| [stderr] File "/usr/local/autotest/server/hosts/ssh_host.py", line 323, in run /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.556 ERROR| utils:0282| [stderr] return self.run_very_slowly(*args, **kwargs) /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.556 ERROR| utils:0282| [stderr] File "/usr/local/autotest/server/hosts/ssh_host.py", line 312, in run_very_slowly /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.556 ERROR| utils:0282| [stderr] ssh_failure_retry_ok) /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.556 ERROR| utils:0282| [stderr] File "/usr/local/autotest/server/hosts/ssh_host.py", line 262, in _run /tmp/cbuildbotoLNvNS/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite 05/11 14:43:40.557 ERROR| utils:0282| [stderr] raise error.AutoservRunError("command execution error", result)
,
May 12 2018
The trackback on the VM getting a HWID has always been there and should not be the cause of a failure- looking.
,
May 12 2018
The issue was from this log https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/moblab-generic-vm-paladin/R68-10669.0.0-rc2/moblab_vm_test_results/results-1-moblab_DummyServerNoSspSuite/sysinfo/mnt/moblab/results/2-moblab/192.168.231.100/lucifer/ panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x596bae90a6fb] goroutine 1 [running]: cloud.google.com/go/bigquery.(*Client).Dataset(...) /build/moblab-generic-vm/usr/lib/gopath/src/cloud.google.com/go/bigquery/dataset.go:74 main.recordHostStatus(0x596baeb79320, 0xc420288270, 0x7ffec7f0eb4e, 0xf, 0x596bae9102bb, 0x7) /build/moblab-generic-vm/tmp/portage/chromeos-base/lucifer-0.0.1-r82/work/lucifer-0.0.1/src/lucifer/cmd/lucifer_run_job/metrics.go:22 +0x3b main.recordAllHostStatus(0x596baeb79320, 0xc420288270, 0x596baeb71b20, 0xc420200080, 0x596bae9102bb, 0x7) /build/moblab-generic-vm/tmp/portage/chromeos-base/lucifer-0.0.1-r82/work/lucifer-0.0.1/src/lucifer/cmd/lucifer_run_job/metrics.go:45 +0x94 main.doRunningStep(0x596baeb79320, 0xc420288270, 0xc420200080, 0x0, 0x0) /build/moblab-generic-vm/tmp/portage/chromeos-base/lucifer-0.0.1-r82/work/lucifer-0.0.1/src/lucifer/cmd/lucifer_run_job/main.go:176 +0x19b main.runStartingJob(0x596baeb79260, 0xc42003e280, 0xc420200080, 0x9, 0x0) /build/moblab-generic-vm/tmp/portage/chromeos-base/lucifer-0.0.1-r82/work/lucifer-0.0.1/src/lucifer/cmd/lucifer_run_job/main.go:124 +0x17b main.runJob(0x596baeb79260, 0xc42003e280, 0xc4200d62c0, 0x0, 0x0) /build/moblab-generic-vm/tmp/portage/chromeos-base/lucifer-0.0.1-r82/work/lucifer-0.0.1/src/lucifer/cmd/lucifer_run_job/main.go:83 +0x20f main.main() /build/moblab-generic-vm/tmp/portage/chromeos-base/lucifer-0.0.1-r82/work/lucifer-0.0.1/src/lucifer/cmd/lucifer_run_job/main.go:60 +0x340 j And was fixed in this CL https://chromium-review.googlesource.com/c/chromiumos/infra/lucifer/+/1056093 Marking as fixed - re-open if I am missing something
,
May 12 2018
Thanks; updated the build annotation. |
|||
►
Sign in to add a comment |
|||
Comment 1 by jclinton@chromium.org
, May 12 2018