New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 736795 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jul 20
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Caroline PFQ builder failing with VM test failures

Project Member Reported by keta...@chromium.org, Jun 26 2017

Issue description

Caroline PFQ builder has been failing the VM tests.
Chrome OS - 9686.0.0
Chrome - 61.0.3138.0

tradefed harness issues:
  ERROR: Command </tmp/autotest-tradefed-install_SRg0jj/6f0cb3a8c25a34507dc7320e878de111/android-cts-7.1_r6-linux_x86-x86/android-cts/tools/cts-tradefed run commandAndExit cts --module CtsAccountManagerTestCases --precondition-arg skip-media-download --log-level-display=DEBUG --screenshot-on-failure --disable-reboot --logcat-on-failure> failed, rc=-15, Command(s) did not complete within 360 seconds
/tmp/cbuildbotv9018V/smoke_suite/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-cheets_CTS_N.7.1_r6.x86.CtsAccountManagerTestCases/cheets_CTS_N.CtsAccountManagerTestCases                     [  FAILED  ]
/tmp/cbuildbotv9018V/smoke_suite/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-cheets_CTS_N.7.1_r6.x86.CtsAccountManagerTestCases/cheets_CTS_N.CtsAccountManagerTestCases                       ERROR: Command </tmp/autotest-tradefed-install_SRg0jj/6f0cb3a8c25a34507dc7320e878de111/android-cts-7.1_r6-linux_x86-x86/android-cts/tools/cts-tradefed run commandAndExit cts --module CtsAccountManagerTestCases --precondition-arg skip-media-download --log-level-display=DEBUG --screenshot-on-failure --disable-reboot --logcat-on-failure> failed, rc=-15, Command(s) did not complete within 360 seconds
/tmp/cbuildbotv9018V/smoke_suite/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-cheets_CTS_N.7.1_r6.x86.CtsAccountManagerTestCases/cheets_CTS_N.CtsAccountManagerTestCases                       06/26 02:06:14.340 ERROR|             utils:0280| [stderr] openjdk version "1.8.0_131"




Persist requested.  Use --ssh_port 9228  --ssh_private_key /b/c/cbuild/repository/src/build/images/caroline/latest-cbuildbot/id_rsa  --kvm_pid /tmp/kvm.9228 to re-connect to it.

INFO    : QEMU binary: /b/c/cbuild/repository/chroot/usr/bin/qemu-system-x86_64
INFO    : QEMU version: QEMU emulator version 2.6.0, Copyright (c) 2003-2008 Fabrice Bellard
Stopping the KVM instance
Saving memory snapshot to /b/c/cbuild/repository/chroot/tmp/cbuildbotv9018V/smoke_suite/test_harness/failed/SimpleTestVerify/1_autotest_tests/chromiumos_qemu_mem.bin.KDWpE1...
    freezing VM...
    saving memory, piping through cat...
    flushing disk buffers...
    asking KVM to quit...
    done.

======================================================================
FAIL: SimpleTestVerify (crostestutils.au_test_harness.au_test.AUTest)
Test that only verifies the target image.
----------------------------------------------------------------------

----------------------------------------------------------------------
Ran 1 test in 3539.059s

FAILED (failures=1)
2017/06/26 01:00:44 - cros_build_lib.py - ERROR   : <class 'chromite.lib.parallel.BackgroundFailure'>: Test harness failed. See logs for details.
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/cros_au_test_harness.py", line 77, in run
    raise parallel.BackgroundFailure(msg)
BackgroundFailure: Test harness failed. See logs for details.

2017/06/26 01:00:44 - cros_build_lib.py - ERROR   : /b/c/cbuild/repository/src/platform/crostestutils/au_test_harness/cros_au_test_harness.py --base_image=/b/c/cbuild/repository/src/build/images/caroline/latest-cbuildbot/chromiumos_test_image.bin --target_image=/b/c/cbuild/repository/src/build/images/caroline/latest-cbuildbot/chromiumos_test_image.bin --board=caroline --type=vm --remote=0.0.0.0 --verbose --jobs=1 --ssh_private_key=/b/c/cbuild/repository/src/build/images/caroline/latest-cbuildbot/id_rsa --verify_suite_name=smoke --test_prefix=SimpleTestVerify --test_results_root=/b/c/cbuild/repository/chroot/tmp/cbuildbotv9018V/smoke_suite/test_harness --no_graphics exited with code 1: None
01:00:44: ERROR: 
!!!VMTests failed!!!
 
Cc: levarum@chromium.org
+Arc constable

The last time I saw this symptom (VMTests failed with no specific test failure propagated) was a problem with Android starting.

Cc: kbleicher@chromium.org

Comment 3 by jen...@chromium.org, Jun 26 2017

It looks like related to some kind of android test failure.

Looking into the sysinfo messages log.
https://00e9e64baccf408cea6d6191292a07cf8ec647252fc6a7112a-apidata.googleusercontent.com/download/storage/v1/b/chromeos-image-archive/o/caroline-chrome-pfq%2FR61-9677.0.0-rc1%2Fvm_test_results_2%2Fsmoke_suite%2Ftest_harness%2Fall%2FSimpleTestVerify%2F1_autotest_tests%2Fdebug%2Ftest_that.DEBUG?qk=AD5uMEsQmvs0TtESbUb42PPGJf37utM1b1lN5khPoJB26tEWEn5ir0lbBLfY0YsYXNBSj7r7ptVLe8FBAXGViK7V2Q6mpO2JFLTRsvqSduwS-nkLaLAsVTL4CSU4_7WBDvLIjeSQ9Wjjsgo79n6CZOVtU9B2BBBFuoNu8Z1PEK_9cgRKXP0yvoaI_znzE1u1psFQmxsVIYNVbWRjtn299S1wr1R2Hu5sOqBoRiikeQwgyAoVC_FnSxz9malOEapDAt0p3dnL6StpdBFf2X3Ph1UpqxC1MUa04vo9LrLShgzIqGph9a-VZP4l0C03gi6nlnSE6fqQrWOoGT69QSgguIflwLyNWByOZagMlsr3-uogPbhFBpeAtTV8h1j02uwl1CoP6VMMV0KGiGOJ-S35RXFK10ECsKF9cr7PC5221IcTfd8NRnTGT3KxelbX7hDhUs9ncLeSLxfUVrNkRKKkjXK4ATxdWsIW4kMHFn-8X60WXAvxU_JOZSX4CTu0U6y5w_S7btvsvG5IpNTICOATwWHMtVMgRGxOCVrqLgGq3Q8oec_D4uz5PsPvvI0fpq0AFaawm-7sDCh-Y_lapuZaMZVu5K6lL-0qYLBHC0Iqjh-yng5pOaOSgiXLG9QwV23DDzevbfe-EkD4PRV8yYaNF8lfQr46IQ3MYTCJKH7W58JgWc-e7EHXQugPOc-9saFc_JRvjIPcr5l-_GS-_9ZZUrxHsBHPkPR4KotrxfYciqMD-S7zvd_aJXjl880rvE9jNqm5Xxi9XjHjBpXohrQap-mmxqtqPz64K-BxcMvdPsd9OxoMWEMl2Ne-nB8A5-l-IjplMe209CMzUfAxMYmfn836agDOpU2zYiZdQv8eaTijOmMYkzMkIRHNqFPiVZbSYj99w2PZzaKGqL1oTeG5x6Rsx1FYBa_1PqIPVMlMwIH7hOsXCv_D0WC3M3anXp8aD8cWvw96KoVjx3h4XO0IgsipM9DOSiAUvw

There following error messages repeated shown in the log. It seems keep retrying but failed.
2017-06-23T01:15:56.545615-05:00 WARNING cryptohomed[2365]: TSS: Failed unix connect: /var/run/tcsd.socket - No such file or directory
2017-06-23T01:15:56.545707-05:00 WARNING cryptohomed[2365]: TSS: Got a list of valid IPs
2017-06-23T01:15:56.545833-05:00 WARNING cryptohomed[2365]: TSS: Could not connect to machine: localhost
2017-06-23T01:15:56.545880-05:00 ERR cryptohomed[2365]: TSS: Could not connect to any machine in the list.
2017-06-23T01:15:56.545888-05:00 ERR cryptohomed[2365]: TSS: Failed to send packet
2017-06-23T01:15:56.646058-05:00 ERR cryptohomed[2365]: TPM error 0x3011 (Communication failure): Error calling Tspi_Context_Connect
2017-06-23T01:15:56.646229-05:00 ERR cryptohomed[2365]: ConnectContextAsUser: Could not open the TPM
2017-06-23T01:15:56.646249-05:00 ERR cryptohomed[2365]: GetDictionaryAttackInfo: Failed to connect to the TPM.
And then tons of messages of:
2017-06-23T00:49:29.511077-05:00 WARNING kernel: [  222.487568] Task android.gms:car:7090 refused to die

Does anyone know who owns the test case CtsAccountManagerTestCases?

Comment 4 by jen...@chromium.org, Jun 26 2017

Cc: kaznacheev@chromium.org

Comment 5 by jen...@chromium.org, Jun 26 2017

Just chatted with levarum@, the fix has just landed b/62935227.
Components: Infra

Comment 7 by mmoss@chromium.org, Jan 2 2018

Components: -Infra Infra>Client>ChromeOS
[It appears that a bunch of old cros issues bulk-added the "Infra" component recently, but they should probably be "Infra>Client>ChromeOS".]
Components: Infra>Client>ChromeOS>CI
Components: -Infra>Client>ChromeOS
Status: Fixed (was: Assigned)

Sign in to add a comment