New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 818859 link

Starred by 5 users
Status: Fixed
Owner:
█ jkop@chromium.org
Last visit > 30 days ago
Closed: Apr 2018
Cc:
norvez@chromium.org
akahuang@chromium.org
█ jkop@chromium.org
█ jinjingl@chromium.org
dgarr...@chromium.org
█ eugenegonzalez@chromium.org
pprabhu@chromium.org
█ sha...@chromium.org
ihf@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocked on:
issue 819736
Blocking:
issue 823832



Sign in to add a comment

VMTests (attempt 1) times out

Project Member Reported by norvez@chromium.org, Mar 5 2018 
Reproduced several times, here are 2 examples

Without --swarming:
https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/release/20686
With --swarming:
https://ci.chromium.org/p/chromeos/builds/b8952879337432176352

In the latter, VMTests1 is stopped after 1 hour. It's marked green because of b/72806309 but it does indeed fail.

It looks like the swarming builders are slower so they hit the timeout.

Comment 1 by ayatane@chromium.org, Mar 5 2018

Cc: -dgarr...@chromium.org
Owner: dgarr...@chromium.org
Status: Assigned (was: Untriaged)
dgarrett, what do you think the next AI for this should be?

Comment 2 by dgarr...@chromium.org, Mar 5 2018

Owner: norvez@chromium.org
Swarming GCE builders have an identical configuration to buildbot builders (in as far as we have control). They aren't being heavily used right now, so they have cold caches, but should otherwise behave identically.

b/72806309 only affects the web UI, not behavior on the builder. cbuildbot is determining that the test passed, and so not retring the stage.


To my reading of the logs, everything passed, through two tests had warnings.


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-cheets_GTS.5.1_r3.GtsAdminTestCases                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-cheets_GTS.5.1_r3.GtsAdminTestCases/cheets_GTS.GtsAdminTestCases                        [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-cheets_CTS_N.7.1_r14.arm.CtsAppUsageHostTestCases                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-cheets_CTS_N.7.1_r14.arm.CtsAppUsageHostTestCases/cheets_CTS_N.CtsAppUsageHostTestCases [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-03-cheets_CTS_N.7.1_r14.x86.CtsAppUsageHostTestCases                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-03-cheets_CTS_N.7.1_r14.x86.CtsAppUsageHostTestCases/cheets_CTS_N.CtsAppUsageHostTestCases [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-cheets_CTS_N.7.1_r14.arm.CtsDramTestCases                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-cheets_CTS_N.7.1_r14.arm.CtsDramTestCases/cheets_CTS_N.CtsDramTestCases                 [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-05-cheets_CTS_N.7.1_r14.x86.CtsDramTestCases                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-05-cheets_CTS_N.7.1_r14.x86.CtsDramTestCases/cheets_CTS_N.CtsDramTestCases                 [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-06-cheets_StartAndroid.smoke                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-06-cheets_StartAndroid.smoke/cheets_StartAndroid.sanity                                    [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-07-login_LoginSuccess                                                                      [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-07-login_LoginSuccess/login_LoginSuccess                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-08-login_OwnershipRetaken                                                                  [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-08-login_OwnershipRetaken/login_OwnershipRetaken                                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-09-platform_CrosDisksDBus                                                                  [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-09-platform_CrosDisksDBus/platform_CrosDisksDBus                                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-10-security_OpenSSLBlacklist                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-10-security_OpenSSLBlacklist/security_OpenSSLBlacklist                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-11-security_Firewall                                                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-11-security_Firewall/security_Firewall                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-12-kernel_CryptoAPI                                                                        [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-12-kernel_CryptoAPI/kernel_CryptoAPI                                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-13-security_ProtocolFamilies                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-13-security_ProtocolFamilies/security_ProtocolFamilies                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-14-cheets_SettingsBridge                                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-14-cheets_SettingsBridge/cheets_SettingsBridge                                             [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-15-login_OwnershipNotRetaken                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-15-login_OwnershipNotRetaken/login_OwnershipNotRetaken                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-16-cheets_LoginScreen                                                                      [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-16-cheets_LoginScreen/cheets_LoginScreen                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-17-login_MultiUserPolicy                                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-17-login_MultiUserPolicy/login_MultiUserPolicy                                             [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-18-crosvm_Sanity                                                                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-18-crosvm_Sanity                                                                             WARN: Warning: Expecting CPU flags ['abm', 'aes', 'apic', 'arat', 'avx', 'avx2', 'bmi1', 'bmi2', 'clflush', 'cmov', 'constant_tsc', 'cx16', 'cx8', 'de', 'eagerfpu', 'erms', 'f16c', 'fma', 'fpu', 'fsgsbase', 'fxsr', 'hypervisor', 'kaiser', 'lahf_lm', 'lm', 'mca', 'mce', 'mmx', 'movbe', 'msr', 'mtrr', 'nopl', 'nx', 'pae', 'pat', 'pcid', 'pclmulqdq', 'pge', 'pni', 'popcnt', 'pse', 'pse36', 'rdrand', 'rdtscp', 'rep_good', 'sep', 'smep', 'sse', 'sse2', 'sse4_1', 'sse4_2', 'ssse3', 'syscall', 'tsc', 'vme', 'x2apic', 'xsave', 'xsaveopt'] but found ['aes', 'apic', 'arat', 'avx', 'clflush', 'cmov', 'constant_tsc', 'cx16', 'cx8', 'de', 'eagerfpu', 'fpu', 'fxsr', 'hypervisor', 'kaiser', 'lahf_lm', 'lm', 'mca', 'mce', 'mmx', 'msr', 'mtrr', 'nopl', 'nx', 'pae', 'pat', 'pclmulqdq', 'pge', 'pni', 'popcnt', 'pse', 'pse36', 'rdtscp', 'rep_good', 'retpoline', 'sep', 'sse', 'sse2', 'sse4_1', 'sse4_2', 'ssse3', 'syscall', 'tsc', 'vme', 'x2apic', 'xsave', 'xsaveopt']
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-18-crosvm_Sanity/crosvm_Sanity                                                             [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-18-crosvm_Sanity/crosvm_Sanity                                                               WARN: Warning: Expecting CPU flags ['abm', 'aes', 'apic', 'arat', 'avx', 'avx2', 'bmi1', 'bmi2', 'clflush', 'cmov', 'constant_tsc', 'cx16', 'cx8', 'de', 'eagerfpu', 'erms', 'f16c', 'fma', 'fpu', 'fsgsbase', 'fxsr', 'hypervisor', 'kaiser', 'lahf_lm', 'lm', 'mca', 'mce', 'mmx', 'movbe', 'msr', 'mtrr', 'nopl', 'nx', 'pae', 'pat', 'pcid', 'pclmulqdq', 'pge', 'pni', 'popcnt', 'pse', 'pse36', 'rdrand', 'rdtscp', 'rep_good', 'sep', 'smep', 'sse', 'sse2', 'sse4_1', 'sse4_2', 'ssse3', 'syscall', 'tsc', 'vme', 'x2apic', 'xsave', 'xsaveopt'] but found ['aes', 'apic', 'arat', 'avx', 'clflush', 'cmov', 'constant_tsc', 'cx16', 'cx8', 'de', 'eagerfpu', 'fpu', 'fxsr', 'hypervisor', 'kaiser', 'lahf_lm', 'lm', 'mca', 'mce', 'mmx', 'msr', 'mtrr', 'nopl', 'nx', 'pae', 'pat', 'pclmulqdq', 'pge', 'pni', 'popcnt', 'pse', 'pse36', 'rdtscp', 'rep_good', 'retpoline', 'sep', 'sse', 'sse2', 'sse4_1', 'sse4_2', 'ssse3', 'syscall', 'tsc', 'vme', 'x2apic', 'xsave', 'xsaveopt']
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-19-login_RemoteOwnership                                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-19-login_RemoteOwnership/login_RemoteOwnership                                             [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-20-security_HardlinkRestrictions                                                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-20-security_HardlinkRestrictions/security_HardlinkRestrictions                             [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-21-logging_CrashSender                                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-21-logging_CrashSender/logging_CrashSender                                                 [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-22-login_OwnershipTaken                                                                    [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-22-login_OwnershipTaken/login_OwnershipTaken                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-23-security_SymlinkRestrictions                                                            [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-23-security_SymlinkRestrictions/security_SymlinkRestrictions                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-24-kernel_ConfigVerify                                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-24-kernel_ConfigVerify/kernel_ConfigVerify                                                 [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-25-platform_CUPSDaemon                                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-25-platform_CUPSDaemon/platform_CUPSDaemon                                                 [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-26-security_RuntimeExecStack                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-26-security_RuntimeExecStack/security_RuntimeExecStack                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-27-security_ModuleLocking                                                                  [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-27-security_ModuleLocking/security_ModuleLocking                                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-28-security_SuidBinaries                                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-28-security_SuidBinaries/security_SuidBinaries.fscap                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-28-security_SuidBinaries/security_SuidBinaries.sgid                                        [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-28-security_SuidBinaries/security_SuidBinaries.suid                                        [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-29-security_ASLR                                                                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-29-security_ASLR/security_ASLR                                                             [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-30-login_Cryptohome                                                                        [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-30-login_Cryptohome/login_Cryptohome                                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-31-login_CryptohomeIncognito                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-31-login_CryptohomeIncognito/login_CryptohomeIncognito                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-32-security_SandboxedServices                                                              [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-32-security_SandboxedServices/security_SandboxedServices                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-33-security_SysLogPermissions                                                              [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-33-security_SysLogPermissions/security_SysLogPermissions                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-34-security_SandboxLinuxUnittests                                                          [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-34-security_SandboxLinuxUnittests/security_SandboxLinuxUnittests                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-35-security_RootfsStatefulSymlinks                                                         [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-35-security_RootfsStatefulSymlinks/security_RootfsStatefulSymlinks                         [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-36-security_RootCA                                                                         [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-36-security_RootCA/security_RootCA                                                         [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-37-platform_OSLimits                                                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-37-platform_OSLimits/platform_OSLimits                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-38-security_ChromiumOSLSM                                                                  [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-38-security_ChromiumOSLSM/security_ChromiumOSLSM                                           [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-39-platform_FilePerms                                                                      [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-39-platform_FilePerms/platform_FilePerms                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-40-cheets_SELinuxTest                                                                      [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-40-cheets_SELinuxTest/cheets_SELinuxTest                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-41-login_UserPolicyKeys                                                                    [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-41-login_UserPolicyKeys/login_UserPolicyKeys                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-42-security_DbusOwners                                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-42-security_DbusOwners/security_DbusOwners                                                 [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-43-login_OwnershipApi                                                                      [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-43-login_OwnershipApi/login_OwnershipApi                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-44-logging_UserCrash                                                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-44-logging_UserCrash/logging_UserCrash                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-45-security_mprotect                                                                       [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-45-security_mprotect/security_mprotect                                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-46-desktopui_KillRestart                                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-46-desktopui_KillRestart/desktopui_KillRestart.chrome                                      [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-46-desktopui_KillRestart/desktopui_KillRestart.session                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-47-cheets_ContainerSmokeTest                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-47-cheets_ContainerSmokeTest/cheets_ContainerSmokeTest                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-48-security_Minijail0                                                                      [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-48-security_Minijail0/security_Minijail0                                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-49-login_LogoutProcessCleanup                                                              [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-49-login_LogoutProcessCleanup/login_LogoutProcessCleanup                                   [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-50-security_NetworkListeners                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-50-security_NetworkListeners/security_NetworkListeners                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-51-security_Minijail_seccomp                                                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-51-security_Minijail_seccomp/security_Minijail_seccomp                                     [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-52-login_RetrieveActiveSessions                                                            [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-52-login_RetrieveActiveSessions/login_RetrieveActiveSessions                               [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-53-login_GuestAndActualSession                                                             [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-53-login_GuestAndActualSession/login_GuestAndActualSession                                 [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-54-login_SameSessionTwice                                                                  [  PASSED  ]
/tmp/cbuildbotVfazxo/smoke/test_harness/all/SimpleTestVerify/1_autotest_tests/results-54-login_SameSessionTwice/login_SameSessionTwice                                           [  PASSED  ]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total PASS: 111/111 (100%)

Comment 3 by norvez@chromium.org, Mar 5 2018

Cc: dgarr...@chromium.org
Which log are you looking at? Looking at the logs of VMTests1 of https://ci.chromium.org/p/chromeos/builds/b8952879337432176352 (https://logs.chromium.org/v/?s=chromeos%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8952879337432176352%2F%2B%2Fsteps%2FVMTest__attempt_1_%2F0%2Fstdout), that step is stopped after 1 hour. At that point it's still running cheets_ContainerSmokeTest

Comment 4 by dgarr...@chromium.org, Mar 5 2018 

Oh... I looked at the wrong of the two builds, sorry. Re looking.

Ah... the first VMTest stage did time out, but the retry passed. It doesn't look like the first attempt hung, just didn't finish within the time limit.

In other words, fairly standard flake.


************************************************************
** FAILED BUT FORGIVEN VMTest (attempt 1) (1:01:55)
************************************************************
** PASS VMTest (attempt 2) (1:02:56)
************************************************************

Comment 5 by dgarr...@chromium.org, Mar 5 2018 

Again, the swarming builders are identical to our standard GCE builders except that they were all allocated very recently. This means they tend to have high IP addresses, and may have different unlying hardware.

The only real difference to how builds run is the mechanism used to communicate the build request to the build machine.

There have a been some issues with the build recipes, but I believe they have all been ironed out. In general, this can be see by looking at the cbuildbot is command line invocation.

Comment 6 by norvez@chromium.org, Mar 6 2018 

It doesn't look like flake, there is something specific about VMTests1

https://ci.chromium.org/p/chromeos/builds/b8953118911565464736
https://ci.chromium.org/p/chromeos/builds/b8952882168176597840

In both cases VMTests1 times out after 1 hour before finishing the tests in the suite, but VMTests2 passes and the test suite (i.e. ignoring dev_install and friends) takes 40 minutes.

Comment 7 by dgarr...@chromium.org, Mar 6 2018

Cc: pprabhu@chromium.org ihf@chromium.org
Owner: jkop@chromium.org
Then we need an owner that understands VMTests.

cc'ing a few people, and passing off to someone investigating vmtests in detail.

Comment 8 by jkop@chromium.org, Mar 7 2018

Blocking: 750918
I guess I meet that description. I did some investigation and don't see any plausible causes. We're hoping to turn off the stage retry and have VM tests be retried only at the test level, so

Comment 9 by norvez@chromium.org, Mar 7 2018 

Looking at an example build https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8952767469722987136, it looks like VMTest1 runs in parallel with lots of other stages. Could we simply be overusing the CPU? How many cores do we have?

Comment 10 by dgarr...@chromium.org, Mar 7 2018 

All of the relevant GCE builders have 32 cores, and 208G of ram.

Comment 11 by ihf@chromium.org, Mar 7 2018 

Quite possible, do we have a CPU plot of the builder?

Comment 12 by dgarr...@chromium.org, Mar 7 2018 

It's not working for swarming yet, I haven't investigated why. Mostly because I don't know where it comes from for normal builds.

Comment 13 by dgarr...@chromium.org, Mar 7 2018

Cc: jinjingl@chromium.org

Comment 14 by jkop@chromium.org, Mar 7 2018 

Re #11, I don't see how to get one

Comment 15 by jinjingl@google.com, Mar 7 2018 

https://bugs.chromium.org/p/chromium/issues/detail?id=819736#c3 for getting a CPU plot for swarming builders.

Comment 17 by norvez@chromium.org, Mar 8 2018 

Some -but not all- canary builders have very recently shown similar behaviour, although in that case both VMTest stages time out. Maybe the same root cause, maybe a different root cause...

novato-arc64-release (bad):
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8952660048683298640
novato-release (bad):
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8952599926785273488
betty-release (good):
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8952599919228302544)

Comment 18 by jkop@chromium.org, Mar 9 2018 

The problems are always surfacing in the auto-update tests. Could this be related to the last bad interaction with those and swarming? (crbug.com/808045)

Comment 19 by dgarr...@chromium.org, Mar 9 2018

Cc: norvez@chromium.org
If so, I would expect the problems to be consistent for both vmtest stage runs.

Comment 20 by jkop@chromium.org, Mar 9 2018

Blockedon: 819736
Reasonable. In that case, it's likely to be CPU contention, so blocked on getting that data. I'll look into lengthening the timeout as a stopgap.

Comment 21 by norvez@chromium.org, Mar 9 2018 

Re: #18, afaict it's not in the auto-update test. The timeout happens in the smoke suite, the auto-update test is not run. See for example https://ci.chromium.org/p/chromeos/builds/b8952767469722987136, VMTest1 was stopped while it was running test 49/54

But yes, agreed that I would expect both VMTest stage runs to be identical in that regard.


Re: #17, it looks like it's a different issue for the canary builders after all. login_OwnershipNotRetaken is hanging. So ignore #17, I'll file a separate bug for that.

Comment 22 by norvez@chromium.org, Mar 9 2018 

FYI, filed  Issue 820315  for #17. It looks completely unrelated to this bug.

Comment 23 by ihf@chromium.org, Mar 9 2018 

Just lengthening the suite timeout is probably not going to be reliable, as there are many shorter timeouts inside of tests (and even Chrome itself) that would need to be adjusted.

Can we kick off all non-vmtest stages at high niceness? (nice and ionice)

Comment 24 by jkop@chromium.org, Mar 9 2018 

Wouldn't it be simpler to run the vmtest stage at a lower niceness?

Comment 25 by ihf@chromium.org, Mar 9 2018 

It would be simpler, not sure we can fully get the same effect with ionice. Probably close. Also, we are discussing VM inside of VM (maybe inside of VM). Outcomes are hard to predict.

But why not set the VMTest stage at "ionice -c 2 -n 0" plus "nice -n -10" and see what happens?

Comment 26 by jkop@chromium.org, Mar 13 2018 

I've looked, and I'm still not sure how to do that. I don't understand enough about how stages are kicked off.

Comment 27 by jkop@chromium.org, Mar 20 2018

Blocking: -750918 823832

Comment 28 by jkop@chromium.org, Mar 26 2018

Cc: akahuang@chromium.org sha...@chromium.org jkop@chromium.org eugenegonzalez@chromium.org
 Issue 825373  has been merged into this issue.

Comment 29 by jkop@chromium.org, Mar 26 2018

Summary: VMTests (attempt 1) times out (was: VMTests (attempt 1) times out for swarming tryjobs)
Hitting paladin, canary, and pre-cq builds, so not related to swarming.

Comment 30 by jrbarnette@chromium.org, Mar 26 2018 

Some of the comments on this bug seem to suggest that the second
VMTest run is faster than the first run.  The summary implies as
much, too.  How sure are we of this, really?

Given that we retry the first VMTest stage failure, and given that
the specific failure here is timeout, if the root cause is "VMTest
stage has highly variable run times", it would necessarily manifest
as "first time is slow, second time is fast".  And, if the problem
is merely that run time is variable, by far the most expedient thing
to do is just to increase the timeout to accommodate the variation.

Comment 31 by pho...@chromium.org, Mar 26 2018 

Some graphs of a year's worth of VMTest history:

- VMTests attempt 1 vs attempt 2 duration of passes: http://shortn/_7nmiJvB7Ch

This shows that the averages are about the same, but attempt 2 seems to have more variance.

- Passing VMTests on Golo vs. GCE: http://shortn/_hqkn8lImeb

GCE takes about 15 more to execute passing VMTests.

Comment 32 by pho...@chromium.org, Mar 26 2018 

whoops, typo: I meant GCE takes about 15 more *minutes*.

Comment 33 by ihf@chromium.org, Mar 27 2018 

Re #31: What really interests me (and I have been failing to change the queries) is the duration of the passing attempts 1 vs 2, but restricted to GCE+swarming only (AFAIK most of GCE is still without swarming due to this issue).

As the graphs plot averages/medians over sets with distribution I don't know it is important to be specific. Could you post the query for this?

It is good to know that VMTest on GCE takes 15 minutes longer. But GCE works fine without swarming so this does not explain timeouts.

Comment 34 by jkop@chromium.org, Mar 27 2018 

Re #33, that query is here: http://shortn/_qapaBnfROq

Comment 35 by ihf@chromium.org, Mar 28 2018 

I guess I don't understand this query. Also the graphs start before August 2017. IIRC VMTest on GCE+swarming was only enabled recently by Don?

Comment 36 by jkop@chromium.org, Mar 28 2018

Status: Started (was: Assigned)
Oh, by GCE+swarming you meant "GCE with swarming". I misunderstood. I'm not sure how you'd get that data; service_name might be the field that would carry that information? But that is only ever "cbuildbot" in this dataset.

Comment 37 by jkop@chromium.org, Mar 28 2018 

Never mind, found it: http://shortn/_dgbHw9pG7I

Comment 38 by dgarr...@chromium.org, Mar 28 2018 

If we have the host name of the build machine, all swarming builders are named "swarm-cros-.*". Other GCE builders will be named "cros-beefy-.*-c2", and Golo builders will be named "build.*-m2".

There are one or two other name patterns, but only with 1-2 builders each on oddball waterfalls, soon to go away.

Comment 39 by jkop@chromium.org, Mar 28 2018 

Yup, that's what I used for the Pcon view in #37
Project Member

Comment 40 by bugdroid1@chromium.org, Mar 29 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a03eafc82feb7f3f18f91a26057dfa9971789828

commit a03eafc82feb7f3f18f91a26057dfa9971789828
Author: Jacob Kopczynski <jkop@google.com>
Date: Thu Mar 29 02:34:42 2018

chromite: Increase VM Test timeout to 90 minutes.

Per http://shortn/_7nmiJvB7Ch, first and second attempts of the VM Test
stage have approximately the same runtime, which means that the
differences are unlikely to be significant. Therefore, in order to
reduce timeouts, the best course is just raising the timeout.

BUG= chromium:818859 
TEST=inspecting tryjob should be sufficient

Change-Id: Ia03e581dc15cc0a87ae594bbba2a48286b80ee28
Reviewed-on: https://chromium-review.googlesource.com/981595
Commit-Ready: Jacob Kopczynski <jkop@chromium.org>
Tested-by: Jacob Kopczynski <jkop@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@google.com>

[modify] https://crrev.com/a03eafc82feb7f3f18f91a26057dfa9971789828/cbuildbot/config_dump.json
[modify] https://crrev.com/a03eafc82feb7f3f18f91a26057dfa9971789828/lib/config_lib.py
Project Member

Comment 41 by bugdroid1@chromium.org, Mar 29 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5145a016df3650855d958bf5b659ff0d84a8cf16

commit 5145a016df3650855d958bf5b659ff0d84a8cf16
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Thu Mar 29 06:03:45 2018

Roll src/third_party/chromite/ 4e367c2f4..85a5ac7c6 (2 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/4e367c2f4075..85a5ac7c6177

$ git log 4e367c2f4..85a5ac7c6 --date=short --no-merges --format='%ad %ae %s'
2018-03-28 ihf cbuildbot: mark caroline/kevin-arcnext important.
2018-03-26 jkop chromite: Increase VM Test timeout to 90 minutes.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:826755 , chromium:818859 


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I8c9699eac4f21cb7c857f820d12011111218406d
Reviewed-on: https://chromium-review.googlesource.com/985650
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#546742}
[modify] https://crrev.com/5145a016df3650855d958bf5b659ff0d84a8cf16/DEPS

Comment 42 by jkop@chromium.org, Mar 29 2018 

Keeping an eye on the effects here: http://shortn/_WZqV55peoc. I'll declare it fixed if it doesn't act up by the end of next week.

Comment 43 by jkop@chromium.org, Apr 27 2018

Status: Fixed (was: Started)
or "the end of the next four weeks", apparently.
►

Sign in to add a comment