Swarming client or Chromite issues killing Canary HWTest |
|||||||
Issue descriptionNot sure if this is an issue with the Swarming Client/Libs or with how Chromite uses it but I saw a good number of these failures on the waterfall this morning. https://uberchromegw.corp.google.com/i/chromeos/builders/glimmer-release/builds/486 Then look at the HWTest Stage: https://uberchromegw.corp.google.com/i/chromeos/builders/glimmer-release/builds/486/steps/HWTest%20%5Bbvt-inline%5D/logs/stdio The error output is huge so only going to post snippets: 05:35:23: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpdwRNnc/tmpu_4x_v/temp_summary.json --raw-cmd --task-name glimmer-release/R54-8582.0.0-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 -- /usr/local/autotest/site_utils/run_suite.py --build glimmer-release/R54-8582.0.0 --board glimmer --suite_name bvt-inline --pool bvt --num 6 --file_bugs True --priority Build --timeout_mins 180 --retry True --max_retries 10 --minimum_duts 4 --suite_min_duts 6 --offload_failures_only False -m 69455019 07:59:24: WARNING: Killing tasks: [<_BackgroundTask(_BackgroundTask-5:7:4, started)>] 07:59:24: WARNING: Killing 10682 (sig=24 SIGXCPU) 07:59:24: WARNING: RunCommand: pstree -Apals 10682 07:59:24: WARNING: (stdout): init,1 `-chromebuild-sta,1098 -u /opt/chromebuild/chromebuild-startup.py --log /var/log/messages/chromebuild/startup.log `-python,1474 /opt/infra-bot-setup/infra-python/run.py infra.tools.bot_setup.start --root_dir /b --password_file /home/chrome-bot/.password_json --slave_name cros-beefy34-c2 `-python,3026 /b/build/slave/run_slave.py --no_save --no-gclient-sync --python buildbot.tac --nodaemon --logfile twistd.log `-python,9356 -u ../../../scripts/slave/annotated_run.py --use-factory-properties-from-disk --build-properties-gz=eNqdUctOwzAQ/BefSd02bprmBHdOSJwQimxnkxr8CGunFUL8O5sXB+DEbTWzO7sz+8GUlQ6siYlVT0yfMTjIVEi3U2kGtwnYsecbplB6fWYVczImQEbIYGxDrY8P9wSfU+pjxfmgAGeZ7rrRAftNF0JngWrHDZ+pEPkqAOjpABLorHEOMEOwICOsvB8cKbJKlMUN0zJBF/Cd2ns0TlJFoFL17+tGVAffmu5P7ZG+SDTSk3HS9bNWZ1KNcDHRBE94qQ6Htj3qQp2adrstizzXbXsSuRCN2Of6kJfyuGt2NDkvrqeba9Ow6iSOxXG/EovJ1T4N9BheQKfpZm9aiCm7AI6LRxahD9Gk2eua7fIe4ydBuyQbw4B6zvc73b8l3waCoLmjpTtRlGK7LU5iJP5jOOozNIMdf7PkV0crLzCumorVMoaYKYD2PReZ3hN7DfjamHGOKz4FxqcB/uNN2fLMzy+02OdO `-python,10007 ../../../scripts/slave/annotated_run.py --use-factory-properties-from-disk --build-properties-gz=eNqdUctOwzAQ/BefSd02bprmBHdOSJwQimxnkxr8CGunFUL8O5sXB+DEbTWzO7sz+8GUlQ6siYlVT0yfMTjIVEi3U2kGtwnYsecbplB6fWYVczImQEbIYGxDrY8P9wSfU+pjxfmgAGeZ7rrRAftNF0JngWrHDZ+pEPkqAOjpABLorHEOMEOwICOsvB8cKbJKlMUN0zJBF/Cd2ns0TlJFoFL17+tGVAffmu5P7ZG+SDTSk3HS9bNWZ1KNcDHRBE94qQ6Htj3qQp2adrstizzXbXsSuRCN2Of6kJfyuGt2NDkvrqeba9Ow6iSOxXG/EovJ1T4N9BheQKfpZm9aiCm7AI6LRxahD9Gk2eua7fIe4ydBuyQbw4B6zvc73b8l3waCoLmjpTtRlGK7LU5iJP5jOOozNIMdf7PkV0crLzCumorVMoaYKYD2PReZ3hN7DfjamHGOKz4FxqcB/uNN2fLMzy+02OdO `-python,10019 -u /b/build_internal/scripts/slave/recipes.py --verbose run --workdir=/b/build/slave/glimmer-release-master/build --properties-file=/b/build/slave/glimmer-release-master/.recipe_runtime/tmpG25Dos/recipe_properties.json cros/cbuildbot_internal `-python,10022 -u /b/build_internal/scripts/slave/.recipe_deps/recipe_engine/recipes.py --package /b/build_internal/scripts/slave/infra/config/recipes.cfg --bootstrap-script /b/build_internal/scripts/slave/recipes.py --verbose run --workdir=/b/build/slave/glimmer-release-master/build --properties-file=/b/build/slave/glimmer-release-master/.recipe_runtime/tmpG25Dos/recipe_properties.json cros/cbuildbot_internal `-cbuildbot,10366 /b/build/slave/glimmer-release-master/build/chromite/bin/cbuildbot --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 486 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 947672 glimmer-release `-cbuildbot,30268 chromite/bin/cbuildbot glimmer-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 486 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 947672 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8582.0.0 --metadata_dump /tmp/cbuildbot-tmpdwRNnc/metadata8BeKWW `-cbuildbot,27399 chromite/bin/cbuildbot glimmer-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 486 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 947672 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8582.0.0 --metadata_dump /tmp/cbuildbot-tmpdwRNnc/metadata8BeKWW `-cbuildbot,8322 chromite/bin/cbuildbot glimmer-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 486 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 947672 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8582.0.0 --metadata_dump /tmp/cbuildbot-tmpdwRNnc/metadata8BeKWW `-cbuildbot,10682 chromite/bin/cbuildbot glimmer-release --buildroot /b/cbuild/internal_master --buildbot --branch master --buildnumber 486 --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal --master-build-id 947672 --resume --timeout 0 --notee --nocgroups --buildroot /b/cbuild/internal_master --version 8582.0.0 --metadata_dump /tmp/cbuildbot-tmpdwRNnc/metadata8BeKWW `-python,15794 /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json /tmp/cbuildbot-tmpdwRNnc/tmpu_4x_v/temp_summary.json --raw-cmd --task-name glimmer-release/R54-8582.0.0-bvt-inline --dimension os Ubuntu-14.04 --dimension pool default --print-status-updates --timeout 14400 --io-timeout 14400 --hard-timeout 14400 --expiration 1200 -- /usr/local/autotest/site_utils/run_suite.py --build glimmer-release/R54-8582.0.0 --board glimmer --suite_name bvt-inline --pool bvt --num 6 --file_bugs True --priority Build --timeout_mins 180 --retry True --max_retries 10 --minimum_duts 4 --suite_min_duts 6 --offload_failures_only False -m 69455019 `-{python},15797 ........ Then a stacktrace: 07:59:55: WARNING: Killing 10682 (sig=15 SIGTERM) 07:59:25: ERROR: pre-kill notification (SIGXCPU); traceback: File "chromite/bin/cbuildbot", line 164, in <module> commandline.ScriptWrapperMain(FindTarget) File "/b/cbuild/internal_master/chromite/lib/commandline.py", line 834, in ScriptWrapperMain ret = target(argv[1:]) File "/b/cbuild/internal_master/chromite/scripts/cbuildbot.py", line 1282, in main _RunBuildStagesWrapper(options, site_config, build_config) File "/b/cbuild/internal_master/chromite/scripts/cbuildbot.py", line 242, in _RunBuildStagesWrapper if not builder.Run(): File "/b/cbuild/internal_master/chromite/cbuildbot/builders/generic_builders.py", line 316, in Run self.RunStages() File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 470, in RunStages super(DistributedBuilder, self).RunStages() File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 334, in RunStages self._RunDefaultTypeBuild() File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 321, in _RunDefaultTypeBuild self.RunBuildStages() File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 282, in RunBuildStages with parallel.BackgroundTaskRunner(task_runner) as queue: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 744, in BackgroundTaskRunner with _BackgroundTask.ParallelTasks(steps, halt_on_error=halt_on_error): File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 531, in ParallelTasks task.start() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 387, in start return multiprocessing.Process.start(self) File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start self._popen = Popen(self) File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__ code = process_obj._bootstrap() File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 398, in run errors = self._Run() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 440, in _Run self._task(*self._task_args, **self._task_kwargs) File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 601, in TaskRunner task(*x, **task_kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 128, in _RunBackgroundStagesForBoardAndMarkAsSuccessful self._RunBackgroundStagesForBoard(builder_run, board) File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 220, in _RunBackgroundStagesForBoard lambda: self._RunHWTests(builder_run, board), File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 673, in RunParallelSteps halt_on_error=halt_on_error): File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 531, in ParallelTasks task.start() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 387, in start return multiprocessing.Process.start(self) File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start self._popen = Popen(self) File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__ code = process_obj._bootstrap() File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 398, in run errors = self._Run() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 440, in _Run self._task(*self._task_args, **self._task_kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 220, in <lambda> lambda: self._RunHWTests(builder_run, board), File "/b/cbuild/internal_master/chromite/cbuildbot/builders/simple_builders.py", line 112, in _RunHWTests self._RunParallelStages(parallel_stages) File "/b/cbuild/internal_master/chromite/cbuildbot/builders/generic_builders.py", line 118, in _RunParallelStages parallel.RunParallelSteps(steps) File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 673, in RunParallelSteps halt_on_error=halt_on_error): File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 531, in ParallelTasks task.start() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 387, in start return multiprocessing.Process.start(self) File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start self._popen = Popen(self) File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__ code = process_obj._bootstrap() File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 398, in run errors = self._Run() File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 440, in _Run self._task(*self._task_args, **self._task_kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/stages/generic_stages.py", line 525, in Run self.PerformStage() File "/b/cbuild/internal_master/chromite/cbuildbot/stages/test_stages.py", line 380, in PerformStage debug=debug, subsystems=subsystems, skip_duts_check=skip_duts_check) File "/b/cbuild/internal_master/chromite/cbuildbot/failures_lib.py", line 172, in wrapped_functor return functor(*args, **kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 903, in RunHWTestSuite _HWTestWait(cmd, job_id, **swarming_args) File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 1132, in _HWTestWait **kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/swarming_lib.py", line 148, in RunSwarmingCommandWithRetries return retry_util.RetryCommand(RunSwarmingCommand, max_retry, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 180, in RetryCommand return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 88, in GenericRetry ret = functor(*args, **kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/swarming_lib.py", line 87, in RunSwarmingCommand result = cros_build_lib.RunCommand(swarming_cmd, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/cros_build_lib.py", line 589, in RunCommand (cmd_result.output, cmd_result.error) = proc.communicate(input) File "/usr/lib/python2.7/subprocess.py", line 796, in communicate self.wait() File "/usr/lib/python2.7/subprocess.py", line 1376, in wait pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0) File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(*args) @@@STEP_FAILURE@@@ 07:59:56: ERROR: Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/cbuildbot/stages/generic_stages.py", line 525, in Run self.PerformStage() File "/b/cbuild/internal_master/chromite/cbuildbot/stages/test_stages.py", line 380, in PerformStage debug=debug, subsystems=subsystems, skip_duts_check=skip_duts_check) File "/b/cbuild/internal_master/chromite/cbuildbot/failures_lib.py", line 172, in wrapped_functor return functor(*args, **kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 903, in RunHWTestSuite _HWTestWait(cmd, job_id, **swarming_args) File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 1132, in _HWTestWait **kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/swarming_lib.py", line 148, in RunSwarmingCommandWithRetries return retry_util.RetryCommand(RunSwarmingCommand, max_retry, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 180, in RetryCommand return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 88, in GenericRetry ret = functor(*args, **kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/swarming_lib.py", line 87, in RunSwarmingCommand result = cros_build_lib.RunCommand(swarming_cmd, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/cros_build_lib.py", line 589, in RunCommand (cmd_result.output, cmd_result.error) = proc.communicate(input) File "/usr/lib/python2.7/subprocess.py", line 796, in communicate self.wait() File "/usr/lib/python2.7/subprocess.py", line 1376, in wait pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0) File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(*args) File "/b/cbuild/internal_master/chromite/lib/cros_build_lib.py", line 341, in _KillChildProcess if not signals.RelaySignal(original_handler, signum, frame): File "/b/cbuild/internal_master/chromite/lib/signals.py", line 26, in RelaySignal handler(signum, frame) File "/b/cbuild/internal_master/chromite/lib/commandline.py", line 731, in _DefaultHandler signum, 'Received signal %i; shutting down' % (signum,)) _ShutDownException: (15, 'Received signal 15; shutting down') Assigning to deputy for now but may require input from people more familiar with this code.
,
Jul 13 2016
Things that would help: - Specify --tags and add tags - Specify --decorate so the task id and URL are printed. The main problem is that the task id is not printed so it's hard to find the task back. The tags helps searching for a kind of tasks. https://chromeos-proxy.appspot.com/user/tasks?state=completed_failure
,
Jul 13 2016
BTW, Swarming sends SIGTERM on hard timeout.
,
Jul 13 2016
Maruel, do you have any idea what is going on in this output though? Is the client dying?
,
Jul 13 2016
It looks to me that the suite is taking very long and chromite decided to kill swarming_client.
,
Jul 13 2016
Ahh so Richard just extended the HWTest timeout to 180 minutes. Guess he needs to extend the chromite timeout as well then?
,
Jul 13 2016
The 180 minute timeout is already in chromite. But it could be there's some other timeout in chromite that doesn't account for the HWTest stage timeout. Really, I hate our system.
,
Jul 13 2016
This is the job and scroll down you can see the log from run_suite https://chromeos-proxy.appspot.com/user/task/2ffd12b2e602eb10 The real problem is the suite timed out. Fixing crbug.com/615569 would make the error message more clear. Autotest instance: cautotest 07-13-2016 [05:35:21] Created suite job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455019 @@@STEP_LINK@Link to suite@http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455019@@@ The suite job has another 2:29:33.357354 till timeout. The suite job has another 1:59:29.335355 till timeout. 07-13-2016 [07:05:58] printing summary of incomplete jobs (55): platform_DMVerityBitCorruption: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455260 platform_DMVerityCorruption: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455263 provision_AutoUpdate.double: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455266 security_ModuleLocking: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455271 login_OwnershipRetaken: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455276 security_ASLR: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455284 login_OwnershipApi: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455292 login_RemoteOwnership: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455298 security_ChromiumOSLSM: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455301 logging_CrashSender: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455306 desktopui_KillRestart: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455309 build_RootFilesystemSize: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455312 login_OwnershipNotRetaken: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455314 security_SuidBinaries: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455316 security_SysLogPermissions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455318 security_RuntimeExecStack: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455320 security_OpenFDs: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455323 security_EnableChromeTesting: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455325 security_SymlinkRestrictions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455327 login_UserPolicyKeys: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455329 login_Cryptohome: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455331 security_RestartJob: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455333 security_HardlinkRestrictions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455335 security_Firewall: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455338 security_ProfilePermissions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455340 security_NetworkListeners: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455344 security_SandboxStatus: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455347 security_RootfsStatefulSymlinks: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455352 platform_CUPSDaemon: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455360 platform_CrosDisksDBus: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455368 login_OwnershipTaken: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455377 security_ptraceRestrictions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455386 security_RootCA: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455399 security_OpenSSLBlacklist: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455413 platform_FilePerms: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455429 security_SandboxLinuxUnittests: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455445 security_Minijail_seccomp: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455472 login_LogoutProcessCleanup: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455486 security_mprotect: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455499 login_MultiUserPolicy: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455510 kernel_ConfigVerify: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455522 security_DbusOwners: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455536 security_OpenSSLRegressions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455549 security_ProtocolFamilies: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455559 security_Minijail0: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455570 login_CryptohomeIncognito: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455579 security_StatefulPermissions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455589 login_LoginSuccess: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455599 security_AccountsBaseline: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455611 security_SandboxedServices: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455623 platform_OSLimits: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455635 login_MultipleSessions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455648 login_SameSessionTwice: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455661 login_GuestAndActualSession: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455671 login_RetrieveActiveSessions: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=69455682 The suite job has another 1:29:22.468529 till timeout. The suite job has another 0:59:19.168579 till timeout. The suite job has another 0:29:13.959466 till timeout. The suite job has another -1 day, 23:59:13.271507 till timeout. Suite timed out. Started on 07-13-2016 [05:35:21], timed out on 07-13-2016 [08:49:28] Suite job [ FAILED ] Suite job ABORT: security_OpenSSLRegressions [ FAILED ] security_OpenSSLRegressions ABORT: Timed out, did not run. security_ProtocolFamilies [ FAILED ] security_ProtocolFamilies ABORT: Timed out, did not run. security_Minijail0 [ FAILED ] security_Minijail0 ABORT: Timed out, did not run. login_CryptohomeIncognito [ FAILED ] login_CryptohomeIncognito ABORT: Timed out, did not run. security_StatefulPermissions [ FAILED ] security_StatefulPermissions ABORT: Timed out, did not run. login_LoginSuccess [ FAILED ] login_LoginSuccess ABORT: Timed out, did not run. security_AccountsBaseline [ FAILED ] security_AccountsBaseline ABORT: Timed out, did not run. security_SandboxedServices [ FAILED ] security_SandboxedServices ABORT: Timed out, did not run. platform_OSLimits [ FAILED ] platform_OSLimits ABORT: Timed out, did not run. login_MultipleSessions [ FAILED ] login_MultipleSessions ABORT: Timed out, did not run. login_SameSessionTwice [ FAILED ] login_SameSessionTwice ABORT: Timed out, did not run. login_GuestAndActualSession [ FAILED ] login_GuestAndActualSession ABORT: Timed out, did not run. login_RetrieveActiveSessions [ FAILED ] login_RetrieveActiveSessions ABORT: Timed out, did not run. platform_DMVerityBitCorruption.first [ PASSED ] platform_DMVerityBitCorruption.middle [ PASSED ] platform_DMVerityBitCorruption.last [ PASSED ] platform_DMVerityCorruption [ PASSED ] provision_AutoUpdate.double [ PASSED ] security_ModuleLocking [ PASSED ] login_OwnershipRetaken [ PASSED ] security_ASLR [ PASSED ] login_OwnershipApi [ PASSED ] login_RemoteOwnership [ PASSED ] security_ChromiumOSLSM [ PASSED ] logging_CrashSender [ PASSED ] desktopui_KillRestart.chrome [ PASSED ] desktopui_KillRestart.session [ PASSED ] build_RootFilesystemSize [ PASSED ] login_OwnershipNotRetaken [ PASSED ] security_SuidBinaries.suid [ PASSED ] security_SuidBinaries.sgid [ PASSED ] security_SuidBinaries.fscap [ PASSED ] security_SysLogPermissions [ PASSED ] security_RuntimeExecStack [ PASSED ] security_OpenFDs [ PASSED ] security_EnableChromeTesting [ PASSED ] security_SymlinkRestrictions [ PASSED ] login_UserPolicyKeys [ PASSED ] login_Cryptohome [ PASSED ] security_RestartJob [ PASSED ] security_HardlinkRestrictions [ PASSED ] security_Firewall [ PASSED ] security_ProfilePermissions.login [ PASSED ] security_ProfilePermissions.guest [ PASSED ] security_NetworkListeners [ PASSED ] security_SandboxStatus [ PASSED ] security_RootfsStatefulSymlinks [ PASSED ] platform_CUPSDaemon [ PASSED ] platform_CrosDisksDBus [ PASSED ] login_OwnershipTaken [ PASSED ] security_ptraceRestrictions [ PASSED ] security_RootCA [ PASSED ] security_OpenSSLBlacklist [ PASSED ] platform_FilePerms [ PASSED ] security_SandboxLinuxUnittests [ PASSED ] security_Minijail_seccomp [ PASSED ] login_LogoutProcessCleanup [ PASSED ] security_mprotect [ PASSED ] login_MultiUserPolicy [ PASSED ] kernel_ConfigVerify [ PASSED ] security_DbusOwners [ PASSED ] Suite timings: Downloads started at 2016-07-13 05:35:16 Payload downloads ended at 2016-07-13 05:35:19 Suite started at 2016-07-13 05:37:25 Artifact downloads ended (at latest) at 2016-07-13 05:37:28 Testing started at 2016-07-13 07:11:03 Testing ended at 2016-07-13 08:46:13 Links to test logs: Suite job http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ security_OpenSSLRegressions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ security_ProtocolFamilies http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ security_Minijail0 http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ login_CryptohomeIncognito http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ security_StatefulPermissions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ login_LoginSuccess http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ security_AccountsBaseline http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ security_SandboxedServices http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ platform_OSLimits http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ login_MultipleSessions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ login_SameSessionTwice http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ login_GuestAndActualSession http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ login_RetrieveActiveSessions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/ platform_DMVerityBitCorruption.first http://cautotest/tko/retrieve_logs.cgi?job=/results/69455260-chromeos-test/ platform_DMVerityBitCorruption.middle http://cautotest/tko/retrieve_logs.cgi?job=/results/69455260-chromeos-test/ platform_DMVerityBitCorruption.last http://cautotest/tko/retrieve_logs.cgi?job=/results/69455260-chromeos-test/ platform_DMVerityCorruption http://cautotest/tko/retrieve_logs.cgi?job=/results/69455263-chromeos-test/ provision_AutoUpdate.double http://cautotest/tko/retrieve_logs.cgi?job=/results/69455266-chromeos-test/ security_ModuleLocking http://cautotest/tko/retrieve_logs.cgi?job=/results/69455271-chromeos-test/ login_OwnershipRetaken http://cautotest/tko/retrieve_logs.cgi?job=/results/69455276-chromeos-test/ security_ASLR http://cautotest/tko/retrieve_logs.cgi?job=/results/69455284-chromeos-test/ login_OwnershipApi http://cautotest/tko/retrieve_logs.cgi?job=/results/69455292-chromeos-test/ login_RemoteOwnership http://cautotest/tko/retrieve_logs.cgi?job=/results/69455298-chromeos-test/ security_ChromiumOSLSM http://cautotest/tko/retrieve_logs.cgi?job=/results/69455301-chromeos-test/ logging_CrashSender http://cautotest/tko/retrieve_logs.cgi?job=/results/69455306-chromeos-test/ desktopui_KillRestart.chrome http://cautotest/tko/retrieve_logs.cgi?job=/results/69455309-chromeos-test/ desktopui_KillRestart.session http://cautotest/tko/retrieve_logs.cgi?job=/results/69455309-chromeos-test/ build_RootFilesystemSize http://cautotest/tko/retrieve_logs.cgi?job=/results/69455312-chromeos-test/ login_OwnershipNotRetaken http://cautotest/tko/retrieve_logs.cgi?job=/results/69455314-chromeos-test/ security_SuidBinaries.suid http://cautotest/tko/retrieve_logs.cgi?job=/results/69455316-chromeos-test/ security_SuidBinaries.sgid http://cautotest/tko/retrieve_logs.cgi?job=/results/69455316-chromeos-test/ security_SuidBinaries.fscap http://cautotest/tko/retrieve_logs.cgi?job=/results/69455316-chromeos-test/ security_SysLogPermissions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455318-chromeos-test/ security_RuntimeExecStack http://cautotest/tko/retrieve_logs.cgi?job=/results/69455320-chromeos-test/ security_OpenFDs http://cautotest/tko/retrieve_logs.cgi?job=/results/69455323-chromeos-test/ security_EnableChromeTesting http://cautotest/tko/retrieve_logs.cgi?job=/results/69455325-chromeos-test/ security_SymlinkRestrictions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455327-chromeos-test/ login_UserPolicyKeys http://cautotest/tko/retrieve_logs.cgi?job=/results/69455329-chromeos-test/ login_Cryptohome http://cautotest/tko/retrieve_logs.cgi?job=/results/69455331-chromeos-test/ security_RestartJob http://cautotest/tko/retrieve_logs.cgi?job=/results/69455333-chromeos-test/ security_HardlinkRestrictions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455335-chromeos-test/ security_Firewall http://cautotest/tko/retrieve_logs.cgi?job=/results/69455338-chromeos-test/ security_ProfilePermissions.login http://cautotest/tko/retrieve_logs.cgi?job=/results/69455340-chromeos-test/ security_ProfilePermissions.guest http://cautotest/tko/retrieve_logs.cgi?job=/results/69455340-chromeos-test/ security_NetworkListeners http://cautotest/tko/retrieve_logs.cgi?job=/results/69455344-chromeos-test/ security_SandboxStatus http://cautotest/tko/retrieve_logs.cgi?job=/results/69455347-chromeos-test/ security_RootfsStatefulSymlinks http://cautotest/tko/retrieve_logs.cgi?job=/results/69455352-chromeos-test/ platform_CUPSDaemon http://cautotest/tko/retrieve_logs.cgi?job=/results/69455360-chromeos-test/ platform_CrosDisksDBus http://cautotest/tko/retrieve_logs.cgi?job=/results/69455368-chromeos-test/ login_OwnershipTaken http://cautotest/tko/retrieve_logs.cgi?job=/results/69455377-chromeos-test/ security_ptraceRestrictions http://cautotest/tko/retrieve_logs.cgi?job=/results/69455386-chromeos-test/ security_RootCA http://cautotest/tko/retrieve_logs.cgi?job=/results/69455399-chromeos-test/ security_OpenSSLBlacklist http://cautotest/tko/retrieve_logs.cgi?job=/results/69455413-chromeos-test/ platform_FilePerms http://cautotest/tko/retrieve_logs.cgi?job=/results/69455429-chromeos-test/ security_SandboxLinuxUnittests http://cautotest/tko/retrieve_logs.cgi?job=/results/69455445-chromeos-test/ security_Minijail_seccomp http://cautotest/tko/retrieve_logs.cgi?job=/results/69455472-chromeos-test/ login_LogoutProcessCleanup http://cautotest/tko/retrieve_logs.cgi?job=/results/69455486-chromeos-test/ security_mprotect http://cautotest/tko/retrieve_logs.cgi?job=/results/69455499-chromeos-test/ login_MultiUserPolicy http://cautotest/tko/retrieve_logs.cgi?job=/results/69455510-chromeos-test/ kernel_ConfigVerify http://cautotest/tko/retrieve_logs.cgi?job=/results/69455522-chromeos-test/ security_DbusOwners http://cautotest/tko/retrieve_logs.cgi?job=/results/69455536-chromeos-test/ Attempting to display pool info: bvt host: chromeos4-row7-rack2-host1, status: Provisioning, locked: False diagnosis: Working labels: ['bluetooth', 'ec:cros', 'audio_loopback_dongle', 'glimmer', 'board:glimmer', 'board_freq_mem:glimmer_2.2GHz_2GB', 'graphics:gl', 'gpu_family:baytrail', 'pool:bvt', 'cts_abi_x86', 'cts_abi_arm', 'servo', 'variant:glimmer', 'sku:glimmer_intel_celeron_n2930_2Gb', 'touchpad', 'os:cros', 'power:battery', 'phase:PVT', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'webcam', 'internal_display', 'storage:mmc', 'cros-version:glimmer-release/R54-8582.0.0'] Last 10 jobs within 3:18:00: 57332770 Cleanup started on: 2016-07-13 08:47:11 status PASS 57332597 Reset started on: 2016-07-13 08:44:17 status FAIL 69785918 glimmer-release/R54-8582.0.0/bvt-inline/login_MultiUserPolicy started on: 2016-07-13 08:04:47 status Aborted 57332223 Reset started on: 2016-07-13 08:02:40 status PASS 69785807 glimmer-release/R54-8582.0.0/bvt-inline/security_RootCA started on: 2016-07-13 07:57:24 status Completed 57331937 Reset started on: 2016-07-13 07:54:26 status PASS 69785760 glimmer-release/R54-8582.0.0/bvt-inline/security_RootfsStatefulSymlinks started on: 2016-07-13 07:51:24 status Completed 57331693 Reset started on: 2016-07-13 07:48:45 status PASS 69785743 glimmer-release/R54-8582.0.0/bvt-inline/security_HardlinkRestrictions started on: 2016-07-13 07:45:21 status Completed 57331394 Reset started on: 2016-07-13 07:42:24 status PASS host: chromeos4-row7-rack3-host7, status: Provisioning, locked: False diagnosis: Working labels: ['bluetooth', 'ec:cros', 'graphics:gl', 'board_freq_mem:glimmer_1.8GHz_2GB', 'board:glimmer', 'glimmer', 'gpu_family:baytrail', 'pool:bvt', 'audio_loopback_dongle', 'cts_abi_x86', 'cts_abi_arm', 'servo', 'variant:glimmer', 'sku:glimmer_intel_celeron_n2930_2Gb', 'touchpad', 'internal_display', 'os:cros', 'power:battery', 'phase:PVT', 'storage:mmc', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'webcam', 'cros-version:glimmer-release/R54-8582.0.0'] Last 10 jobs within 3:18:00: 57332769 Cleanup started on: 2016-07-13 08:47:11 status PASS 57332596 Reset started on: 2016-07-13 08:44:17 status FAIL 69785930 glimmer-release/R54-8582.0.0/bvt-inline/kernel_ConfigVerify started on: 2016-07-13 08:04:47 status Aborted 57332224 Reset started on: 2016-07-13 08:02:40 status PASS 69785837 glimmer-release/R54-8582.0.0/bvt-inline/platform_FilePerms started on: 2016-07-13 07:58:02 status Completed 57331959 Reset started on: 2016-07-13 07:55:05 status PASS 69785755 glimmer-release/R54-8582.0.0/bvt-inline/security_SandboxStatus started on: 2016-07-13 07:51:24 status Completed 57331692 Reset started on: 2016-07-13 07:48:45 status PASS 69785741 glimmer-release/R54-8582.0.0/bvt-inline/security_RestartJob started on: 2016-07-13 07:45:21 status Completed 57331393 Reset started on: 2016-07-13 07:42:23 status PASS host: chromeos4-row7-rack3-host9, status: Provisioning, locked: False diagnosis: Working labels: ['bluetooth', 'ec:cros', 'graphics:gl', 'board_freq_mem:glimmer_1.8GHz_2GB', 'board:glimmer', 'glimmer', 'gpu_family:baytrail', 'audio_loopback_dongle', 'pool:bvt', 'cts_abi_x86', 'cts_abi_arm', 'servo', 'variant:glimmer', 'sku:glimmer_intel_celeron_n2930_2Gb', 'touchpad', 'internal_display', 'os:cros', 'power:battery', 'phase:PVT', 'storage:mmc', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'webcam', 'cros-version:glimmer-release/R54-8582.0.0'] Last 10 jobs within 3:18:00: 57332768 Cleanup started on: 2016-07-13 08:47:11 status PASS 57332394 Reset started on: 2016-07-13 08:42:31 status FAIL 69785880 glimmer-release/R54-8582.0.0/bvt-inline/security_Minijail_seccomp started on: 2016-07-13 08:02:38 status Completed 57332120 Reset started on: 2016-07-13 07:59:30 status PASS 69785785 glimmer-release/R54-8582.0.0/bvt-inline/login_OwnershipTaken started on: 2016-07-13 07:54:20 status Completed 57331835 Reset started on: 2016-07-13 07:52:37 status PASS 69785746 glimmer-release/R54-8582.0.0/bvt-inline/security_Firewall started on: 2016-07-13 07:48:09 status Completed 57331541 Reset started on: 2016-07-13 07:46:09 status PASS 69785735 glimmer-release/R54-8582.0.0/bvt-inline/security_SymlinkRestrictions started on: 2016-07-13 07:41:40 status Completed 57331251 Reset started on: 2016-07-13 07:39:41 status PASS host: chromeos4-row7-rack3-host11, status: Provisioning, locked: False diagnosis: Working labels: ['bluetooth', 'ec:cros', 'graphics:gl', 'board_freq_mem:glimmer_1.8GHz_2GB', 'board:glimmer', 'glimmer', 'gpu_family:baytrail', 'pool:bvt', 'audio_loopback_dongle', 'cts_abi_x86', 'cts_abi_arm', 'servo', 'variant:glimmer', 'sku:glimmer_intel_celeron_n2930_2Gb', 'touchpad', 'internal_display', 'os:cros', 'power:battery', 'phase:PVT', 'storage:mmc', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'webcam', 'cros-version:glimmer-release/R54-8582.0.0'] Last 10 jobs within 3:18:00: 57332767 Cleanup started on: 2016-07-13 08:47:11 status PASS 57332393 Reset started on: 2016-07-13 08:42:30 status FAIL 69785894 glimmer-release/R54-8582.0.0/bvt-inline/login_LogoutProcessCleanup started on: 2016-07-13 08:02:38 status Completed 57332121 Reset started on: 2016-07-13 07:59:30 status PASS 69785794 glimmer-release/R54-8582.0.0/bvt-inline/security_ptraceRestrictions started on: 2016-07-13 07:54:20 status Completed 57331836 Reset started on: 2016-07-13 07:52:37 status PASS 69785752 glimmer-release/R54-8582.0.0/bvt-inline/security_NetworkListeners started on: 2016-07-13 07:48:42 status Completed 57331594 Reset started on: 2016-07-13 07:47:00 status PASS 69785733 glimmer-release/R54-8582.0.0/bvt-inline/security_EnableChromeTesting started on: 2016-07-13 07:42:17 status Completed 57331157 Provision started on: 2016-07-13 07:38:03 status PASS host: chromeos4-row7-rack3-host13, status: Provisioning, locked: False diagnosis: Working labels: ['bluetooth', 'ec:cros', 'graphics:gl', 'board_freq_mem:glimmer_1.8GHz_2GB', 'board:glimmer', 'glimmer', 'gpu_family:baytrail', 'audio_loopback_dongle', 'pool:bvt', 'cts_abi_x86', 'cts_abi_arm', 'servo', 'variant:glimmer', 'sku:glimmer_intel_celeron_n2930_2Gb', 'touchpad', 'internal_display', 'os:cros', 'power:battery', 'phase:PVT', 'storage:mmc', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'webcam', 'cros-version:glimmer-release/R54-8582.0.0'] Last 10 jobs within 3:18:00: 57332831 Cleanup started on: 2016-07-13 08:48:19 status PASS 69785944 glimmer-release/R54-8582.0.0/bvt-inline/security_DbusOwners started on: 2016-07-13 08:44:13 status Aborted 57332347 Reset started on: 2016-07-13 08:07:08 status PASS 69785853 glimmer-release/R54-8582.0.0/bvt-inline/security_SandboxLinuxUnittests started on: 2016-07-13 08:01:05 status Completed 57332087 Reset started on: 2016-07-13 07:58:51 status PASS 69785776 glimmer-release/R54-8582.0.0/bvt-inline/platform_CrosDisksDBus started on: 2016-07-13 07:54:20 status Completed 57331834 Reset started on: 2016-07-13 07:52:37 status PASS 69785748 glimmer-release/R54-8582.0.0/bvt-inline/security_ProfilePermissions started on: 2016-07-13 07:48:42 status Completed 57331593 Reset started on: 2016-07-13 07:47:00 status PASS 69785737 glimmer-release/R54-8582.0.0/bvt-inline/login_UserPolicyKeys started on: 2016-07-13 07:41:40 status Completed host: chromeos4-row7-rack2-host21, status: Provisioning, locked: False diagnosis: Working labels: ['graphics:gl', 'board_freq_mem:glimmer_1.8GHz_2GB', 'glimmer', 'gpu_family:baytrail', 'pool:bvt', 'bluetooth', 'ec:cros', 'board:glimmer', 'cts_abi_x86', 'cts_abi_arm', 'variant:glimmer', 'sku:glimmer_intel_celeron_n2930_2Gb', 'touchpad', 'servo', 'internal_display', 'os:cros', 'power:battery', 'phase:PVT', 'storage:mmc', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'webcam', 'hw_video_acc_enc_h264', 'cros-version:glimmer-release/R54-8582.0.0'] Last 10 jobs within 3:18:00: 57332771 Cleanup started on: 2016-07-13 08:47:11 status PASS 57332598 Reset started on: 2016-07-13 08:44:17 status FAIL 69785907 glimmer-release/R54-8582.0.0/bvt-inline/security_mprotect started on: 2016-07-13 08:04:47 status Aborted 57332222 Reset started on: 2016-07-13 08:02:40 status PASS 69785821 glimmer-release/R54-8582.0.0/bvt-inline/security_OpenSSLBlacklist started on: 2016-07-13 07:58:02 status Completed 57331958 Reset started on: 2016-07-13 07:55:05 status PASS 69785768 glimmer-release/R54-8582.0.0/bvt-inline/platform_CUPSDaemon started on: 2016-07-13 07:52:00 status Completed 57331727 Reset started on: 2016-07-13 07:50:10 status PASS 69785739 glimmer-release/R54-8582.0.0/bvt-inline/login_Cryptohome started on: 2016-07-13 07:45:21 status Completed 57331392 Reset started on: 2016-07-13 07:42:23 status PASS Reason: Suite job failed or provisioning failed. Output below this line is for buildbot consumption: @@@STEP_LINK@[Test-Logs]: Suite job: ABORT@http://cautotest/tko/retrieve_logs.cgi?job=/results/69455019-chromeos-test/@@@ @@@STEP_LINK@[Flake-Dashboard]: Suite job@https://wmatrix.googleplex.com/retry_teststats/?days_back=30&tests=Suite job@@@ Will return from run_suite with status: INFRA_FAILURE
,
Jul 14 2016
I was also looking at this problem and unfortunately didn't notice this bug. I was also puzzled by the lack of messages between the beginning of the swarming.py invocation, and the job killing 2+ hours later. Is it completely unrealistic to try to preserve chronological order of events in the logs? Another thing that might help here (and in other places) is a timestamp attached to the test log links, to get a better picture of times. In fact, would it be possible to aim at standardizing the time stamps, like for instance in the linux syslog?
,
Jul 14 2016
Can you start with the two recommendations I've made in comment #2? I asked to add --tags a few months ago.
,
Jul 14 2016
,
Jul 14 2016
#10 thank you M.A. for bringing back those suggestions to my attention!
,
Jul 18 2016
Passing to this week's deputy to figure out whose job it is to move this forward.
,
Jul 18 2016
Dan/Kevin can one of you do the changes outlined in CM#2. Shouldn't be too hard. Kevin you are familiar with swarming client so might be easier/quicker for you to handle.
,
Jul 19 2016
,
Jul 19 2016
Sure, filed crbug.com/629539 and added it as a blocker of this bug.
,
Jul 21 2016
Timeout from a swarming call failed a CQ run: https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_speedy-paladin/builds/2611/steps/HWTest%20%5Bbvt-cq%5D/logs/stdio The suite was actually created: http://cautotest/afe/#tab_id=view_job&object_id=70321957
,
Sep 11 2017
not happening for a while. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by sbasi@chromium.org
, Jul 13 2016