New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 730997 link

Starred by 1 user
Status: Duplicate
Merged: issue 730729
Owner:
dgarr...@chromium.org
Closed: Jun 2017
Cc:
bleung@chromium.org
dgarr...@chromium.org
mojahsu@chromium.org
█ mcchou@chromium.org
█ nxia@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

daisy_spring-release: The HWTest [sanity] stage failed: (15, 'Received signal 15; shutting down')

Project Member Reported by mojahsu@chromium.org, Jun 8 2017 
Link to build or pfq page.
https://uberchromegw.corp.google.com/i/chromeos/builders/daisy_spring-release/builds/2527

build # for that buildbot.
2527

Snippet of log that contains the failure.

----------------------
From Report stdio
************************************************************
** FAILED BUT FORGIVEN CanaryCompletion (0:00:25)
************************************************************

Failed in stage HWTest [sanity]:

<class 'chromite.lib.parallel.ProcessSilentTimeout'>: No output from <_BackgroundTask(_BackgroundTask-7:7:2, started)> for 8640 seconds


-------------------
From HWTest stdio

22:56:48: WARNING: RunCommand: gdb --nx -q -p 5010 -ex 'set prompt'
22:56:48: WARNING: (stdout):
Attaching to process 5010
>>> info proc all
>>> info threads
No threads.
>>> thread apply all py-list
>>> thread apply all py-bt
>>> thread apply all bt
>>> detach
quit

22:56:48: WARNING: (stderr):
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
No current process: you must name one.
The program is not being run.

22:57:18: WARNING: Killing 28528 (sig=15 SIGTERM)
22:57:20: INFO: Refreshing due to a 401 (attempt 1/2)
22:57:20: INFO: Refreshing access_token
22:56:48: ERROR: pre-kill notification (SIGXCPU); traceback:
  File "chromite/bin/cbuildbot", line 168, in <module>
    DoMain()
  File "chromite/bin/cbuildbot", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/cbuild/chromite/lib/commandline.py", line 837, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/cbuild/chromite/scripts/cbuildbot.py", line 1331, in main
    _RunBuildStagesWrapper(options, site_config, build_config)
  File "/b/cbuild/chromite/scripts/cbuildbot.py", line 248, in _RunBuildStagesWrapper
    if not builder.Run():
  File "/b/cbuild/chromite/cbuildbot/builders/generic_builders.py", line 327, in Run
    self.RunStages()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 502, in RunStages
    super(DistributedBuilder, self).RunStages()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 362, in RunStages
    self._RunDefaultTypeBuild()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 349, in _RunDefaultTypeBuild
    self.RunBuildStages()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 310, in RunBuildStages
    with parallel.BackgroundTaskRunner(task_runner) as queue:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 745, in BackgroundTaskRunner
    with _BackgroundTask.ParallelTasks(steps, halt_on_error=halt_on_error):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 532, in ParallelTasks
    task.start()
  File "/b/cbuild/chromite/lib/parallel.py", line 388, in start
    return multiprocessing.Process.start(self)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/b/cbuild/chromite/lib/parallel.py", line 399, in run
    errors = self._Run()
  File "/b/cbuild/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 146, in _RunBackgroundStagesForBoardAndMarkAsSuccessful
    self._RunBackgroundStagesForBoard(builder_run, board)
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 239, in _RunBackgroundStagesForBoard
    lambda: self._RunDebugSymbolStages(builder_run, board),
  File "/b/cbuild/chromite/lib/parallel.py", line 674, in RunParallelSteps
    halt_on_error=halt_on_error):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 532, in ParallelTasks
    task.start()
  File "/b/cbuild/chromite/lib/parallel.py", line 388, in start
    return multiprocessing.Process.start(self)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/b/cbuild/chromite/lib/parallel.py", line 399, in run
    errors = self._Run()
  File "/b/cbuild/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 238, in <lambda>
    lambda: self._RunHWTests(builder_run, board),
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 118, in _RunHWTests
    self._RunParallelStages(parallel_stages)
  File "/b/cbuild/chromite/cbuildbot/builders/generic_builders.py", line 119, in _RunParallelStages
    parallel.RunParallelSteps(steps)
  File "/b/cbuild/chromite/lib/parallel.py", line 674, in RunParallelSteps
    halt_on_error=halt_on_error):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 532, in ParallelTasks
    task.start()
  File "/b/cbuild/chromite/lib/parallel.py", line 388, in start
    return multiprocessing.Process.start(self)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/b/cbuild/chromite/lib/parallel.py", line 399, in run
    errors = self._Run()
  File "/b/cbuild/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/chromite/cbuildbot/stages/generic_stages.py", line 629, in Run
    self.PerformStage()
  File "/b/cbuild/chromite/cbuildbot/stages/test_stages.py", line 458, in PerformStage
    ('Build', build_id, 'BuildStage', self._build_stage_id)})
  File "/b/cbuild/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/commands.py", line 965, in RunHWTestSuite
    pass_hwtest = _HWTestWait(cmd, job_id, **swarming_args)
  File "/b/cbuild/chromite/cbuildbot/commands.py", line 1238, in _HWTestWait
    **kwargs)
  File "/b/cbuild/chromite/cbuildbot/swarming_lib.py", line 153, in RunSwarmingCommandWithRetries
    return retry_util.RetryCommand(RunSwarmingCommand, max_retry, *args, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 219, in RetryCommand
    return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 109, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/swarming_lib.py", line 92, in RunSwarmingCommand
    result = cros_build_lib.RunCommand(swarming_cmd, *args, **kwargs)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 613, in RunCommand
    (cmd_result.output, cmd_result.error) = proc.communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 796, in communicate
    self.wait()
  File "/usr/lib/python2.7/subprocess.py", line 1376, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
    return func(*args)

22:57:19: ERROR: Traceback (most recent call last):
  File "/b/cbuild/chromite/cbuildbot/stages/generic_stages.py", line 629, in Run
    self.PerformStage()
  File "/b/cbuild/chromite/cbuildbot/stages/test_stages.py", line 458, in PerformStage
    ('Build', build_id, 'BuildStage', self._build_stage_id)})
  File "/b/cbuild/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/commands.py", line 965, in RunHWTestSuite
    pass_hwtest = _HWTestWait(cmd, job_id, **swarming_args)
  File "/b/cbuild/chromite/cbuildbot/commands.py", line 1238, in _HWTestWait
    **kwargs)
  File "/b/cbuild/chromite/cbuildbot/swarming_lib.py", line 153, in RunSwarmingCommandWithRetries
    return retry_util.RetryCommand(RunSwarmingCommand, max_retry, *args, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 219, in RetryCommand
    return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 109, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/swarming_lib.py", line 92, in RunSwarmingCommand
    result = cros_build_lib.RunCommand(swarming_cmd, *args, **kwargs)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 613, in RunCommand
    (cmd_result.output, cmd_result.error) = proc.communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 796, in communicate
    self.wait()
  File "/usr/lib/python2.7/subprocess.py", line 1376, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
    return func(*args)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 360, in _KillChildProcess
    if not signals.RelaySignal(original_handler, signum, frame):
  File "/b/cbuild/chromite/lib/signals.py", line 26, in RelaySignal
    handler(signum, frame)
  File "/b/cbuild/chromite/lib/commandline.py", line 733, in _DefaultHandler
    signum, 'Received signal %i; shutting down' % (signum,))
_ShutDownException: (15, 'Received signal 15; shutting down')

22:57:19: INFO: Translating result (15, 'Received signal 15; shutting down') to fail.
22:57:19: INFO: Running cidb query on pid 28528, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7fb4f042b910>
22:57:20: INFO: Running cidb query on pid 28528, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7fb4f03df990>
************************************************************
** Finished Stage HWTest [sanity] - Wed, 07 Jun 2017 22:57:20 -0700 (PDT)
************************************************************
Traceback (most recent call last):
  File "/b/cbuild/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/chromite/cbuildbot/stages/generic_stages.py", line 629, in Run
    self.PerformStage()
  File "/b/cbuild/chromite/cbuildbot/stages/test_stages.py", line 458, in PerformStage
    ('Build', build_id, 'BuildStage', self._build_stage_id)})
  File "/b/cbuild/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/commands.py", line 965, in RunHWTestSuite
    pass_hwtest = _HWTestWait(cmd, job_id, **swarming_args)
  File "/b/cbuild/chromite/cbuildbot/commands.py", line 1238, in _HWTestWait
    **kwargs)
  File "/b/cbuild/chromite/cbuildbot/swarming_lib.py", line 153, in RunSwarmingCommandWithRetries
    return retry_util.RetryCommand(RunSwarmingCommand, max_retry, *args, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 219, in RetryCommand
    return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/chromite/lib/retry_util.py", line 109, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/chromite/cbuildbot/swarming_lib.py", line 92, in RunSwarmingCommand
    result = cros_build_lib.RunCommand(swarming_cmd, *args, **kwargs)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 613, in RunCommand
    (cmd_result.output, cmd_result.error) = proc.communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 796, in communicate
    self.wait()
  File "/usr/lib/python2.7/subprocess.py", line 1376, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
    return func(*args)
  File "/b/cbuild/chromite/lib/cros_build_lib.py", line 360, in _KillChildProcess
    if not signals.RelaySignal(original_handler, signum, frame):
  File "/b/cbuild/chromite/lib/signals.py", line 26, in RelaySignal
    handler(signum, frame)
  File "/b/cbuild/chromite/lib/commandline.py", line 733, in _DefaultHandler
    signum, 'Received signal %i; shutting down' % (signum,))
_ShutDownException: (15, 'Received signal 15; shutting down')
  File "chromite/bin/cbuildbot", line 168, in <module>
    DoMain()
  File "chromite/bin/cbuildbot", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/cbuild/chromite/lib/commandline.py", line 837, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/cbuild/chromite/scripts/cbuildbot.py", line 1331, in main
    _RunBuildStagesWrapper(options, site_config, build_config)
  File "/b/cbuild/chromite/scripts/cbuildbot.py", line 248, in _RunBuildStagesWrapper
    if not builder.Run():
  File "/b/cbuild/chromite/cbuildbot/builders/generic_builders.py", line 327, in Run
    self.RunStages()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 502, in RunStages
    super(DistributedBuilder, self).RunStages()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 362, in RunStages
    self._RunDefaultTypeBuild()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 349, in _RunDefaultTypeBuild
    self.RunBuildStages()
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 310, in RunBuildStages
    with parallel.BackgroundTaskRunner(task_runner) as queue:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 745, in BackgroundTaskRunner
    with _BackgroundTask.ParallelTasks(steps, halt_on_error=halt_on_error):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 532, in ParallelTasks
    task.start()
  File "/b/cbuild/chromite/lib/parallel.py", line 388, in start
    return multiprocessing.Process.start(self)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/b/cbuild/chromite/lib/parallel.py", line 399, in run
    errors = self._Run()
  File "/b/cbuild/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 146, in _RunBackgroundStagesForBoardAndMarkAsSuccessful
    self._RunBackgroundStagesForBoard(builder_run, board)
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 239, in _RunBackgroundStagesForBoard
    lambda: self._RunDebugSymbolStages(builder_run, board),
  File "/b/cbuild/chromite/lib/parallel.py", line 674, in RunParallelSteps
    halt_on_error=halt_on_error):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 532, in ParallelTasks
    task.start()
  File "/b/cbuild/chromite/lib/parallel.py", line 388, in start
    return multiprocessing.Process.start(self)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/b/cbuild/chromite/lib/parallel.py", line 399, in run
    errors = self._Run()
  File "/b/cbuild/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 238, in <lambda>
    lambda: self._RunHWTests(builder_run, board),
  File "/b/cbuild/chromite/cbuildbot/builders/simple_builders.py", line 118, in _RunHWTests
    self._RunParallelStages(parallel_stages)
  File "/b/cbuild/chromite/cbuildbot/builders/generic_builders.py", line 119, in _RunParallelStages
    parallel.RunParallelSteps(steps)
  File "/b/cbuild/chromite/lib/parallel.py", line 675, in RunParallelSteps
    pass
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 546, in ParallelTasks
    task_errors = task.Wait()
  File "/b/cbuild/chromite/lib/parallel.py", line 361, in Wait
    traceback.print_stack()
22:57:20: WARNING: No output from <_BackgroundTask(_BackgroundTask-7:7:2, started)> for 8640 seconds
22:57:20: INFO: 
22:57:20: ERROR: BaseException in _RunParallelStages <class 'chromite.lib.parallel.ProcessSilentTimeout'>: No output from <_BackgroundTask(_BackgroundTask-7:7:2, started)> for 8640 seconds
Traceback (most recent call last):
  File "/b/cbuild/chromite/cbuildbot/builders/generic_builders.py", line 119, in _RunParallelStages
    parallel.RunParallelSteps(steps)
  File "/b/cbuild/chromite/lib/parallel.py", line 678, in RunParallelSteps
    return [queue.get_nowait() for queue in queues]
  File "/b/cbuild/chromite/lib/parallel.py", line 675, in RunParallelSteps
    pass
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/chromite/lib/parallel.py", line 561, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
BackgroundFailure: <class 'chromite.lib.parallel.ProcessSilentTimeout'>: No output from <_BackgroundTask(_BackgroundTask-7:7:2, started)> for 8640 seconds

22:57:20: INFO: Running cidb query on pid 27886, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x7fb4f0507650; Select object>
22:57:20: INFO: Running cidb query on pid 27886, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x7fb4f0507490; Select object>
20:27:55: INFO: Created cidb engine bot@130.211.191.11 for pid 27905
20:27:55: INFO: Running cidb query on pid 27905, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7fb4ebcb5110>

Comment 1 by nxia@chromium.org, Jun 9 2017

Cc: nxia@chromium.org dgarr...@chromium.org
20:32:46: INFO: RunCommand: /b/cbuild/chromite/third_party/swarming.client/swarming.py run --swarming chromeos-proxy.appspot.com --task-summary-json ....

22:56:47: WARNING: Killing tasks: [<_BackgroundTask(_BackgroundTask-7:7:2, started)>]


BackgroundFailure: <class 'chromite.lib.parallel.ProcessSilentTimeout'>: No output from <_BackgroundTask(_BackgroundTask-7:7:2, started)> for 8640 seconds

it's timed out as no output or result from the command for 8640 seconds. 
+ primary deputy. did this happen on CQs this week?

Comment 2 by aut...@google.com, Jun 12 2017

Owner: dgarr...@chromium.org

Comment 3 by dgarr...@chromium.org, Jun 14 2017

Mergedinto: 730729
Status: Duplicate (was: Untriaged)
Yep, it's still happening.
►

Sign in to add a comment