New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 713289 link

Starred by 1 user
Status: Duplicate
Owner:
x...@chromium.org
Closed: Apr 2017
Cc:
akes...@chromium.org
█ zhihongyu@chromium.org
█ nxia@chromium.org
steve...@chromium.org
ihf@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: Bug



Sign in to add a comment

veyron_minnie-chrome-pfq failed at HWTest [arc-bvt-cq]

Project Member Reported by x...@chromium.org, Apr 19 2017 
It showed up on veyron_minnie-chrome-pfq starting from https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/1018 intermittently. (Sometimes it was covered by other failures)

Selected error messages:
************************************************************
** Finished Stage HWTest [arc-bvt-cq] - Wed, 19 Apr 2017 03:42:56 -0700 (PDT)
************************************************************
Traceback (most recent call last):
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 628, in Run
    self.PerformStage()
  File "/b/cbuild/repository/chromite/cbuildbot/stages/test_stages.py", line 416, in PerformStage
    ('Build', build_id, 'BuildStage', self._build_stage_id)})
  File "/b/cbuild/repository/chromite/lib/failures_lib.py", line 190, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/repository/chromite/cbuildbot/commands.py", line 1019, in RunHWTestSuite
    _HWTestWait(cmd, job_id, **swarming_args)
  File "/b/cbuild/repository/chromite/cbuildbot/commands.py", line 1263, in _HWTestWait
    **kwargs)
  File "/b/cbuild/repository/chromite/cbuildbot/swarming_lib.py", line 153, in RunSwarmingCommandWithRetries
    return retry_util.RetryCommand(RunSwarmingCommand, max_retry, *args, **kwargs)
  File "/b/cbuild/repository/chromite/lib/retry_util.py", line 224, in RetryCommand
    return GenericRetry(ShouldRetry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/repository/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/repository/chromite/cbuildbot/swarming_lib.py", line 92, in RunSwarmingCommand
    result = cros_build_lib.RunCommand(swarming_cmd, *args, **kwargs)
  File "/b/cbuild/repository/chromite/lib/cros_build_lib.py", line 593, in RunCommand
    (cmd_result.output, cmd_result.error) = proc.communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 796, in communicate
    self.wait()
  File "/usr/lib/python2.7/subprocess.py", line 1376, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call
    return func(*args)
  File "/b/cbuild/repository/chromite/lib/cros_build_lib.py", line 340, in _KillChildProcess
    if not signals.RelaySignal(original_handler, signum, frame):
  File "/b/cbuild/repository/chromite/lib/signals.py", line 26, in RelaySignal
    handler(signum, frame)
  File "/b/cbuild/repository/chromite/lib/commandline.py", line 733, in _DefaultHandler
    signum, 'Received signal %i; shutting down' % (signum,))
_ShutDownException: (15, 'Received signal 15; shutting down')

@@@STEP_FAILURE@@@
  File "chromite/bin/cbuildbot", line 168, in <module>
    DoMain()
  File "chromite/bin/cbuildbot", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/cbuild/repository/chromite/lib/commandline.py", line 837, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/cbuild/repository/chromite/scripts/cbuildbot.py", line 1331, in main
    _RunBuildStagesWrapper(options, site_config, build_config)
  File "/b/cbuild/repository/chromite/scripts/cbuildbot.py", line 248, in _RunBuildStagesWrapper
    if not builder.Run():
  File "/b/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 327, in Run
    self.RunStages()
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 502, in RunStages
    super(DistributedBuilder, self).RunStages()
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 362, in RunStages
    self._RunDefaultTypeBuild()
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 349, in _RunDefaultTypeBuild
    self.RunBuildStages()
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 310, in RunBuildStages
    with parallel.BackgroundTaskRunner(task_runner) as queue:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 745, in BackgroundTaskRunner
    with _BackgroundTask.ParallelTasks(steps, halt_on_error=halt_on_error):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 532, in ParallelTasks
    task.start()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 388, in start
    return multiprocessing.Process.start(self)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 399, in run
    errors = self._Run()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 146, in _RunBackgroundStagesForBoardAndMarkAsSuccessful
    self._RunBackgroundStagesForBoard(builder_run, board)
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 239, in _RunBackgroundStagesForBoard
    lambda: self._RunDebugSymbolStages(builder_run, board),
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 674, in RunParallelSteps
    halt_on_error=halt_on_error):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 532, in ParallelTasks
    task.start()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 388, in start
    return multiprocessing.Process.start(self)
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 399, in run
    errors = self._Run()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 238, in <lambda>
    lambda: self._RunHWTests(builder_run, board),
  File "/b/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 122, in _RunHWTests
    self._RunParallelStages(parallel_stages)
  File "/b/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 119, in _RunParallelStages
    parallel.RunParallelSteps(steps)
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 675, in RunParallelSteps
    pass
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 546, in ParallelTasks
    task_errors = task.Wait()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 361, in Wait
    traceback.print_stack()
03:42:56: WARNING: No output from <_BackgroundTask(_BackgroundTask-7:7:4, started)> for 8640 seconds
03:42:56: INFO: 
03:42:56: ERROR: BaseException in _RunParallelStages <class 'chromite.lib.parallel.ProcessSilentTimeout'>: No output from <_BackgroundTask(_BackgroundTask-7:7:4, started)> for 8640 seconds
Traceback (most recent call last):
  File "/b/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 119, in _RunParallelStages
    parallel.RunParallelSteps(steps)
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 678, in RunParallelSteps
    return [queue.get_nowait() for queue in queues]
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 675, in RunParallelSteps
    pass
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/repository/chromite/lib/parallel.py", line 561, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
BackgroundFailure: <class 'chromite.lib.parallel.ProcessSilentTimeout'>: No output from <_BackgroundTask(_BackgroundTask-7:7:4, started)> for 8640 seconds

Comment 1 by x...@chromium.org, Apr 19 2017 

It seems the veyron_minnie-chrome-pfq has reached the timeout when running HWTest [arc-bvt-cq] so was aborted. 

Compared the time used in the latest-successful-run (https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/1020) and the latest-failed-run (https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-chrome-pfq/builds/1040), there was a big difference (50 mins vs 2 hrs 24 mins).

Actually I noticed this failure shows up on the build intermittently, for example, we had an exact same failure on Mar 26 but soon the failure was gone. But recently it happened more and more often (from run #1026 to run #1031 we had 6 consecutive timeout failures). 

Anyone has any clue?

Comment 2 by x...@chromium.org, Apr 22 2017

Mergedinto: 712102
Status: Duplicate (was: Assigned)
I don't see this failure any more. I think it maybe a dup of Issue 712102. Merge into it.
►

Sign in to add a comment