New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 682352 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner:
Last visit > 30 days ago
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

critical:root:no test configurations generated, nothing to do

Project Member Reported by semenzato@chromium.org, Jan 18 2017

Issue description

https://uberchromegw.corp.google.com/i/chromeos/builders/arkham-release/builds/772/steps/AUTest%20%5Bau%5D/logs/stdio

I am copying the entire log here.  I don't see any clue as to why no test configurations were generated.

@@@BUILD_STEP@AUTest [au]@@@
************************************************************
@@@STEP_LINK@stdout-->stdio@https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Farkham-release%2F772%2F%2B%2Frecipes%2Fsteps%2FAUTest__au_%2F0%2Fstdout@@@
** Start Stage AUTest [au] - Wed, 18 Jan 2017 03:07:15 -0800 (PST)
** 
** Stage for au hw test suites that requires special pre-processing.
************************************************************
03:07:15: INFO: Created cidb engine bot@173.194.81.53 for pid 27736
03:07:15: INFO: Running cidb query on pid 27736, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7ff3df2c4ad0>
03:07:16: INFO: Waiting up to forever for payloads and test artifacts ...
Preconditions for the stage successfully met. Beginning to execute stage...
03:07:16: INFO: Running cidb query on pid 27736, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7ff3df2cf150>
03:07:16: INFO: RunCommand: site_utils/autoupdate/full_release_test.py --npo --nmo --dump --dump_dir /tmp/cbuildbot-tmppobNum/tmpdw4Zbz/autotest/au_control_files --archive_url gs://chromeos-image-archive/arkham-release/R57-9196.0.0 9196.0.0 arkham '--log=debug' in /b/cbuild/internal_master/src/third_party/autotest/files
CRITICAL:root:no test configurations generated, nothing to do

@@@STEP_FAILURE@@@
03:07:24: ERROR: Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/generic_stages.py", line 606, in Run
    self.PerformStage()
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/test_stages.py", line 420, in PerformStage
    self.version, self.upload_url)
  File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 2069, in BuildAUTestTarball
    cros_build_lib.RunCommand(cmd, env=run_env, cwd=cwd)
  File "/b/cbuild/internal_master/chromite/lib/cros_build_lib.py", line 625, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: site_utils/autoupdate/full_release_test.py --npo --nmo --dump --dump_dir /tmp/cbuildbot-tmppobNum/tmpdw4Zbz/autotest/au_control_files --archive_url gs://chromeos-image-archive/arkham-release/R57-9196.0.0 9196.0.0 arkham '--log=debug'
cmd=['site_utils/autoupdate/full_release_test.py', '--npo', '--nmo', '--dump', '--dump_dir', '/tmp/cbuildbot-tmppobNum/tmpdw4Zbz/autotest/au_control_files', '--archive_url', 'gs://chromeos-image-archive/arkham-release/R57-9196.0.0', '9196.0.0', 'arkham', '--log=debug'], cwd=/b/cbuild/internal_master/src/third_party/autotest/files

03:07:24: INFO: Running cidb query on pid 27736, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7ff3dfaad510>
03:07:24: INFO: Running cidb query on pid 27736, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7ff3dfaad890>
************************************************************
** Finished Stage AUTest [au] - Wed, 18 Jan 2017 03:07:24 -0800 (PST)
************************************************************
03:07:25: ERROR: BaseException in _RunParallelStages <class 'chromite.lib.failures_lib.StepFailure'>: 
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/generic_stages.py", line 625, in Run
    raise failures_lib.StepFailure()
StepFailure
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/cbuildbot/builders/generic_builders.py", line 118, in _RunParallelStages
    parallel.RunParallelSteps(steps)
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 678, in RunParallelSteps
    return [queue.get_nowait() for queue in queues]
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 675, in RunParallelSteps
    pass
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 561, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
BackgroundFailure: <class 'chromite.lib.failures_lib.StepFailure'>: 
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/generic_stages.py", line 625, in Run
    raise failures_lib.StepFailure()
StepFailure

03:07:25: INFO: Running cidb query on pid 14453, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x7ff3df2e3b10; Select object>
03:07:25: INFO: Running cidb query on pid 14453, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7ff3df2e3c90>
03:07:25: INFO: Running cidb query on pid 14453, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7ff3df2e3d10>
03:07:25: INFO: Running cidb query on pid 14453, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x7ff3df2e3cd0; Select object>
03:38:49: INFO: Running cidb query on pid 7183, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7ff3e5314090>
03:38:54: INFO: cidb query succeeded after 1 retries
 
To clarify: there are two bugs here.  One is that no test configurations were generated.  The other is that I have no idea why.  If needed we can split this into two bugs.

Also see issue 639314.  We need to catch and log GS failures.

If GS fails by quietly returning an empty file, we need to figure that out too, possibly by making that file always non-empty.

Comment 3 by dshi@chromium.org, Jan 19 2017

Cc: akes...@chromium.org pprabhu@chromium.org dgarr...@chromium.org
I don't see any obvious reason, add some build people to comment.
This is almost surely issue 639314

I was able to reproduce this locally by running: 
site_utils/autoupdate/full_release_test.py --npo --nmo --dump --dump_dir /tmp/newdir --archive_url gs://chromeos-image-archive/arkham-release/R57-9196.0.0 9196.0.0 arkham --log=debug

But then, when I tried to trace the python calls, the error disappeared, and the same command no longer fails for me. That smells like a silent GS list returning empty.

The difference I saw in passing / failing runs:

A passing run looked like this:
pprabhu@pprabhu:~/chromiumos/src/third_party/autotest/files$ site_utils/autoupdate/full_release_test.py --npo --nmo --dump --dump_dir /tmp/newdir --archive_url gs://chromeos-image-archive/arkham-release/R57-9196.0.0 9196.0.0 arkham --log=debug
DEBUG:root:Searching for pattern *_delta_* from url gs://chromeos-image-archive/arkham-release/R57-9196.0.0
DEBUG:root:Searching for pattern *_full_* from url gs://chromeos-image-archive/arkham-release/R57-9196.0.0
INFO:root:dumped control file for test [arkham/npo/delta/9196.0.0 -> 9196.0.0] to /tmp/newdir/arkham/control.npo_delta_9196.0.0

A failing run, OTOH, never even tried to search for the *_full_* payloads, and died before that.
Mergedinto: 639314
Status: Duplicate (was: Untriaged)

Sign in to add a comment