New issue
Advanced search Search tips

Issue 859114 link

Starred by 2 users

Issue metadata

Status: Duplicate
Merged: issue 859031
Owner:
Closed: Jun 2018
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

amd64-generic-telemetry failed at UnitTest stage

Project Member Reported by x...@chromium.org, Jun 29 2018

Issue description

The failure started from https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8942407590562653392

Selected error message:
Packages failed:
	net-wireless/bluez-5.44-r140
	chromeos-base/chromite-0.0.2-r3984
00:31:31: ERROR: Unittests failed.
cros_run_unit_tests: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/bin/cros_run_unit_tests", line 169, in <module>
    DoMain()
  File "/mnt/host/source/chromite/bin/cros_run_unit_tests", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/scripts/cros_run_unit_tests.py", line 103, in main
    jobs=min(10, multiprocessing.cpu_count()))
  File "/mnt/host/source/chromite/lib/chroot_util.py", line 187, in RunUnittests
    cros_build_lib.SudoRunCommand(command, extra_env=env, mute_output=False)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 325, in SudoRunCommand
    return RunCommand(sudo_cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 669, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: sudo 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpkS_c4S' 'FEATURES=test' 'PKGDIR=/build/amd64-generic/test-packages' -- /mnt/host/source/chromite/bin/parallel_emerge '--sysroot=/build/amd64-generic' '--jobs=10' sys-apps/rootdev chromeos-base/vpd chromeos-base/libbrillo chromeos-base/vboot_reference chromeos-base/p2p chromeos-base/chaps chromeos-base/lorgnette chromeos-base/ippusb_manager chromeos-base/google-breakpad chromeos-base/minijail chromeos-base/permission_broker chromeos-base/chromeos-config-tools chromeos-base/crosvm chromeos-base/authpolicy net-misc/modemmanager-next chromeos-base/chromeos-login chromeos-base/crosdns chromeos-base/chromeos-trim chromeos-base/libweave chromeos-base/chromeos-init chromeos-base/bootstat chromeos-base/chromeos-installer chromeos-base/run_oci sys-apps/flashmap chromeos-base/libcontainer chromeos-base/quipper dev-util/puffin dev-util/hdctools net-wireless/bluez chromeos-base/libchromeos-ui chromeos-base/imageloader chromeos-base/vpn-manager media-sound/adhd net-libs/libmbim chromeos-base/chromite chromeos-base/smbprovider chromeos-base/bluetooth chromeos-base/gestures chromeos-base/crash-reporter chromeos-base/debugd chromeos-base/power_manager chromeos-base/chromeos-firmware-null chromeos-base/autotest chromeos-base/libpasswordprovider chromeos-base/verity chromeos-base/cryptohome chromeos-base/mist chromeos-base/update_engine chromeos-base/regions dev-util/bsdiff chromeos-base/shill chromeos-base/buffet chromeos-base/factory_installer chromeos-base/chromeos-storage-info chromeos-base/memd chromeos-base/chromeos-imageburner chromeos-base/mtpd chromeos-base/metrics chromeos-base/ci-results-archiver chromeos-base/toolchain-tests chromeos-base/vm_host_tools chromeos-base/crosh chromeos-base/cros-disks
cmd=['sudo', 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpkS_c4S', 'FEATURES=test', 'PKGDIR=/build/amd64-generic/test-packages', '--', '/mnt/host/source/chromite/bin/parallel_emerge', '--sysroot=/build/amd64-generic', '--jobs=10', 'sys-apps/rootdev', 'chromeos-base/vpd', 'chromeos-base/libbrillo', 'chromeos-base/vboot_reference', 'chromeos-base/p2p', 'chromeos-base/chaps', 'chromeos-base/lorgnette', 'chromeos-base/ippusb_manager', 'chromeos-base/google-breakpad', 'chromeos-base/minijail', 'chromeos-base/permission_broker', 'chromeos-base/chromeos-config-tools', 'chromeos-base/crosvm', 'chromeos-base/authpolicy', 'net-misc/modemmanager-next', 'chromeos-base/chromeos-login', 'chromeos-base/crosdns', 'chromeos-base/chromeos-trim', 'chromeos-base/libweave', 'chromeos-base/chromeos-init', 'chromeos-base/bootstat', 'chromeos-base/chromeos-installer', 'chromeos-base/run_oci', 'sys-apps/flashmap', 'chromeos-base/libcontainer', 'chromeos-base/quipper', 'dev-util/puffin', 'dev-util/hdctools', 'net-wireless/bluez', 'chromeos-base/libchromeos-ui', 'chromeos-base/imageloader', 'chromeos-base/vpn-manager', 'media-sound/adhd', 'net-libs/libmbim', 'chromeos-base/chromite', 'chromeos-base/smbprovider', 'chromeos-base/bluetooth', 'chromeos-base/gestures', 'chromeos-base/crash-reporter', 'chromeos-base/debugd', 'chromeos-base/power_manager', 'chromeos-base/chromeos-firmware-null', 'chromeos-base/autotest', 'chromeos-base/libpasswordprovider', 'chromeos-base/verity', 'chromeos-base/cryptohome', 'chromeos-base/mist', 'chromeos-base/update_engine', 'chromeos-base/regions', 'dev-util/bsdiff', 'chromeos-base/shill', 'chromeos-base/buffet', 'chromeos-base/factory_installer', 'chromeos-base/chromeos-storage-info', 'chromeos-base/memd', 'chromeos-base/chromeos-imageburner', 'chromeos-base/mtpd', 'chromeos-base/metrics', 'chromeos-base/ci-results-archiver', 'chromeos-base/toolchain-tests', 'chromeos-base/vm_host_tools', 'chromeos-base/crosh', 'chromeos-base/cros-disks']
00:31:31: ERROR: 
return code: 1; command: /b/swarming/w/ir/cache/cbuild/repository/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpkS_c4S' -- /mnt/host/source/chromite/bin/cros_run_unit_tests '--board=amd64-generic'
cmd=['/b/swarming/w/ir/cache/cbuild/repository/chromite/bin/cros_sdk', 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpkS_c4S', '--', '/mnt/host/source/chromite/bin/cros_run_unit_tests', u'--board=amd64-generic'], cwd=/b/swarming/w/ir/cache/cbuild/repository, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/tmp/tmpkS_c4S'}
00:31:31: ERROR: Packages failed in /mnt/host/source/chromite/bin/cros_run_unit_tests: chromeos-base/chromite net-wireless/bluez
00:31:31: INFO: Translating result Packages failed in /mnt/host/source/chromite/bin/cros_run_unit_tests: chromeos-base/chromite net-wireless/bluez to fail.
 

Comment 1 by x...@chromium.org, Jun 29 2018

Cc: achuith@chromium.org
achuith@, do you recall any changes that's made to telemetry recently?

Comment 2 by x...@chromium.org, Jun 29 2018

Cc: vapier@chromium.org
Looking more deeply into the log, it looks like a lot of tests in chromite.scripts.cros_portage_upgrade_unittest failed. See a few examples:
chromite-0.0.2-r3984: ERROR: [chromite.scripts.cros_portage_upgrade_unittest] MainTest.testFlowStatusReportOneBoard
chromite-0.0.2-r3984: Test main flow for basic one-board status report.
chromite-0.0.2-r3984: ----------------------------------------------------------------------
chromite-0.0.2-r3984: Traceback (most recent call last):
chromite-0.0.2-r3984:   File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 280, in _stacked_tearDown
chromite-0.0.2-r3984:     target(obj)
chromite-0.0.2-r3984:   File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 1558, in tearDown
chromite-0.0.2-r3984:     self.mox.VerifyAll()
chromite-0.0.2-r3984:   File "/usr/lib64/python2.7/site-packages/mox.py", line 284, in VerifyAll
chromite-0.0.2-r3984:     mock_obj._Verify()
chromite-0.0.2-r3984:   File "/usr/lib64/python2.7/site-packages/mox.py", line 504, in _Verify
chromite-0.0.2-r3984:     raise ExpectedMethodCallsError(self._expected_calls_queue)
chromite-0.0.2-r3984: ExpectedMethodCallsError: Verify: Expected methods never called:
chromite-0.0.2-r3984:   0.  Stub for <unbound method Upgrader.PreRunChecks>.__call__() -> None
chromite-0.0.2-r3984: 
chromite-0.0.2-r3984: ======================================================================
chromite-0.0.2-r3984: ERROR: [chromite.scripts.cros_portage_upgrade_unittest] MainTest.testFlowStatusReportOneBoardNotSetUp
chromite-0.0.2-r3984: Test main flow for basic one-board status report.
chromite-0.0.2-r3984: ----------------------------------------------------------------------
chromite-0.0.2-r3984: Traceback (most recent call last):
chromite-0.0.2-r3984:   File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 280, in _stacked_tearDown
chromite-0.0.2-r3984:     target(obj)
chromite-0.0.2-r3984:   File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 1558, in tearDown
chromite-0.0.2-r3984:     self.mox.VerifyAll()
chromite-0.0.2-r3984:   File "/usr/lib64/python2.7/site-packages/mox.py", line 284, in VerifyAll
chromite-0.0.2-r3984:     mock_obj._Verify()
chromite-0.0.2-r3984:   File "/usr/lib64/python2.7/site-packages/mox.py", line 504, in _Verify
chromite-0.0.2-r3984:     raise ExpectedMethodCallsError(self._expected_calls_queue)
chromite-0.0.2-r3984: ExpectedMethodCallsError: Verify: Expected methods never called:
chromite-0.0.2-r3984:   0.  Stub for <unbound method Upgrader.PreRunChecks>.__call__() -> None
chromite-0.0.2-r3984: 

vapier@, I noticed you landed several changes to cros_portage_upgrade, do you think it might be caused by one of your changes? 

At the same time, I'll try to see if I can repro it locally.

Comment 3 by x...@chromium.org, Jun 29 2018

Looks like all unittests in cros_portage_upgrade has been disabled in  Issue 858925 , I'll wait and see if the bot cycles back to green.

Comment 4 by vapier@chromium.org, Jun 29 2018

Mergedinto: 859031
Owner: vapier@chromium.org
Status: Duplicate (was: Untriaged)
should be fixed in ToT
We should probably get rid of the UnitTest stage for this bot 

Sign in to add a comment