Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: Duplicate
Merged: issue 755914
Owner:
Closed: Aug 22
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment
testReapAllInOrder: AssertionError: False is not true
Project Member Reported by briannorris@chromium.org, Aug 22 Back to list
butterfly-release:3990 failed

Builders failed on: 
- butterfly-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/butterfly-release/3990


chromite-0.0.2-r3285: ======================================================================
chromite-0.0.2-r3285: FAIL: testReapAllInOrder (chromite.lib.workqueue.tasks_unittest.ProcessPoolTaskManagerTests)
chromite-0.0.2-r3285: ----------------------------------------------------------------------
chromite-0.0.2-r3285: Traceback (most recent call last):
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 158, in testReapAllInOrder
chromite-0.0.2-r3285:     self._ReapAll(True)
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 118, in _ReapAll
chromite-0.0.2-r3285:     self._StartAll()
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 105, in _StartAll
chromite-0.0.2-r3285:     self._StartTask(rqid)
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 76, in _StartTask
chromite-0.0.2-r3285:     self.assertTrue(os.path.exists(rqfile))
chromite-0.0.2-r3285: AssertionError: False is not true
chromite-0.0.2-r3285: 
chromite-0.0.2-r3285: ----------------------------------------------------------------------

Reading the code...this looks really prone to failure:

# N.B. The latency numbers need to be generous for the sake of
# builders that may be loaded when they run here.
_START_LATENCY = 0.5
_REAP_LATENCY = 1.0

...

  def _StartTask(self, rqid):
    """Call the task manager's `StartTask()` method."""
    rqfile = os.path.join(self._temp_dir, rqid)
    self._task_manager.StartTask(rqid, rqfile)
    self._pending_tasks[rqid] = rqfile
    time.sleep(_START_LATENCY)
    self.assertTrue(os.path.exists(rqfile))


Apparently, 0.5 seconds isn't long enough? It looks like this was even a concern on the code review... Maybe a looping timeout (to retry for longer, if needed) would at least be better, without having to do too much more invasive mockups.
 
Cc: pprabhu@chromium.org
And in case this was deemed a single flake, I see the same failure pop up on another build; it fails once, and succeeds after retry:

chromite-0.0.2-r3287: ======================================================================
chromite-0.0.2-r3287: FAIL: testTerminateThenRun (chromite.lib.workqueue.tasks_unittest.ProcessPoolTaskManagerTests)
chromite-0.0.2-r3287: ----------------------------------------------------------------------
chromite-0.0.2-r3287: Traceback (most recent call last):
chromite-0.0.2-r3287:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 197, in testTerminateThenRun
chromite-0.0.2-r3287:     self._StartTask(self._REQUEST_IDS[1])
chromite-0.0.2-r3287:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 76, in _StartTask
chromite-0.0.2-r3287:     self.assertTrue(os.path.exists(rqfile))
chromite-0.0.2-r3287: AssertionError: False is not true
chromite-0.0.2-r3287: 
chromite-0.0.2-r3287: ----------------------------------------------------------------------


https://uberchromegw.corp.google.com/i/chromeos/builders/eve-release/builds/847/steps/UnitTest/logs/stdio

On either build, it looks like there were other failure(s) that would have killed the build anyway, so it's not quite the long pole yet.
Labels: -Pri-2 Pri-1
Mergedinto: 755914
Status: Duplicate
I don't like flaky tests.
Sign in to add a comment