New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 755914
Closed: Aug 2017
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Sign in to add a comment

testReapAllInOrder: AssertionError: False is not true

Project Member Reported by, Aug 22 2017

Issue description

butterfly-release:3990 failed

Builders failed on: 
- butterfly-release:

chromite-0.0.2-r3285: ======================================================================
chromite-0.0.2-r3285: FAIL: testReapAllInOrder (chromite.lib.workqueue.tasks_unittest.ProcessPoolTaskManagerTests)
chromite-0.0.2-r3285: ----------------------------------------------------------------------
chromite-0.0.2-r3285: Traceback (most recent call last):
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/", line 158, in testReapAllInOrder
chromite-0.0.2-r3285:     self._ReapAll(True)
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/", line 118, in _ReapAll
chromite-0.0.2-r3285:     self._StartAll()
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/", line 105, in _StartAll
chromite-0.0.2-r3285:     self._StartTask(rqid)
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/", line 76, in _StartTask
chromite-0.0.2-r3285:     self.assertTrue(os.path.exists(rqfile))
chromite-0.0.2-r3285: AssertionError: False is not true
chromite-0.0.2-r3285: ----------------------------------------------------------------------

Reading the code...this looks really prone to failure:

# N.B. The latency numbers need to be generous for the sake of
# builders that may be loaded when they run here.


  def _StartTask(self, rqid):
    """Call the task manager's `StartTask()` method."""
    rqfile = os.path.join(self._temp_dir, rqid)
    self._task_manager.StartTask(rqid, rqfile)
    self._pending_tasks[rqid] = rqfile

Apparently, 0.5 seconds isn't long enough? It looks like this was even a concern on the code review... Maybe a looping timeout (to retry for longer, if needed) would at least be better, without having to do too much more invasive mockups.
And in case this was deemed a single flake, I see the same failure pop up on another build; it fails once, and succeeds after retry:

chromite-0.0.2-r3287: ======================================================================
chromite-0.0.2-r3287: FAIL: testTerminateThenRun (chromite.lib.workqueue.tasks_unittest.ProcessPoolTaskManagerTests)
chromite-0.0.2-r3287: ----------------------------------------------------------------------
chromite-0.0.2-r3287: Traceback (most recent call last):
chromite-0.0.2-r3287:   File "/mnt/host/source/chromite/lib/workqueue/", line 197, in testTerminateThenRun
chromite-0.0.2-r3287:     self._StartTask(self._REQUEST_IDS[1])
chromite-0.0.2-r3287:   File "/mnt/host/source/chromite/lib/workqueue/", line 76, in _StartTask
chromite-0.0.2-r3287:     self.assertTrue(os.path.exists(rqfile))
chromite-0.0.2-r3287: AssertionError: False is not true
chromite-0.0.2-r3287: ----------------------------------------------------------------------

On either build, it looks like there were other failure(s) that would have killed the build anyway, so it's not quite the long pole yet.
Labels: -Pri-2 Pri-1
Mergedinto: 755914
Status: Duplicate (was: Assigned)
I don't like flaky tests.

Sign in to add a comment