butterfly-release:3990 failed

Builders failed on: 
- butterfly-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/butterfly-release/3990


chromite-0.0.2-r3285: ======================================================================
chromite-0.0.2-r3285: FAIL: testReapAllInOrder (chromite.lib.workqueue.tasks_unittest.ProcessPoolTaskManagerTests)
chromite-0.0.2-r3285: ----------------------------------------------------------------------
chromite-0.0.2-r3285: Traceback (most recent call last):
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 158, in testReapAllInOrder
chromite-0.0.2-r3285:     self._ReapAll(True)
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 118, in _ReapAll
chromite-0.0.2-r3285:     self._StartAll()
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 105, in _StartAll
chromite-0.0.2-r3285:     self._StartTask(rqid)
chromite-0.0.2-r3285:   File "/mnt/host/source/chromite/lib/workqueue/tasks_unittest.py", line 76, in _StartTask
chromite-0.0.2-r3285:     self.assertTrue(os.path.exists(rqfile))
chromite-0.0.2-r3285: AssertionError: False is not true
chromite-0.0.2-r3285: 
chromite-0.0.2-r3285: ----------------------------------------------------------------------

Reading the code...this looks really prone to failure:

# N.B. The latency numbers need to be generous for the sake of
# builders that may be loaded when they run here.
_START_LATENCY = 0.5
_REAP_LATENCY = 1.0

...

  def _StartTask(self, rqid):
    """Call the task manager's `StartTask()` method."""
    rqfile = os.path.join(self._temp_dir, rqid)
    self._task_manager.StartTask(rqid, rqfile)
    self._pending_tasks[rqid] = rqfile
    time.sleep(_START_LATENCY)
    self.assertTrue(os.path.exists(rqfile))


Apparently, 0.5 seconds isn't long enough? It looks like this was even a concern on the code review... Maybe a looping timeout (to retry for longer, if needed) would at least be better, without having to do too much more invasive mockups.