Hmmm, any way to get a better stack? Thanks!

Re #1: This failed on Fuchsia/ARM64 FYI, which is a Release builder so |symbol_level| should be zero; I'm actually a little surprised that we're getting symbols at all.

We could increase the |symbol_level| on our FYI bots but then we'd be running (much) bigger binaries, which will increase build & run times, etc.

That's unfortunate. On Windows we have symbols even for Release builds. Can't do much to diagnose failures othewise...

FWIW, this stack looks similar to  issue 813651 .

Maybe the worker is calling into a freed TaskTracker and the rest is garbage..? (shouldn't happen per my current mental model but eh)

 Issue 813651  has been merged into this issue.

I dug into this and other bugs to figure out why a worker could possibly outlive the pool/scheduler/TaskTracker (we always ensure that the pool/workers were joined in the destructor after all): 

TaskSchedulerImpl::~TaskSchedulerImpl() {
#if DCHECK_IS_ON()
  DCHECK(join_for_testing_returned_.IsSet());
#endif
}

I now have a theory: we rely on atomics in an incorrect way in one testing-only method.

That is SchedulerWorkerPoolImpl::DisallowWorkerCleanupForTesting() sets an atomic bit and workers are expected to not cleanup when they see that bit. However absolutely nothing in the atomic rules forces the other threads to see that value (atomics just specify that *once* they see the value, they're also guaranteed to see what happened before it was set (per usage of release/acquire barriers in AtomicFlag), but that's irrelevant here).

Because of this, a worker can get CanCleanup() == true after the main thread invoked SchedulerWorkerPoolImpl::DisallowWorkerCleanupForTesting(). In which case it will remove itself for |workers_| and will not be waited upon when JoinForTesting() is invoked on the main thread. And hence we have our worker outliving its pool..!

(working on fix now)

CL @ https://chromium-review.googlesource.com/#/c/chromium/src/+/929061 

Note to self: try to revert https://chromium-review.googlesource.com/925546 to re-enable some TaskScheduler tests after this is fixed.

Re #5: Based on your description this sounds like a duplicate of  issue 810464 , right?

Ah, yes, missed that one when digging for open bugs.. thanks!

Could this also fit  issue 806003 ?

Yes, thanks! Marked as duplicate. I think this encompasses all the flakes except the timing ones based on worker capacity.

Clearing up #5 after some offline discussion:

There is a synchronization issue between CanWorkerCleanupForTesting() and the state sync'ed by the SchedulerWorkerPoolImpl lock_. CanWorkerCleanupForTesting() could transition from true to false before CleanupLockRequired() occurs. 

However, in this unsynchronized case. It's not clear how the thread would make it to RunNextTask(). Once the cleanup path is triggered, the thread will exit out of its loop.

For the thread to continue to RunNextTask(), GetWork() will need to have returned a sequence, which means we haven't yet started the cleanup work.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fc40b1417b669d7df75b4d61491c521048027fda

commit fc40b1417b669d7df75b4d61491c521048027fda
Author: Gabriel Charette <gab@chromium.org>
Date: Thu Feb 22 10:02:04 2018

Make SchedulerWorkerPoolImpl::DisallowWorkerPoolCleanupForTesting() synchronized with SchedulerWorkers' cleanup step.

Atomics didn't guarantee synchronization as a worker could observe it
was allowed to cleanup, then be preempted, and later actually cleanup
after cleanup had been disallowed. Synchronizing on the pool's lock
prevents this.

I believe this is the source of most TaskScheduler flakes we've seen on
Fuschia. The actual test being blamed were also random because a leaky
worker would result in a user-after-free in the following test, not
the test at fault.

While this CL synchronizes cleanups with DisallowWorkerPoolCleanupForTesting(),
a follow-up CL will be required to ensure detached workers do not touch pool
state after cleanup (since they removed themselves from |workers_| they will
not be joined and hence using pool state can result in a use-after-free).

R=fdoray@chromium.org, robliao@chromium.org

Bug:  813278 
Change-Id: I455a2c8cff3aa9b1176567cc4a6409ff6ca6807d
Reviewed-on: https://chromium-review.googlesource.com/929061
Commit-Queue: Gabriel Charette <gab@chromium.org>
Reviewed-by: Robert Liao <robliao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538388}
[modify] https://crrev.com/fc40b1417b669d7df75b4d61491c521048027fda/base/task_scheduler/scheduler_worker_pool_impl.cc
[modify] https://crrev.com/fc40b1417b669d7df75b4d61491c521048027fda/base/task_scheduler/scheduler_worker_pool_impl.h

@#12: right, but I'm not trusting this stack much. This is a Release build and symbols may be misaligned. All we know is the worker is doing a use-after-free, everything else may be garbage IMO.