This is a serious correctness issue that is possibly a root cause of persistent flakiness on the waterfall, per  Issue 609252 .

Sami, as TL of the Blink Scheduling effort, please find someone to investigate this. Thank you.

I'm getting a slightly different error here -- still task runner related:

    #0 0x7fef6ed2909e  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libblink_platform.so+0x70e09e)
    #1 0x7fef6ed09e77  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libblink_platform.so+0x6eee77)
    #2 0x7fef6ed05d0e  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libblink_platform.so+0x6ead0e)
    #3 0x7fef7f6602c1  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libbase.so+0xe62c1)
    #4 0x7fef7f6cb344  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libbase.so+0x151344)
    #5 0x7fef7f6cbfb5  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libbase.so+0x151fb5)
    #6 0x7fef7f6cd618  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libbase.so+0x153618)
    #7 0x7fef7f6d1e41  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libbase.so+0x157e41)
    #8 0x7fef7f73e666  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libbase.so+0x1c4666)
    #9 0x7fef7a87a29f  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libcontent.so+0x2dd629f)
    #10 0x7fef7abd393c  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libcontent.so+0x312f93c)
    #11 0x7fef7abd67dc  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libcontent.so+0x31327dc)
    #12 0x7fef7abd2c1a  (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libcontent.so+0x312ec1a)
    #13 0x7fef80f5f573  (/usr/local/google/code3/clankium-master/src/out_linux/Release/chrome+0x1196573)
    #14 0x7fef67ab3f44  (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

AddressSanitizer can not describe address in more detail (wild memory access suspected).
SUMMARY: AddressSanitizer: use-after-poison (/usr/local/google/code3/clankium-master/src/out_linux/Release/./libblink_platform.so+0x70e09e) 
Shadow bytes around the buggy address:
  0x0fdad7ae49b0: f7 f7 f7 f7 f7 00 00 00 00 00 00 00 00 00 f7 f7
  0x0fdad7ae49c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fdad7ae49d0: f7 f7 f7 f7 f7 00 00 00 00 00 00 00 00 f7 00 00
  0x0fdad7ae49e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 f7 00 00
  0x0fdad7ae49f0: 00 00 00 00 00 00 00 f7 00 00 00 00 00 00 00 00
=>0x0fdad7ae4a00: 00 00 00 00 f7 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7
  0x0fdad7ae4a10: f7 f7 f7 f7 f7 f7 f7 00 00 00 00 00 00 00 00 f7
  0x0fdad7ae4a20: 00 00 00 00 00 00 00 00 f7 00 00 00 00 00 00 00
  0x0fdad7ae4a30: 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fdad7ae4a40: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fdad7ae4a50: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

Which symbolizes to:

    #0 0x7fef6ed2909e in ?? third_party/WebKit/Source/platform/scheduler/child/webthread_base.cc:32:16
    #1 0x7fef6ed09e77 in ProcessTaskFromWorkQueue third_party/WebKit/Source/platform/scheduler/base/task_queue_manager.cc:346:5
    #2 0x7fef6ed05d0e in DoWork third_party/WebKit/Source/platform/scheduler/base/task_queue_manager.cc:234:13
    #3 0x7fef7f6602c1 in Run base/callback.h:64:12
    #4 0x7fef7f6602c1 in RunTask base/debug/task_annotator.cc:54:0
    #5 0x7fef7f6cb344 in RunTask base/message_loop/message_loop.cc:487:19
    #6 0x7fef7f6cbfb5 in DeferOrRunPendingTask base/message_loop/message_loop.cc:496:5
    #7 0x7fef7f6cd618 in DoDelayedWork base/message_loop/message_loop.cc:659:10
    #8 0x7fef7f6d1e41 in Run base/message_loop/message_pump_default.cc:39:27
    #9 0x7fef7f73e666 in Run base/run_loop.cc:35:10
    #10 0x7fef7a87a29f in RendererMain content/renderer/renderer_main.cc:198:23
    #11 0x7fef7abd393c in RunZygote content/app/content_main_runner.cc:343:14
    #12 0x7fef7abd67dc in Run content/app/content_main_runner.cc:786:12
    #13 0x7fef7abd2c1a in ContentMain content/app/content_main.cc:20:28
    #14 0x7fef80f5f573 in ChromeMain chrome/app/chrome_main.cc:85:12
    #15 0x7fef67ab3f44 in __libc_start_main /build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c:287:0

I'll try adding more instrumentation next.

With some help from __asan_address_is_poisoned() I was able to narrow this down to the following WebThread::TaskObserver (call stack when the observer is registered):

#0 0x7f4f077111e1 __interceptor_backtrace
#1 0x7f4f05e95c03 base::debug::StackTrace::StackTrace()
#2 0x7f4ef555fdc3 blink::scheduler::WebThreadBase::addTaskObserver()
#3 0x7f4ee6fa6466 blink::EXTDisjointTimerQuery::endQueryEXT()
#4 0x7f4ee665aebb blink::EXTDisjointTimerQueryV8Internal::endQueryEXTMethodCallback()
#5 0x7f4ef9f5930a v8::internal::FunctionCallbackArguments::Call()
#6 0x7f4efa0e7099 v8::internal::(anonymous namespace)::HandleApiCallHelper<>()
#7 0x7f4efa0e492f v8::internal::Builtin_Impl_HandleApiCall()
#8 0x7f4ebb0843a7 <unknown>

Looks like its finalizer is never getting called for this observer, although it is a GarbageCollectedFinalized.

Looking at the way the task observer is used, I wonder if we could just replace it with a simple PostTask that sets the same flag? Ken, what do you think?

Sami, thank you for tracking that down. I thought the registering and unregistering code for these TaskObservers was "correct" but maybe there's some corner case I missed. Not sure how the TaskObserver is getting deleted without unregistering itself.

The intent here was to determine when control returned to the main loop, but from talking with vmiura@, too, it looks like a simple PostTask will be sufficient -- as long as the conformance tests still pass.

Taking this bug. Thanks again for your help investigating this.

Actually, after talking with Victor -- given that the problems in  Issue 609252  are also TaskObserver-related, and occur in the GPU process -- I think there is a non-zero probability that there is an actual bug in the management of the TaskObservers. Going to build ASAN and investigate this.

Thanks to a tip from haraken@ the issue became obvious.

"""
1) GC's marking phase finishes.
2) All dead objects are poisoned. At this point, no destructor has been called yet.
3) Lazy sweeping starts, which means that the sweeping phase runs incrementally while the user program runs.
If you touch WebGLTimerQueryEXT in the lazy sweeping phase, you'll hit the poisoning error because the object is already poisoned.
"""

We still have the WebGLTimerQueryEXT registered as a task observer at #2.  It needs to be unregistered in a pre-finalizer.

As discussed offline, moving the unregistration to a pre-finalizer will solve the problem.

That being said, do you really need to register WebGLTimerQueryEXT as a TaskObserver? WebThread::TaskObserver is discouraged because the task observation adds a non-negligible overhead to each task handling. (Remember that the observer callback needs to be called for each task.)

kbr@ and I discussed the efficiency of this earlier.  It is heavyweight.  A PostTask would enable the needed functionality.

Yes, adding one TaskObserver slows down all other task executions, so it should be avoided :)

FYI I've got a patch here that replaces the TaskObserver with a posted task: https://codereview.chromium.org/2341043002/

Thanks everyone for your help tracking this down and sorry for the bug I introduced.

Per comment on https://codereview.chromium.org/2341043002/ I think it has the same basic problem. If a raw pointer to a GarbageCollectedFinalized object is ever exposed and there is a way of revoking that raw pointer in the object's destructor, then the revocation needs to be done in a pre-finalizer. Correct?

If pre-finalizers are expensive then perhaps we should allocate separate C-heap objects which use WeakMembers to point to the GC'd objects. Or is that equally as expensive as adding a pre-finalizer?

Sami, thanks for putting together the patch above -- since you're already working on this can I assign this to you again?

FYI Sami is OOO till Thursday.

OK, let me take this back and put up a slightly different patch.

It turned out I was totally swamped this week and this will be the next thing I work on. Sami, if you get in on Thursday and want to revise your patch https://codereview.chromium.org/2341043002/ , please go ahead and take this back from me. Thanks.

Sami, since you're already working with haraken@ to update https://codereview.chromium.org/2341043002/ let me please assign this to you again.

sgtm.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fcfd66a5660a6ec631facdc050e8f8fc3ded7858

commit fcfd66a5660a6ec631facdc050e8f8fc3ded7858
Author: skyostil <skyostil@chromium.org>
Date: Mon Sep 26 20:17:09 2016

webgl: Replace query task observer with a posted task

This patch replaces the task observers used by WebGL queries (timer and
regular) with an explicitly posted task. In general task observers
should be used sparingly because they add overhead to every task and
their lifetime must be explicitly managed.

We now post a regular task, which runs only after the current task has
finished, meaning it is safe to update the cached query availability
flag again.

BUG= 644575 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2341043002
Cr-Commit-Position: refs/heads/master@{#420975}

[modify] https://crrev.com/fcfd66a5660a6ec631facdc050e8f8fc3ded7858/third_party/WebKit/Source/modules/webgl/WebGLQuery.cpp
[modify] https://crrev.com/fcfd66a5660a6ec631facdc050e8f8fc3ded7858/third_party/WebKit/Source/modules/webgl/WebGLQuery.h
[modify] https://crrev.com/fcfd66a5660a6ec631facdc050e8f8fc3ded7858/third_party/WebKit/Source/modules/webgl/WebGLTimerQueryEXT.cpp
[modify] https://crrev.com/fcfd66a5660a6ec631facdc050e8f8fc3ded7858/third_party/WebKit/Source/modules/webgl/WebGLTimerQueryEXT.h