DCHECK failure in tast.video.WebRTCCamera on ChromeOS VM |
||||||||||||
Issue descriptionSee https://chromium-review.googlesource.com/c/chromium/src/+/1352979 Example failure: https://chromium-swarm.appspot.com/task?id=417282de2d963210 Specifically: 2018/11/28 04:16:12 2 failed: 2018/11/28 04:16:12 video.WebRTCCamera 2018/11/28 04:16:12 video.WebRTCPeerConnCameraVP8 Logs look to have quite a few stack traces. Kinda hard to determine which is relevant. Chrome logs: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=1989558bb2d2bf459adc85eb639fc426a0d8167f&as=chrome_20181128-041416 UI logs: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=ca6439a1546e5fb56be2ed9d62b4cb4d0ff67f11&as=ui.20181128-041416 I see a few "WeakPtrs must be checked on the same sequenced thread." failures... might be the same root cause as bug 909736.
,
Nov 29
log/chrome/chrome_20181128-015056: [8355:8590:1128/015119.048287:FATAL:history_backend.cc(838)] Check failed: false. Adding URL failed. #0 0x5ab8f14c1ccf base::debug::StackTrace::StackTrace() #1 0x5ab8f1416dfb logging::LogMessage::~LogMessage() #2 0x5ab8f323cf81 history::HistoryBackend::AddPageVisit() #3 0x5ab8f323c1db history::HistoryBackend::AddPage() #4 0x5ab8edb2a248 _ZN4base8internal7InvokerINS0_9BindStateIMN8chromeos23CryptohomeAuthenticatorEFvRKNS3_11AuthFailureEEJ13scoped_refptrIS4_ES5_EEEFvvEE7RunOnceEPNS0_13BindStateBaseE #5 0x5ab8f14e75b7 base::debug::TaskAnnotator::RunTask() #6 0x5ab8f1420abf base::MessageLoopImpl::RunTask() #7 0x5ab8f1421172 base::MessageLoopImpl::DoWork() #8 0x5ab8f142314a base::MessagePumpDefault::Run() #9 0x5ab8f1420595 base::MessageLoopImpl::Run() #10 0x5ab8f144ad06 base::RunLoop::Run() #11 0x5ab8f1493c1a base::Thread::Run() #12 0x5ab8f1493fa8 base::Thread::ThreadMain() #13 0x5ab8f14d50b8 base::(anonymous namespace)::ThreadFunc() #14 0x7eb9f97344fe <unknown> #15 0x7eb9f8b8cbef clone
,
Nov 29
add camera folks
,
Nov 29
Did we recently enable DCHECK on these tests? The failures may not be specific to the camera code, just getting triggered there more often (many possible reasons). Regardless, we should fix these. The history_backend.cc NOTREACHED has been around for a while, although it's possible something recent is triggering it? https://cs.chromium.org/chromium/src/components/history/core/browser/history_backend.cc?q=history_backend.cc&sq=package:chromium&dr&l=838
,
Nov 29
> Did we recently enable DCHECK on these tests? The chromium bot's been running vm tests w/ DCHECKs for a while now. But it looks like these specific tests are new. They're not present in the VM image for the current LKGM, only in the images for the later LKGMs that the auto roller is trying to update to. So unless these tests run w/ DCHECKs anywhere on chromeos bots, this is the first time they'd be hitting that check. I'll poke at them a little, but this is likely the same thing as bug 909736, since I'm seeing a bunch of "WeakPtrs must be checked on the same sequenced thread" when ran locally.
,
Nov 29
Hmm. So should DCHECK be disabled when running these tests on the Chrome side? Right now, it sounds like Chrome is pulling in tests and executing them using a configuration that hasn't been previously vetted on the OS side.
,
Nov 29
Possibly? There's been a lot of discussion around dcheck_always_on, but the only related change I am aware of is Ben enabling it on daisy chromium builders to bring them in alignment with amd64, but maybe it wasn't actually enabled on amd64 before? (It's surprising that we are failing amd64 and not daisy; I am aware of two outstanding arm specific DCHECKs). We could disable dchecks for now, but I'd really like to make sure we fix these and then enable them. I'll do what I can to help.
,
Nov 29
I've been playing around with DCHECK enabled on kevin and haven't been able to repro any of the above crashes in the browser process. I can repro a render crash when opening a Hangouts window, but I don't actually know how to attach to a render process or get a backtrace. +jamescook@ - do you know how to get a callstack from a render process on a dev machine?
,
Nov 29
Um, I haven't debugged a renderer crash in forever. Maybe --no-sandbox and an unstripped build would give you a backtrace?
,
Nov 30
,
Dec 3
,
Dec 5
I'm able to reproduce the crash for video.WebRTCCamera: [5629:5673:1205/155519.206936:FATAL:weak_ptr.cc(28)] Check failed: (sequence_checker_).CalledOnValidSequence(). WeakPtrs must be checked on the same sequenced thread. #0 0x5a1147ad6d0f base::debug::StackTrace::StackTrace() #1 0x5a1147a2c07b logging::LogMessage::~LogMessage() #2 0x5a1147a32594 base::internal::WeakReference::IsValid() #3 0x5a11453ff108 base::ObserverList<>::AddObserver() #4 0x5a11493d7200 media::ScreenObserverDelegate::AddObserverOnDisplayThread() #5 0x5a1143d661fb _ZN4base8internal7InvokerINS0_9BindStateIMN3net12SerialWorkerEFvvEJ13scoped_refptrIS4_EEEEFvvEE7RunOnceEPNS0_13BindStateBaseE #6 0x5a1147afc607 base::debug::TaskAnnotator::RunTask() #7 0x5a1147a9ca65 base::internal::TaskTracker::RunOrSkipTask() #8 0x5a1147ae9613 base::internal::TaskTrackerPosix::RunOrSkipTask() #9 0x5a1147a9bd6f base::internal::TaskTracker::RunAndPopNextTask() #10 0x5a1147b0a56d base::internal::SchedulerWorker::RunWorker() #11 0x5a1147b0a2e1 base::internal::SchedulerWorker::RunDedicatedWorker() #12 0x5a1147aea0f8 base::(anonymous namespace)::ThreadFunc() #13 0x7b66e026c4fe <unknown> #14 0x7b66df6c4bef clone
,
Dec 6
I believe this is the crashing line: https://cs.chromium.org/chromium/src/media/capture/video/chromeos/display_rotation_observer.cc?l=60 The code invoking it is here: https://cs.chromium.org/chromium/src/media/capture/video/chromeos/display_rotation_observer.cc?l=20-23 This code was added here: https://chromium-review.googlesource.com/c/chromium/src/+/1147896/ Assigning to tzik@ for resolution or triage. Repo steps - enter simple chrome with a recent cros, download the VM, turn dchecks on, build, start the VM, deploy chrome with symbols, run the failing command: cros chrome-sdk --debug --download-vm --board=amd64-generic --version=11327.0.0 --gn-extra-args='dcheck_always_on=true' autoninja -C out_$SDK_BOARD/Release/ chromiumos_preflight cros_vm --start deploy_chrome --build-dir=out_amd64-generic/Release --to=localhost --port=9222 --mount --nostrip cros_vm --cmd -- local_test_runner video.WebRTCCamera You should be able to see the crash log with this stack in /var/log/chrome/chrome.PREVIOUS
,
Dec 6
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0f41308e426d37bc8e6cc97234c4f41ca89e4d47 commit 0f41308e426d37bc8e6cc97234c4f41ca89e4d47 Author: Ben Pastene <bpastene@chromium.org> Date: Thu Dec 06 00:12:20 2018 Disable video.WebRTCCamera* tast tests on chrome bots. This has been blocking LKGM updates for a while now. Disable it while we work on a proper fix. Bug: 909975 Change-Id: I850e49137a68e013e745dff42507767bb8e4b798 Reviewed-on: https://chromium-review.googlesource.com/c/1363805 Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: James Cook <jamescook@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#614189} [modify] https://crrev.com/0f41308e426d37bc8e6cc97234c4f41ca89e4d47/chromeos/BUILD.gn
,
Dec 6
Is this issue unblocked? Can we assign the disabled tests to a member of the team and close this bug?
,
Dec 6
,
Dec 7
Let me change the summary of this issue because this failure doesn't block CHROMEOS_LKGM updates anymore. Now, the goal of this issue is enabling tast.video.WebRTCCamera and video.WebRTCPeerConnCameraVP8 on Chrome CQ, which are disabled by http://crrev.com/c/1363805.
,
Dec 7
,
Dec 7
Looks like it's due to a wrong task_runner setup around video_capture. |display_task_runner| on media::ScreenObserverDelegate must be the UI thread, since other call sites use the UI thread, (e.g. CrosDisplayConfig), but the failing DCHECK indicates |display_task_runner_| there is not the UI thread. The task runner is propagated from DeviceFactoryProviderImpl. Though the comment says "UI thread equivalent", the task runner is not the UI thread one. https://chromium.googlesource.com/chromium/src/+/HEAD/services/video_capture/device_factory_provider_impl.cc#121 It's likely a dedicated thread created at ServiceManagerContext: https://chromium.googlesource.com/chromium/src/+/HEAD/content/browser/service_manager/service_manager_context.cc#667 To fix this, we should pass the UI thread task runner instead of base::ThreadTaskRunnerHandle::Get() at DeviceFactoryProviderImpl::LazyInitializeDeviceFactory. And to align to the layering around this, we should pass it from //content on its initialization.
,
Dec 7
,
Dec 10
,
Dec 10
Though the root cause of this failure would be fixed by tzik@'s crrev.com/c/1367388, WebRTC tast tests are still failing by disk-full error on Chrome build infra. (cf. crrev.com/c/1367333) I created Issue 913356 for this problem.
,
Dec 10
,
Dec 12
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/18144b14e8f07ed78ec66399ac2bf4ef606c4b17 commit 18144b14e8f07ed78ec66399ac2bf4ef606c4b17 Author: tzik <tzik@chromium.org> Date: Wed Dec 12 13:40:04 2018 Fix a thread restriction violation around video_capture CreateVideoCaptureDeviceFactory requires the task runner for UI thread, but the task runner passed by DeviceFactoryProviderImpl to CVCDF is not. That causes DCHECK failure around AddObserverOnDisplayThread on ScreenObserverDelegate. Bug: 909975 Change-Id: I778647b87c619002c64fa871252e88ac184ea5cf Reviewed-on: https://chromium-review.googlesource.com/c/1367388 Reviewed-by: Emircan Uysaler <emircan@chromium.org> Reviewed-by: Kinuko Yasuda <kinuko@chromium.org> Commit-Queue: Taiju Tsuiki <tzik@chromium.org> Cr-Commit-Position: refs/heads/master@{#615873} [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/content/browser/service_manager/service_manager_context.cc [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/content/utility/utility_service_factory.cc [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/services/video_capture/device_factory_provider_impl.cc [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/services/video_capture/device_factory_provider_impl.h [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/services/video_capture/service_impl.cc [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/services/video_capture/service_impl.h [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/services/video_capture/service_main.cc [modify] https://crrev.com/18144b14e8f07ed78ec66399ac2bf4ef606c4b17/services/video_capture/test/device_factory_provider_connectortest.cc
,
Dec 12
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/76eecc11cf117df5b07fedcf58f5a44d01b2f89a commit 76eecc11cf117df5b07fedcf58f5a44d01b2f89a Author: Keiichi Watanabe <keiichiw@chromium.org> Date: Wed Dec 12 16:16:37 2018 Enable video.WebRTC* tast tests on chrome bots. video.WebRTC* tasts tests are disabled on chrome bot by CL:1363805 because of a bug reported at crbug.com/909975 . Since this bug was fixed by CL:1367388, we can run these tests now. Bug: 909975 Change-Id: I74b719988470552fe5abc39f217dda0086e15e71 Reviewed-on: https://chromium-review.googlesource.com/c/1367333 Commit-Queue: Keiichi Watanabe <keiichiw@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> Cr-Commit-Position: refs/heads/master@{#615910} [modify] https://crrev.com/76eecc11cf117df5b07fedcf58f5a44d01b2f89a/chromeos/BUILD.gn
,
Dec 13
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by derat@chromium.org
, Nov 29