Enable DCHECKs on amd64-generic simplechrome CQ bot. |
||||||
Issue descriptionPer http://crbug.com/867077 , it would be really nice to get DCHECKs on in the amd64-generic simplechrome CQ bot. At the moment, they are not turned on because: (at least) Reason for revert: Browser crashes flakily with DCHECKs on See https://chromium-swarm.appspot.com/task?id=3f063cbec4a26e10 https://chromium-swarm.appspot.com/task?id=3f063760c6162910 https://chromium-swarm.appspot.com/task?id=3f063164005d9710 Example crashes https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=e4fef843d0fef24b39d59d06bfddbe1569402f2b&as=chrome.PREVIOUS https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=4d98a300e399d97b4b02cf89f01134050e2f985e&as=chrome_20180730-160158 https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=a4068c05476b786213ab26a80cf35e426fd54f44&as=chrome_20180730-155114 Proposal: of these, we have: [2622:2634:0730/155118.629178:FATAL:hardware_display_plane_manager_legacy.cc(105)] Check failed: false. HardwareDisplayPlaneManagerLegacy doesn't support per plane CTM See http://crbug.com/869901 to fix. This one: [2655:2655:0730/160221.762254:FATAL:resource_bundle.cc(123)] Check failed: false. Unable to load image with id 10511, scale=2 I have been seeing this in personal builds from ToT recently. Maybe turn that DCHECK off on CrOS temporarily while re-enabling DCHECKs overall to stop things from getting worse?
,
Aug 1
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b4311d0f07775f7ee4556bcaeb437f045b1d76cf commit b4311d0f07775f7ee4556bcaeb437f045b1d76cf Author: Ben Pastene <bpastene@chromium.org> Date: Wed Aug 01 21:35:01 2018 Enable DCHECKs on the fyi amd64-generic simplechrome bot. TBR=jbudorick@chromium.org Bug: 869904 Change-Id: I23e8a323b3cd36e997c7aaa5eff7d5d2c0c7c869 Reviewed-on: https://chromium-review.googlesource.com/1158790 Reviewed-by: Ben Pastene <bpastene@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#579952} [modify] https://crrev.com/b4311d0f07775f7ee4556bcaeb437f045b1d76cf/tools/mb/mb_config.pyl
,
Aug 2
I'm also seeing: [4649:4649:0801/171212.341432:FATAL:thread_restrictions.cc(29)] Check failed: !g_blocking_disallowed.Get().Get(). Function marked as blocking was called from a scope that disallows blocking! If this task is running inside the TaskScheduler, it needs to have MayBlock() in its TaskTraits. Otherwise, consider making this blocking work asynchronous or, as a last resort, you may use ScopedAllowBlocking (see its documentation for best practices). The stack is worthless. I'm not seeing the other 2 NOT_REACHED. Were you running a test?
,
Aug 2
Stack trace: [3008:3008:0801/172017.334242:FATAL:thread_restrictions.cc(29)] Check failed: !g_blocking_disallowed.Get().Get(). Function marked as blocking was called from a scope that disallows blocking! If this task is running inside the TaskScheduler, it needs to have MayBlock() in its TaskTraits. Otherwise, consider making this blocking work asynchronous or, as a last resort, you may use ScopedAllowBlocking (see its documentation for best practices). #0 0x5bef0223691c base::debug::StackTrace::StackTrace() #1 0x5bef0218f4f0 logging::LogMessage::~LogMessage() #2 0x5bef02209a56 base::AssertBlockingAllowed() #3 0x5bef0223bebe base::PathExists() #4 0x5bef01f07bb7 component_updater::CrOSComponentManager::IsRegistered() #5 0x5bef00bbe907 crostini::CrostiniManager::MaybeUpgradeCrostini() #6 0x5bef00d24c22 chromeos::UserSessionManager::FinalizePrepareProfile() #7 0x5bef00d248ae chromeos::UserSessionManager::PrepareTpmDeviceAndFinalizeProfile() #8 0x5beeff4bb2d8 _ZN4base8internal7InvokerINS0_9BindStateIMN2ui14GbmSurfacelessEFvPNS4_12PendingFrameEEJNS_7WeakPtrIS4_EES6_EEEFvvEE7RunOnceEPNS0_13BindStateBaseE #9 0x5bef022596eb base::debug::TaskAnnotator::RunTask() #10 0x5bef02197c56 base::MessageLoop::RunTask() #11 0x5bef02198162 base::MessageLoop::DoWork() #12 0x5bef022551c9 base::MessagePumpLibevent::Run() #13 0x5bef02197621 base::MessageLoop::Run() #14 0x5bef021c4a16 base::RunLoop::Run() #15 0x5bef01d74b58 ChromeBrowserMainParts::MainMessageLoopRun() #16 0x5beeffd553f4 content::BrowserMainLoop::RunMainMessageLoopParts() #17 0x5beeffd583c3 content::BrowserMainRunnerImpl::Run() #18 0x5beeffd5121d content::BrowserMain() #19 0x5bef01d617df content::ContentMainRunnerImpl::Run() #20 0x5bef01d69afc service_manager::Main() #21 0x5bef01d5f811 content::ContentMain() #22 0x5beefed6d39f ChromeMain #23 0x7ad15e668736 __libc_start_main #24 0x5beefed6d1c9 _start
,
Aug 2
,
Aug 2
https://chromium-review.googlesource.com/c/chromium/src/+/1159700 I wasn't able to get either of these DCHECKS to trigger with vm_sanity.
,
Aug 2
It was flakey. Maybe 1 out of every 10 runs of vm_sanity the browser would crash. FWIW, I just enabled DCHECKs on the fyi/experimental amd64-generic simplechrome bot: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests?limit=200 Give it a bit and we should start seeing vm_sanity fail at a similar rate.
,
Aug 2
Incidentally, the incantation for getting an unstripped binary is: deploy_chrome --build-dir=out_$SDK_BOARD/Release/ --to=localhost --port=9222 --nostrip --mount Unfortunately, this will not work with the default downloaded VMs because their stateful partition is too small.
,
Aug 2
I think we can land the CL in #c6 without detailed info of the stack if these are the only 2 crashes.
,
Aug 2
There's been a few more crashes on the fyi bot and they're all either of the two known checks. thread_restrictions.cc(29): https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=ffc1366c33d375403cb1ee166a5eb1d13c74d202&as=chrome_20180801-185431 https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=99d632a7e307607e26032d2177dbbd1e0265bfea&as=chrome_20180801-194643 https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=c1ccbc4302c94aa07735d0d9c53594ddaf64001f&as=chrome_20180802-000538 resource_bundle.cc(123): https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=d90f96c6f1f9114dcb8c0ed47fc4c63550e0d459&as=chrome_20180802-043355 https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=a3e8108979bc4fa0519976ab2416313858283f14&as=chrome_20180802-053837 So turning them off sgtm, though who knows if something else will creep up after them.
,
Aug 2
The thread_restrictions one ought to be fixed by: https://chromium-review.googlesource.com/c/chromium/src/+/1159936 I'll try to reproduce the other one. Incidentally we've always had a problem with DCHECKs failing on the device/VM. The first thing developers do when they need to debug something with symbols is find all the DCHECKS they need to disable. So this is good work.
,
Aug 3
In 100 runs, I saw 4 of this crash: [22918:22932:0803/133135.639466:FATAL:hardware_display_plane_manager_legacy.cc(105)] Check failed: false. HardwareDisplayPlaneManagerLegacy doesn't support per plane CTM #0 0x5a4a42df242c base::debug::StackTrace::StackTrace() #1 0x5a4a42d4b1d0 logging::LogMessage::~LogMessage() #2 0x5a4a400904ad ui::HardwareDisplayPlaneManagerLegacy::SetColorCorrectionOnAllCrtcPlanes() #3 0x5a4a4008d0c8 ui::HardwareDisplayPlaneManager::SetColorMatrix() #4 0x5a4a4006f3af ui::DrmDisplay::SetColorMatrix() #5 0x5a4a400713e1 ui::DrmGpuDisplayManager::SetColorMatrix() #6 0x5a4a42e1511b base::debug::TaskAnnotator::RunTask() #7 0x5a4a42d53846 base::MessageLoop::RunTask() #8 0x5a4a42d53d52 base::MessageLoop::DoWork() #9 0x5a4a42e10bf9 base::MessagePumpLibevent::Run() #10 0x5a4a42d53211 base::MessageLoop::Run() #11 0x5a4a42d80656 base::RunLoop::Run() #12 0x5a4a42dc33aa base::Thread::Run() #13 0x5a4a42dc3737 base::Thread::ThreadMain() #14 0x5a4a42e0445f base::(anonymous namespace)::ThreadFunc() #15 0x7fb598d862b8 <unknown> #16 0x7fb598237fad clone Tracked in https://bugs.chromium.org/p/chromium/issues/detail?id=869901#c3
,
Aug 7
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/d3cb83bfaf21f1ec339c315fb67eeeacd66eb5c4 commit d3cb83bfaf21f1ec339c315fb67eeeacd66eb5c4 Author: Achuith Bhandarkar <achuith@chromium.org> Date: Tue Aug 07 08:51:25 2018 autotest: Add --count to vm_sanity BUG= chromium:869904 TEST=manual Change-Id: I0a57ea4e5a95ffdf00cc433b73f958d247da3fd6 Reviewed-on: https://chromium-review.googlesource.com/1162710 Commit-Ready: Achuith Bhandarkar <achuith@chromium.org> Tested-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> [modify] https://crrev.com/d3cb83bfaf21f1ec339c315fb67eeeacd66eb5c4/client/bin/vm_sanity.py
,
Aug 13
vm_sanity is not triggering any DCHECKS now (which is great), but looks like we have some failures in net_unittests.
,
Aug 13
,
Aug 14
net_unittests isn't on the CQ, just the experimental bot (though I'd like to move it over at some point) And yeah, sanity tests seems much happier now w/ DCHECKs. I'll try enabling it on the CQ again.
,
Aug 14
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a1694874de3980b98b88324da9fe06fff7a54bc1 commit a1694874de3980b98b88324da9fe06fff7a54bc1 Author: Ben Pastene <bpastene@chromium.org> Date: Tue Aug 14 23:38:51 2018 Re-enable DCHECKs on amd64-generic simplechrome cros CQ bot. The sanity test looks much more stable: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests This should be safe to land now. Bug: 869904 Change-Id: Ic0d7e5537e9212b7f7aaf0250041c5a73353279d Reviewed-on: https://chromium-review.googlesource.com/1174968 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#583082} [modify] https://crrev.com/a1694874de3980b98b88324da9fe06fff7a54bc1/tools/mb/mb_config.pyl
,
Sep 9
I believe DCHECKs are now enabled. Please re-open if that's not true.
,
Sep 10
Yup, thanks for tracking down all the failures achuith! |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by bpastene@chromium.org
, Aug 1