New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 869904 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Sep 9
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocked on:
issue 869901
issue 870117
issue 873851



Sign in to add a comment

Enable DCHECKs on amd64-generic simplechrome CQ bot.

Project Member Reported by rjkroege@chromium.org, Aug 1

Issue description

Per  http://crbug.com/867077 , it would be really nice to get DCHECKs on in the amd64-generic simplechrome CQ bot.

At the moment, they are not turned on because:

(at least)

Reason for revert: Browser crashes flakily with DCHECKs on

See
https://chromium-swarm.appspot.com/task?id=3f063cbec4a26e10
https://chromium-swarm.appspot.com/task?id=3f063760c6162910
https://chromium-swarm.appspot.com/task?id=3f063164005d9710

Example crashes
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=e4fef843d0fef24b39d59d06bfddbe1569402f2b&as=chrome.PREVIOUS
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=4d98a300e399d97b4b02cf89f01134050e2f985e&as=chrome_20180730-160158
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=a4068c05476b786213ab26a80cf35e426fd54f44&as=chrome_20180730-155114

Proposal: of these, we have:

[2622:2634:0730/155118.629178:FATAL:hardware_display_plane_manager_legacy.cc(105)] Check failed: false. HardwareDisplayPlaneManagerLegacy doesn't support per plane CTM

See  http://crbug.com/869901  to fix.


This one:
[2655:2655:0730/160221.762254:FATAL:resource_bundle.cc(123)] Check failed: false. Unable to load image with id 10511, scale=2

I have been seeing this in personal builds from ToT recently. Maybe turn that DCHECK off on  CrOS temporarily while re-enabling DCHECKs overall to stop things from getting worse?

 
Thanks for filing this. Probably better to turn on DCHECKs sooner rather than later, otherwise more and more crashes might sneak in.

As for the resource_bundle.cc check, I can work on providing a symbolized stack if it helps. What we have from the bots might not be too helpful:

[2583:2583:0730/155136.341790:FATAL:resource_bundle.cc(123)] Check failed: false. Unable to load image with id 10511, scale=2
#0 0x58a7e9f0888c <unknown>
#1 0x58a7e9e61710 <unknown>
#2 0x58a7e9f985f2 <unknown>
#3 0x58a7ea319023 <unknown>
#4 0x58a7ea318cfd <unknown>
#5 0x58a7ea319890 <unknown>
#6 0x58a7e9f96a16 <unknown>
#7 0x58a7e9d277ab <unknown>
#8 0x58a7e7fcdb51 <unknown>
#9 0x58a7e7fce8f5 <unknown>
#10 0x58a7e9f2b58b <unknown>
#11 0x58a7e9e69e76 <unknown>
#12 0x58a7e9e6a382 <unknown>
#13 0x58a7e9f27069 <unknown>
#14 0x58a7e9e69841 <unknown>
#15 0x58a7e9e96b06 <unknown>
#16 0x58a7e9a48418 <unknown>
#17 0x58a7e7a327f4 <unknown>
#18 0x58a7e7a355b3 <unknown>
#19 0x58a7e7a2e60d <unknown>
#20 0x58a7e9a3495f <unknown>
#21 0x58a7e9a3d3bc <unknown>
#22 0x58a7e9a32941 <unknown>
#23 0x58a7e6a5021f <unknown>
#24 0x7f95c36cc736 __libc_start_main
#25 0x58a7e6a50049 <unknown>
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 1

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b4311d0f07775f7ee4556bcaeb437f045b1d76cf

commit b4311d0f07775f7ee4556bcaeb437f045b1d76cf
Author: Ben Pastene <bpastene@chromium.org>
Date: Wed Aug 01 21:35:01 2018

Enable DCHECKs on the fyi amd64-generic simplechrome bot.

TBR=jbudorick@chromium.org

Bug:  869904 
Change-Id: I23e8a323b3cd36e997c7aaa5eff7d5d2c0c7c869
Reviewed-on: https://chromium-review.googlesource.com/1158790
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#579952}
[modify] https://crrev.com/b4311d0f07775f7ee4556bcaeb437f045b1d76cf/tools/mb/mb_config.pyl

I'm also seeing:
[4649:4649:0801/171212.341432:FATAL:thread_restrictions.cc(29)] Check failed: !g_blocking_disallowed.Get().Get(). Function marked as blocking was called from a scope that disallows blocking! If this task is running inside the TaskScheduler, it needs to have MayBlock() in its TaskTraits. Otherwise, consider making this blocking work asynchronous or, as a last resort, you may use ScopedAllowBlocking (see its documentation for best practices).

The stack is worthless.

I'm not seeing the other 2 NOT_REACHED. Were you running a test?
Labels: -Pri-3 Pri-2
Stack trace:
[3008:3008:0801/172017.334242:FATAL:thread_restrictions.cc(29)] Check failed: !g_blocking_disallowed.Get().Get(). Function marked as blocking was called from a scope that disallows blocking! If this task is running inside the TaskScheduler, it needs to have MayBlock() in its TaskTraits. Otherwise, consider making this blocking work asynchronous or, as a last resort, you may use ScopedAllowBlocking (see its documentation for best practices).
#0 0x5bef0223691c base::debug::StackTrace::StackTrace()
#1 0x5bef0218f4f0 logging::LogMessage::~LogMessage()
#2 0x5bef02209a56 base::AssertBlockingAllowed()
#3 0x5bef0223bebe base::PathExists()
#4 0x5bef01f07bb7 component_updater::CrOSComponentManager::IsRegistered()
#5 0x5bef00bbe907 crostini::CrostiniManager::MaybeUpgradeCrostini()
#6 0x5bef00d24c22 chromeos::UserSessionManager::FinalizePrepareProfile()
#7 0x5bef00d248ae chromeos::UserSessionManager::PrepareTpmDeviceAndFinalizeProfile()
#8 0x5beeff4bb2d8 _ZN4base8internal7InvokerINS0_9BindStateIMN2ui14GbmSurfacelessEFvPNS4_12PendingFrameEEJNS_7WeakPtrIS4_EES6_EEEFvvEE7RunOnceEPNS0_13BindStateBaseE
#9 0x5bef022596eb base::debug::TaskAnnotator::RunTask()
#10 0x5bef02197c56 base::MessageLoop::RunTask()
#11 0x5bef02198162 base::MessageLoop::DoWork()
#12 0x5bef022551c9 base::MessagePumpLibevent::Run()
#13 0x5bef02197621 base::MessageLoop::Run()
#14 0x5bef021c4a16 base::RunLoop::Run()
#15 0x5bef01d74b58 ChromeBrowserMainParts::MainMessageLoopRun()
#16 0x5beeffd553f4 content::BrowserMainLoop::RunMainMessageLoopParts()
#17 0x5beeffd583c3 content::BrowserMainRunnerImpl::Run()
#18 0x5beeffd5121d content::BrowserMain()
#19 0x5bef01d617df content::ContentMainRunnerImpl::Run()
#20 0x5bef01d69afc service_manager::Main()
#21 0x5bef01d5f811 content::ContentMain()
#22 0x5beefed6d39f ChromeMain
#23 0x7ad15e668736 __libc_start_main
#24 0x5beefed6d1c9 _start

Blockedon: 870117
https://chromium-review.googlesource.com/c/chromium/src/+/1159700

I wasn't able to get either of these DCHECKS to trigger with vm_sanity.
It was flakey. Maybe 1 out of every 10 runs of vm_sanity the browser would crash.

FWIW, I just enabled DCHECKs on the fyi/experimental amd64-generic simplechrome bot:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests?limit=200

Give it a bit and we should start seeing vm_sanity fail at a similar rate.
Incidentally, the incantation for getting an unstripped binary is:
deploy_chrome --build-dir=out_$SDK_BOARD/Release/ --to=localhost --port=9222 --nostrip --mount

Unfortunately, this will not work with the default downloaded VMs because their stateful partition is too small.
I think we can land the CL in #c6 without detailed info of the stack if these are the only 2 crashes.
The thread_restrictions one ought to be fixed by:
https://chromium-review.googlesource.com/c/chromium/src/+/1159936

I'll try to reproduce the other one.

Incidentally we've always had a problem with DCHECKs failing on the device/VM. The first thing developers do when they need to debug something with symbols is find all the DCHECKS they need to disable. So this is good work.
Owner: achuith@chromium.org
In 100 runs, I saw 4 of this crash:
[22918:22932:0803/133135.639466:FATAL:hardware_display_plane_manager_legacy.cc(105)] Check failed: false. HardwareDisplayPlaneManagerLegacy doesn't support per plane CTM
#0 0x5a4a42df242c base::debug::StackTrace::StackTrace()
#1 0x5a4a42d4b1d0 logging::LogMessage::~LogMessage()
#2 0x5a4a400904ad ui::HardwareDisplayPlaneManagerLegacy::SetColorCorrectionOnAllCrtcPlanes()
#3 0x5a4a4008d0c8 ui::HardwareDisplayPlaneManager::SetColorMatrix()
#4 0x5a4a4006f3af ui::DrmDisplay::SetColorMatrix()
#5 0x5a4a400713e1 ui::DrmGpuDisplayManager::SetColorMatrix()
#6 0x5a4a42e1511b base::debug::TaskAnnotator::RunTask()
#7 0x5a4a42d53846 base::MessageLoop::RunTask()
#8 0x5a4a42d53d52 base::MessageLoop::DoWork()
#9 0x5a4a42e10bf9 base::MessagePumpLibevent::Run()
#10 0x5a4a42d53211 base::MessageLoop::Run()
#11 0x5a4a42d80656 base::RunLoop::Run()
#12 0x5a4a42dc33aa base::Thread::Run()
#13 0x5a4a42dc3737 base::Thread::ThreadMain()
#14 0x5a4a42e0445f base::(anonymous namespace)::ThreadFunc()
#15 0x7fb598d862b8 <unknown>
#16 0x7fb598237fad clone


Tracked in https://bugs.chromium.org/p/chromium/issues/detail?id=869901#c3
Project Member

Comment 13 by bugdroid1@chromium.org, Aug 7

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/d3cb83bfaf21f1ec339c315fb67eeeacd66eb5c4

commit d3cb83bfaf21f1ec339c315fb67eeeacd66eb5c4
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Tue Aug 07 08:51:25 2018

autotest: Add --count to vm_sanity

BUG= chromium:869904 
TEST=manual

Change-Id: I0a57ea4e5a95ffdf00cc433b73f958d247da3fd6
Reviewed-on: https://chromium-review.googlesource.com/1162710
Commit-Ready: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/d3cb83bfaf21f1ec339c315fb67eeeacd66eb5c4/client/bin/vm_sanity.py

vm_sanity is not triggering any DCHECKS now (which is great), but looks like we have some failures in net_unittests.
Blockedon: 873851
net_unittests isn't on the CQ, just the experimental bot (though I'd like to move it over at some point)

And yeah, sanity tests seems much happier now w/ DCHECKs. I'll try enabling it on the CQ again.
Project Member

Comment 17 by bugdroid1@chromium.org, Aug 14

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a1694874de3980b98b88324da9fe06fff7a54bc1

commit a1694874de3980b98b88324da9fe06fff7a54bc1
Author: Ben Pastene <bpastene@chromium.org>
Date: Tue Aug 14 23:38:51 2018

Re-enable DCHECKs on amd64-generic simplechrome cros CQ bot.

The sanity test looks much more stable:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests

This should be safe to land now.

Bug:  869904 
Change-Id: Ic0d7e5537e9212b7f7aaf0250041c5a73353279d
Reviewed-on: https://chromium-review.googlesource.com/1174968
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583082}
[modify] https://crrev.com/a1694874de3980b98b88324da9fe06fff7a54bc1/tools/mb/mb_config.pyl

Status: Fixed (was: Available)
I believe DCHECKs are now enabled. Please re-open if that's not true.
Yup, thanks for tracking down all the failures achuith!

Sign in to add a comment