New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 872064 link

Starred by 6 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 8
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

crosvm fails to start on grunt: ForkingWhileMultiThreaded

Project Member Reported by smbar...@chromium.org, Aug 7

Issue description

crosvm can be run manually on grunt, but when invoked via concierge it fails to start: https://stainless.corp.google.com/search?exclude_cts=true&exclude_non_release=true&board=%5Egrunt%24&test=%5Etast%5C.vm%5C.&view=matrix&col=build&row=test&first_date=2018-08-01&last_date=2018-08-07

The error thrown is:
2018-08-07T15:56:16.332439-07:00 INFO localhos[18141]: crosvm[4]: [src/linux.rs:1012] crosvm entering multiprocess mode
2018-08-07T15:56:16.374346-07:00 ERR localhos[18141]: crosvm[4]: [src/main.rs:464] error registering block device: ProxyDeviceCreation(ForkingJail(ForkingWhileMultiThreaded))
2018-08-07T15:56:46.326921-07:00 ERR vm_concierge[18113]: VM failed to start in 30 seconds

...which shouldn't happen.
 
This appears to be minigbm calling into dri_init for wayland-dmabuf.

strace of crosvm:

ioctl(11, DRM_IOCTL_AMDGPU_INFO or DRM_IOCTL_SIS_FB_FREE, 0x7ffe575d8100) = 0
ioctl(10, DRM_IOCTL_GET_CAP, 0x7ffe575d80f8) = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [CHLD], 8) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f2335a0f000
mprotect(0x7f2335a0f000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f233620eef0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f233620f9d0, tls=0x7f233620f700, child_tidptr=0x7f233620f9d0) = 6
rt_sigprocmask(SIG_SETMASK, [CHLD], NULL, 8) = 0
futex(0x7f233811a840, FUTEX_WAKE_PRIVATE, 2147483647) = 0
uname({sysname="Linux", nodename="localhost", ...}) = 0
stat("/usr/lib64/dri/radeonsi_dri.so", {st_mode=S_IFREG|0755, st_size=30148424, ...}) = 0

ick. Is the thread creation in minigbm avoidable? Or do we need to rework the jailing sequence?
Cc: gurcheta...@chromium.org
To avoid the thread creation, we would either have to adjust minigbm or adjust the amdgpu dri driver. Ideally minigbm should not have depended on dri, but I recall this being the worse of two evils. Adding gurchetan, who might be able to give us context on how feasible it would be too avoid thread creating in minigbm.
Can you trace where is the new thread being created?  I assume it's when creating a new context:

https://chromium.googlesource.com/chromiumos/platform/minigbm/+/master/dri.c#148

https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/state_trackers/dri/dri_context.c#n193 

If that's the case, we may be able to disable this.
Cc: ddavenp...@chromium.org
Labels: -Pri-2 Pri-1
Grunt launches on M70, so we really need this one fixed ASAP unless we want to disable it for launch.
FYI here's a completed untested change that disables new thread creation when creating a new context:

crrev.com/c/1194659
update?
Labels: -M-70 M-71
Issue 890929 has been merged into this issue.
Project Member

Comment 11 by bugdroid1@chromium.org, Oct 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/e56bba39be7005599cef6acd0c78496f30a0a80b

commit e56bba39be7005599cef6acd0c78496f30a0a80b
Author: Zach Reizner <zachr@google.com>
Date: Wed Oct 03 20:59:14 2018

vm_host_tools: add crosvm_wl_dmabuf to IUSE

This USE flag is used in an #if statement in concierge.

BUG= chromium:872064 
TEST=emerge-grunt vm_host_tools

Change-Id: Ic7298649288f0623e3def47c4d064987a4ee27d9
Reviewed-on: https://chromium-review.googlesource.com/1257594
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>

[modify] https://crrev.com/e56bba39be7005599cef6acd0c78496f30a0a80b/chromeos-base/vm_host_tools/vm_host_tools-9999.ebuild

Project Member

Comment 12 by bugdroid1@chromium.org, Oct 6

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/04a2b3dcaf1abdd105b8f1539e311ac321e28a7d

commit 04a2b3dcaf1abdd105b8f1539e311ac321e28a7d
Author: Zach Reizner <zachr@google.com>
Date: Sat Oct 06 02:37:58 2018

grunt: disable crosvm-wl-dmabuf USE flag

The multiprocess mode of crosvm does not work when it uses minigbm on
radeonsi cards (such as on grunt), which spawns a background thread. The
background thread trips a protective assert when crosvm forks device
processes which behave badly when in multithreaded processes.

BUG= chromium:872064 
TEST=on grunt: vmc start termina

Change-Id: I12c6fa63a7e9064215bfd93b7d8992b1dd1bb527
Reviewed-on: https://chromium-review.googlesource.com/1257650
Commit-Ready: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Drew Davenport <ddavenport@chromium.org>
Reviewed-by: Daniel Kurtz <djkurtz@chromium.org>

[modify] https://crrev.com/04a2b3dcaf1abdd105b8f1539e311ac321e28a7d/overlay-grunt/profiles/base/make.defaults

Project Member

Comment 13 by bugdroid1@chromium.org, Oct 13

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/aff70f4b3c6de6496aa5639c598e7907297ee30f

commit aff70f4b3c6de6496aa5639c598e7907297ee30f
Author: Chirantan Ekbote <chirantan@chromium.org>
Date: Sat Oct 13 06:07:15 2018

vm_host_tools: Enable crosvm_wl_dmabuf by default

Without this flag, concierge will not enable wayland dmabufs by default,
regressing graphics performance by requiring lots of buffer copying.

BUG= chromium:872064 
TEST="vmc start termina" and see that crosvm has the --wayland-dmabuf
     flag set

Change-Id: Icca6f6985b2cca5163cc422934d262ddc91734e5
Signed-off-by: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1279267
Reviewed-by: Zach Reizner <zachr@chromium.org>

[modify] https://crrev.com/aff70f4b3c6de6496aa5639c598e7907297ee30f/chromeos-base/vm_host_tools/vm_host_tools-9999.ebuild

Project Member

Comment 14 by bugdroid1@chromium.org, Oct 15

Labels: merge-merged-firmware-grunt-11031.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/3b44ab29e878cf5558c13c068ddb5d7bfd0f3947

commit 3b44ab29e878cf5558c13c068ddb5d7bfd0f3947
Author: Zach Reizner <zachr@google.com>
Date: Mon Oct 15 00:50:25 2018

grunt: disable crosvm-wl-dmabuf USE flag

The multiprocess mode of crosvm does not work when it uses minigbm on
radeonsi cards (such as on grunt), which spawns a background thread. The
background thread trips a protective assert when crosvm forks device
processes which behave badly when in multithreaded processes.

BUG= chromium:872064 
TEST=on grunt: vmc start termina

Change-Id: I12c6fa63a7e9064215bfd93b7d8992b1dd1bb527
Reviewed-on: https://chromium-review.googlesource.com/1257650
Commit-Ready: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Drew Davenport <ddavenport@chromium.org>
Reviewed-by: Daniel Kurtz <djkurtz@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/1279933
Reviewed-by: Martin Roth <martinroth@chromium.org>
Tested-by: Martin Roth <martinroth@chromium.org>

[modify] https://crrev.com/3b44ab29e878cf5558c13c068ddb5d7bfd0f3947/overlay-grunt/profiles/base/make.defaults

Project Member

Comment 15 by bugdroid1@chromium.org, Oct 15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/10afe0c2942c7032a0cf8f3888f849b76f48c816

commit 10afe0c2942c7032a0cf8f3888f849b76f48c816
Author: Zach Reizner <zachr@google.com>
Date: Mon Oct 15 23:01:56 2018

vm_tools: concierge: check for crosvm-wl-dmabuf use flag

If the crosvm-wl-dmabuf USE flag is unset, crosvm will not recognize the
wayland-dmabuf argument, causing it to fail. This change removes that
argument from the crosvm invocation in the absense of the
crosvm-wl-dmabuf USE flag to prevent that failure.

BUG= chromium:872064 
TEST=on grunt: vmc start termina
CQ-DEPEND=CL:1257594

Change-Id: Ie450133c3731d6d0c6cc6a7bfb035377411887d4
Reviewed-on: https://chromium-review.googlesource.com/1257614
Commit-Ready: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>

[modify] https://crrev.com/10afe0c2942c7032a0cf8f3888f849b76f48c816/vm_tools/host.gypi
[modify] https://crrev.com/10afe0c2942c7032a0cf8f3888f849b76f48c816/vm_tools/concierge/virtual_machine.cc
[modify] https://crrev.com/10afe0c2942c7032a0cf8f3888f849b76f48c816/common-mk/common.gypi

Project Member

Comment 16 by bugdroid1@chromium.org, Oct 19

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/4ef106fc61f412045df549a4a9bd068189e955e8

commit 4ef106fc61f412045df549a4a9bd068189e955e8
Author: Stephen Barber <smbarber@chromium.org>
Date: Fri Oct 19 05:38:02 2018

vm_host_tools: fix crosvm-wl-dmabuf USE flag

The USE flag should match the one in crosvm's ebuild.

BUG= chromium:872064 
TEST=crosvm starts on grunt

Change-Id: I9e94e4d94bd033a3349009528192479e76316a82
Reviewed-on: https://chromium-review.googlesource.com/1287769
Commit-Ready: Stephen Barber <smbarber@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>

[modify] https://crrev.com/4ef106fc61f412045df549a4a9bd068189e955e8/chromeos-base/vm_host_tools/vm_host_tools-9999.ebuild

Discussed offline, vm.CrostiniStartEverything is failing since sommelier will still try to use dmabuf for wayland apps.

https://stainless.corp.google.com/browse/chromeos-autotest-results/249822248-chromeos-test/
Ping. I think we need a sommelier fix here.
Project Member

Comment 19 by bugdroid1@chromium.org, Nov 1

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/f797eeec7b10cd81057719050e6f6fbfd3e16bcd

commit f797eeec7b10cd81057719050e6f6fbfd3e16bcd
Author: Zach Reizner <zachr@google.com>
Date: Thu Nov 01 04:59:38 2018

vm_tools: sommelier: check that virtwl-dmabuf works, fallback to virtwl

On some hosts, dmabufs over virti-wayland are not supported, but
sommelier is compiled with the virtwl-dmabuf shm-driver as the default
on all termina images. In order to gracefully handle situtations where
the default will not work, this CL adds a quick check that
VIRTWL_IOCTL_NEW_DMABUF does not return ENOTTY before using
virtwl-dmabuf. If that ioctl does return ENOTTY, sommelier falls back to
using virtwl shm-driver.

BUG= chromium:872064 
TEST=wayland_demo

Change-Id: Ie5ddf575ba9995aa6e4e102c1ce2102cd2c935bd
Reviewed-on: https://chromium-review.googlesource.com/1306905
Commit-Ready: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/f797eeec7b10cd81057719050e6f6fbfd3e16bcd/vm_tools/sommelier/sommelier.c

Labels: -M-71 M-73 M-72
Observing this issue on 11151.100.0, 71.0.3578.120(FSI for Grunt). As M71 is already moved to stable channel, removing M71 label and adding M72 & M73. 
Please feel free to change milestone labels if anyone feel otherwise. Thanks!
Labels: -M-73
Status: Fixed (was: Assigned)

Sign in to add a comment