New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 849028 link

Starred by 7 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Crostini is broken since at least 10743.0.0

Project Member Reported by smbar...@chromium.org, Jun 2 2018

Issue description

Chrome version: 69.0.3447.0
OS: Chrome OS 10743.0.0

There are multiple things wrong:
1) Click Terminal app, nothing happens. VM doesn't start. concierge is never started.

2) Run `vmc start termina`, this returns:

Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.debugd was not provided by any .service files
Unable to start concierge service

debugd is also not started?

3) If I manually start debugd and vm_concierge, then I can start a VM via `vmc start termina`. But the Terminal app is still very much busted. It'll start a VM, but the container won't start.

I suspect this is from https://chromium-review.googlesource.com/c/chromium/src/+/1082055 - does Chrome know whether the container is running? The old entry path would always start the container if it wasn't running, and the new one won't.

If we want to revert this for now, I'm okay with this waiting until I improve the container lifecycle situation.
 
Cc: lhchavez@chromium.org
if debugd isn't working, might look into
  https://chromium-review.googlesource.com/1053426
Labels: Hotlist-Crostini-Platform ReleaseBlock-Dev M-69
Owner: smbar...@chromium.org
Status: Assigned (was: Untriaged)
I'm seeing the debugd error randomly on 69.0.3449.0 / 10751.0.0. Restarting sometimes fixes it, though as you noted the terminal is busted. The window flashes briefly and immediately closes.
Can you do

$ sudo stop debugd
$ sudo strace -o /tmp/debugd.log -f debugd

and try again? If there are any ENOENT errors in the log, we might need to add more bind-mounts to debugd.
If I start debugd manually, it can start concierge, but it looks like it's hitting a race with cups during boot:

2018-06-04T12:35:08.530085-07:00 WARNING kernel: [    4.551081] init: debugd main process ended, respawning
2018-06-04T12:35:08.587455-07:00 INFO debugd[2021]: libminijail[2021]: mount / -> / type ''
2018-06-04T12:35:08.587502-07:00 INFO debugd[2021]: libminijail[2021]: mount none -> /proc type 'proc'
2018-06-04T12:35:08.587520-07:00 INFO debugd[2021]: libminijail[2021]: mount /var -> /var type ''
2018-06-04T12:35:08.587536-07:00 INFO debugd[2021]: libminijail[2021]: mount tmpfs -> /run type 'tmpfs'
2018-06-04T12:35:08.587552-07:00 INFO debugd[2021]: libminijail[2021]: mount /run/dbus -> /run/dbus type ''
2018-06-04T12:35:08.587569-07:00 INFO debugd[2021]: libminijail[2021]: mount /tmp -> /tmp type ''
2018-06-04T12:35:08.587588-07:00 INFO debugd[2021]: libminijail[2021]: mount /run/cups -> /run/cups type ''
2018-06-04T12:35:08.587606-07:00 INFO debugd[2021]: libminijail[2021]: mount /dev -> /dev type 'bind'
2018-06-04T12:35:08.587624-07:00 INFO debugd[2021]: libminijail[2021]: mount /sys -> /sys type 'bind'
2018-06-04T12:35:08.587837-07:00 WARNING debugd[2021]: libminijail[2021]: stat(/run/cups) failed: No such file or directory
2018-06-04T12:35:08.587866-07:00 WARNING debugd[2021]: libminijail[2021]: creating mount target '/var/empty/run/cups' failed
2018-06-04T12:35:08.587885-07:00 ERR debugd[2021]: libminijail[2021]: mount_one failed: No such file or directory
2018-06-04T12:35:08.592769-07:00 INFO crash_reporter[2022]: libminijail[2022]: mount /dev/log -> /dev/log type ''
2018-06-04T12:35:08.634650-07:00 WARNING crash_reporter[2022]: [user] Received crash notification for debugd[2021] sig 6, user 0 (developer build - not testing - always dumping)
2018-06-04T12:35:08.635675-07:00 INFO crash_reporter[2022]: State of crashed process [2021]: S (sleeping)
2018-06-04T12:35:08.636048-07:00 WARNING crash_reporter[2022]: Crash directory /var/spool/crash already full with 32 pending reports
2018-06-04T12:35:08.636069-07:00 ERR crash_reporter[2022]: Could not create crash directory
2018-06-04T12:35:08.636077-07:00 ERR crash_reporter[2022]: Unable to find/create process-specific crash path
2018-06-04T12:35:08.642017-07:00 WARNING kernel: [    4.663128] init: debugd main process (2021) killed by ABRT signal
2018-06-04T12:35:08.642027-07:00 WARNING kernel: [    4.663145] init: debugd respawning too fast, stopped
Owner: lhchavez@chromium.org
Status: Started (was: Assigned)
Assigning to me since I broke it and provided the fix.
Hmm ... I'm new to this, but getting the same 

Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.debugd was not provided by any .service files
Unable to start concierge service

with both a 'vmc start dev' and a 'vmc list'

message with my local build of ChromiumOS (amd64-generic, flashed to usb & running on an Eve).  Attached patch did not change that for me, fwiw.  But, admittedly, I may be doing something (else) wrong.
Owner: smbar...@chromium.org
Welp, reassigning to smbarber@ in that case since there might be more to this bug.
Cc: chirantan@chromium.org
Yeah, there's more to this :(

I need to dig in to why GetVMInfo isn't working.
Just to be clear, debugd racing with cups has nothing to do with VMs.  I imagine this is a problem on all boards, not just the ones with VMs enabled.  We need a dependency in the debugd upstart conf to make sure it only starts after cups has started.
Project Member

Comment 11 by bugdroid1@chromium.org, Jun 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/531ca8c8d8e01d141888a7f8fd8204854bc7da41

commit 531ca8c8d8e01d141888a7f8fd8204854bc7da41
Author: Chirantan Ekbote <chirantan@chromium.org>
Date: Mon Jun 04 23:35:06 2018

Revert "Supports the new vsh-into-running-container syntax"

This reverts commit e9753dafe05596d1bb5784df2cb99ed7c549302e.

Reason for revert: Possible suspect for  crbug.com/849028 .  The run_container.sh script would always start the container if it wasn't running but the new code path will not work if the container is not already running.

Bug:  849028 

Original change's description:
> Supports the new vsh-into-running-container syntax
> 
> It is no longer necessary to use the run_container.sh script every time a
> container shell is needed. If the container is already running, vsh now allows
> "direct access".
> 
> Bug:  848447 
> Change-Id: Ia9aeb120befd016386816cdd16fb3336b45426ff
> Reviewed-on: https://chromium-review.googlesource.com/1082055
> Reviewed-by: Timothy Loh <timloh@chromium.org>
> Commit-Queue: Nicholas Verne <nverne@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#563568}

TBR=timloh@chromium.org,smbarber@chromium.org,nverne@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  848447 
Change-Id: I7af784e0d73814e1aa5aae5e70ea6d45f556c71f
Reviewed-on: https://chromium-review.googlesource.com/1086071
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Chirantan Ekbote <chirantan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564291}
[modify] https://crrev.com/531ca8c8d8e01d141888a7f8fd8204854bc7da41/chrome/browser/chromeos/crostini/crostini_manager.cc

Some fixup CLs need to land before the Chrome-side vsh change can land again. I need to plumb vsh through cicerone directly instead of bouncing it off concierge.

Nick and I also found that crosdns isn't starting. Investigating now.

Comment 13 by nverne@google.com, Jun 5 2018

smbarber and I have been diving deeper:

The above revert is needed because concierge is currently failing to proxy LaunchVshd requests to cicerone. Permissions need updating, then it should be safe to roll forward again.

There's also an issue with the startup of crosdns, but it doesn't prevent Terminal from coming up. It may affect files app access, though.

The debugd issue is separate from both of these.
Project Member

Comment 14 by bugdroid1@chromium.org, Jun 5 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/db6a0450068d2ca92a2e2150192908323c4a179e

commit db6a0450068d2ca92a2e2150192908323c4a179e
Author: Luis Hector Chavez <lhchavez@google.com>
Date: Tue Jun 05 15:00:43 2018

debugd: Create /run/cups if it's missing

This change ensures /run/cups exists when starting debugd. This avoids a
crashloop since minijail expects any directories it bind-mounts to be
there.

BUG= chromium:849028 
TEST=start debugd

Change-Id: Ib0a83e800d3ee7d5b556e82d7bb03faf52403cc7
Reviewed-on: https://chromium-review.googlesource.com/1086028
Commit-Ready: Luis Hector Chavez <lhchavez@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/db6a0450068d2ca92a2e2150192908323c4a179e/debugd/share/debugd.conf

Status: Fixed (was: Started)
Crostini is functional on eve 10757.0.0, which has the first Chrome build with Revert "Supports the new vsh-into-running-container syntax"
Project Member

Comment 16 by bugdroid1@chromium.org, Jun 7 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/a98cb102e16995c8ad3a630e66fd238575b4af06

commit a98cb102e16995c8ad3a630e66fd238575b4af06
Author: Stephen Barber <smbarber@chromium.org>
Date: Thu Jun 07 20:11:43 2018

vm_tools: vsh containers connect through cicerone

vsh should connect to cicerone directly to execute
the LaunchVshd RPC. Fix the dbus permissions to permit
chronos to run LaunchVshd, and allow root to run all RPCs.

Also fix an unused variable in concierge's version of LaunchVshd.

BUG= chromium:819324 , chromium:849028 
TEST=vsh to a container works

Change-Id: I2f60aa67fa293f25cfd0285b24494f1c4a0450d1
Reviewed-on: https://chromium-review.googlesource.com/1087493
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>

[modify] https://crrev.com/a98cb102e16995c8ad3a630e66fd238575b4af06/vm_tools/dbus/org.chromium.VmCicerone.conf
[modify] https://crrev.com/a98cb102e16995c8ad3a630e66fd238575b4af06/vm_tools/vsh/vsh.cc
[modify] https://crrev.com/a98cb102e16995c8ad3a630e66fd238575b4af06/vm_tools/concierge/service.cc

Sign in to add a comment