New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 913540 link

Starred by 5 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 12
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

All test failing with 9p mounting failure

Project Member Reported by jkardatzke@chromium.org, Dec 10

Issue description

As of 11366.0.0 all of the tests are solid red (except for StartCrosvm) and have the following error message:

2018-12-08T00:26:04.306717-08:00 ERR vm_concierge[8116]: Failed to mount 9P server on /mnt/shared inside VM 4: failed to mount file system
2018-12-08T00:26:04.306745-08:00 ERR vm_concierge[8116]: Failed to mount 

Here's an example:

https://stainless.corp.google.com/browse/chromeos-autotest-results/264855329-chromeos-test/

Please fix so we can get our testing results back to being useful. :)
 
Looks like there's a crash in 9s...or is that an selinux problem? I thought the audit line says blocked when that's the problem...but then it does say there's an selinux violation.

2018-12-08T00:25:37.968489-08:00 INFO vm_concierge[7028]: Mounting 9P file system from port 16384 on /mnt/shared
2018-12-08T00:25:37.970037-08:00 NOTICE kernel: [  802.599970] audit: type=1326 audit(1544257537.968:339): auid=4294967295 uid=1000 gid=1000 ses=4294967295 subj=u:r:minijail:s0 pid=7051 comm="9s" exe="/usr/bin/9s" sig=31 arch=c000003e syscall=44 compat=0 ip=0x7ba264902fbd code=0x0
2018-12-08T00:25:37.982228-08:00 INFO crash_reporter[7091]: libminijail[7091]: mount '/dev/log' -> '/dev/log' type '' flags 0x1001
2018-12-08T00:25:37.982957-08:00 INFO crash_reporter[7092]: libminijail[37]: mount '/dev/log' -> '/dev/log' type '' flags 0x1001
2018-12-08T00:25:37.995397-08:00 INFO crash_reporter[7092]: Processing selinux violation: always collect from developer builds
2018-12-08T00:25:37.995912-08:00 INFO crash_reporter[7092]: Accessing crash dir '/var/spool/crash' via symlinked handle '/proc/self/fd/8'
2018-12-08T00:25:37.996101-08:00 WARNING crash_reporter[7091]: [user] Received crash notification for 9s[7051] sig 31, user 1000 group 1000 (developer build - not testing - always dumping)
2018-12-08T00:25:37.996276-08:00 WARNING crash_reporter[7092]: Failed to write audit message to /proc/self/fd/8/selinux_violation.20181208.082537.0.log: File exists
2018-12-08T00:25:37.997506-08:00 INFO crash_reporter[7091]: State of crashed process [7051]: S (sleeping)
2018-12-08T00:25:37.997840-08:00 INFO crash_reporter[7091]: Accessing crash dir '/var/spool/crash' via symlinked handle '/proc/self/fd/5'
2018-12-08T00:25:38.019418-08:00 INFO crash_reporter[7091]: Stored minidump to /var/spool/crash/9s.20181208.002537.7051.dmp
2018-12-08T00:25:38.020164-08:00 INFO crash_reporter[7091]: Leaving core file at /proc/self/fd/5/9s.20181208.002537.7051.core due to developer image
2018-12-08T00:25:38.024714-08:00 INFO seneschal[7027]: Process 2 killed by signal 31 (core dumped)
2018-12-08T00:25:38.025889-08:00 ERR vm_concierge[7028]: Failed to mount 9P server on /mnt/shared inside VM 3: failed to mount file system
2018-12-08T00:25:38.025920-08:00 ERR vm_concierge[7028]: Failed to mount 
Cc: cywang@chromium.org wuchengli@chromium.org derat@chromium.org dverkamp@chromium.org
 Issue 913545  has been merged into this issue.
the selinux part doesn't matter

the audit log says it was sig=31 which is SIGSYS which most likely is a seccomp violation. look at the syscall=44 part to see what needs updating in the seccomp policy.
Checked the dump file and it's a "sendto" syscall with 

MDRawContextAMD64
  p1_home       = 0x0
  p2_home       = 0x0
  p3_home       = 0x0
  p4_home       = 0x0
  p5_home       = 0x0
  p6_home       = 0x0
  context_flags = 0x10000f
  mx_csr        = 0x0
  cs            = 0x33
  ds            = 0x0
  es            = 0x0
  fs            = 0x0
  gs            = 0x0
  ss            = 0x2b
  eflags        = 0x246
  dr0           = 0x0
  dr1           = 0x0
  dr2           = 0x0
  dr3           = 0x0
  dr6           = 0x0
  dr7           = 0x0
  rax           = 0x2c
  rcx           = 0x7cceacba5fbd
  rdx           = 0x3a
  rbx           = 0x5d17c095f5f0
  rsp           = 0x7ffc49dbe968
  rbp           = 0x7ffc49dbe880
  rsi           = 0x5d17c095f840
  rdi           = 0x1
  r8            = 0x0
  r9            = 0x0
  r10           = 0x4000
  r11           = 0x246
  r12           = 0x7cceacc28900
  r13           = 0x7cceace54440
  r14           = 0x7ffc49dbe890
  r15           = 0x0
  rip           = 0x7cceacba5fbd

where rdi is the "fd" argument, which equals to 0x1 (stdout)

Adding the line "sendto: arg0 == 0x1" in 9s-seccomp.policy solved the problem.

I failed to load complete symbols for all binaries in rust-gdb so this is the most I can get from the dump file.
(gdb) list
1       in rtld.c
(gdb) bt
#0  0x00007cceacba5fbd in send ()
#1  0x00007cceacb9ea1d in __vsyslog_chk ()
#2  0x00007cceacb9ee20 in syslog ()
#3  0x00005d17c03af681 in ?? ()
#4  0x00005d17c03a5bd0 in ?? ()
#5  0x00005d17c0381fd2 in ?? ()
#6  0x00005d17c03a16ac in ?? ()
#7  0x00007cceacac7a94 in __libc_start_main ()
#8  0x00005d17c038160a in ?? ()

Yes, I had a fix for it out yesterday but it's being blamed for a flaky test on ToT: https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1370845
Project Member

Comment 6 by bugdroid1@chromium.org, Dec 12

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/1ad4dbe38a636b29abb9c94c74c0eb8a5fffeb9a

commit 1ad4dbe38a636b29abb9c94c74c0eb8a5fffeb9a
Author: Chirantan Ekbote <chirantan@chromium.org>
Date: Wed Dec 12 21:55:50 2018

vm_tools: 9s: Add send/sendto to seccomp policy

Add the sendto system call on x86 and the send system call on arm to the
seccomp policy file.  Without them the vm.CrostiniStartEverything test
fails on release builds (but not on dev builds).

BUG= chromium:913540 
TEST=`tast run -build=false $DEVICE vm.CrostiniStartEverything`

Change-Id: I7cf1b8f754deb127e8d212d9d1bd088dcfe9efd0
Signed-off-by: Chirantan Ekbote <chirantan@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/1370845
Commit-Ready: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/1ad4dbe38a636b29abb9c94c74c0eb8a5fffeb9a/vm_tools/9s/seccomp/9s-seccomp-amd64.policy
[modify] https://crrev.com/1ad4dbe38a636b29abb9c94c74c0eb8a5fffeb9a/vm_tools/9s/seccomp/9s-seccomp-arm.policy

Status: Fixed (was: Assigned)

Sign in to add a comment