I verified that a .core, a .dmp and a .meta file is created when authpolicyd crashes. When a binary called by authpolicyd crashes, there's no core dump, but the exit code is reported back to authpolicyd.

As far as I can tell, everything works as intended. Is there anything else we have to check?

How do you set up post-launch monitoring?

This was brought forward by Mattias, iirc.  I'm not sure what the typical setup is for CrOS system daemons, but it might possibly include uploading crashdumps to Google and symbolizing them and including them in some kind of crash dashboard.

Mattias, what exactly did you have in mind?

CCing Jorge, I believe Mattias is ultimately enjoying his parental leave now.

+Mike who probably knows more about crash reporting than me.

It's unclear to me what this bug is actually about. Is it about ensuring crash reports are generated? About finding those crash reports in the crash dashboard?

crash reporting for the system is all automated.  daemons don't have to do anything special to get it working.

it would help to clarify "When a binary called by authpolicyd crashes".  *how* did it crash ?  did it violate a seccomp filter ?  those get killed with SIGKILL and, by design, the kernel does not coredump it.

look at /var/log/messages to see if there's anything interesting there

"When a binary called by authpolicyd crashes" almost always means seccomp. I don't recall a single case where a binary crashed because of something else.

I'm curious how seccomp failures are handled in other projects. If not through coredumps, are they reported in any other way, e.g. UMA stats? How do teams find out if a seccomp filter is too stringent in the open field?

if "all" you're worried about is seccomp failures, then let's make this general because your question applies to all of them

i agree that we should have a way to get a coredump here.  i don't think it's possible today (adding a few security people here to keep me honest), so we should see if we can extend the kernel to support this.

throwing a few ideas out there:
- add a sysctl knob so that all seccomp failures trigger a coredump before it sends the SIGKILL
- add a new seccomp return value like SECCOMP_RET_KILL_DUMP that dumps core and then hits the KILL paths

In Chrome, we had a mode that would segfault the program for certain seccomp failures, at a predictable address. This crash would get picked up by the crash reporter and would allow us to debug seccomp problems in the field. If interested, we could implement this in Minijail.

that doesn't sound like a reliable solution.  it works in Chrome (mostly) because we have full control over the source and can set up the code to do it.  when we're running arbitrary programs we don't want to patch, making sure signal handlers do the right thing sounds patchy.

i don't think what we want here (seccomp failure triggering a coredump) is unique to Chromium OS.  i can think of at least one app myself where this would help.

it might even be simpler: always have SECCOMP_RET_KILL trigger a coredump.  this makes sense too as that is documented as (immediately) killing the process as if SIGSYS was sent and not caught.  and SIGSYS is a signal-that-dumps-core.

let's see how that argument works upstream :).

There's one detail about that which is SECCOMP_RET_KILL only kills the failing thread and not the whole process. In multithreaded programs, we usually make sure SIGSYS is not caught and we return with SECCOMP_RET_TRAP which will kill the whole program. So this might impact crash dumps in multithreaded programs.

(See https://android.googlesource.com/platform/external/minijail/+/713f6fbed8fd14efce499d5d609d24487b2518a4 for context)

Hmm, SIG_KERNEL_COREDUMP_MASK includes SIGSYS. Oh, I see the problem: seccomp goes directly to do_exit(), never passing through get_signal() which does the coredump checking.

So, for seccomp to dumpcore, it'd need to replicate the logic in get_signal()... I'm not opposed to the idea, but I think it'll need some testing; signal handling there can be fragile.

i think my CL covers the issues you raised:
  https://chromium-review.googlesource.com/429332

i've sent that CL to LKML to garner feedback

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d9badf1d90529eada257d53f21200db3e935df5d

commit d9badf1d90529eada257d53f21200db3e935df5d
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Jan 18 07:23:19 2017

UPSTREAM: seccomp: dump core when using SECCOMP_RET_KILL

The SECCOMP_RET_KILL mode is documented as immediately killing the
process as if a SIGSYS had been sent and not caught (similar to a
SIGKILL).  However, a SIGSYS is documented as triggering a coredump
which does not happen today.

This has the advantage of being able to more easily debug a process
that fails a seccomp filter.  Today, most apps need to recompile and
change their filter in order to get detailed info out, or manually run
things through strace, or enable detailed kernel auditing.  Now we get
coredumps that fit into existing system-wide crash reporting setups.

From a security pov, this shouldn't be a problem.  Unhandled signals
can already be sent externally which trigger a coredump independent of
the status of the seccomp filter.  The act of dumping core itself does
not cause change in execution of the program.

BUG= chromium:676357 
TEST=`minijail0 -n -S /tmp/an-empty-file /bin/ls` triggered a coredump
Signed-off-by: Mike Frysinger <vapier@chromium.org>
Acked-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I50e2354d6550b0381103e29d42b16c134f5ec1f6
Reviewed-on: https://chromium-review.googlesource.com/429332
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/d9badf1d90529eada257d53f21200db3e935df5d/kernel/seccomp.c

chromad guys: can you double check things on your end ?  it's in linux-4.4 (which means the xxx-generic bots), so you should be able to test there.  otherwise i'll land it one by one in the other kernels if no one complains (3.18 is in CQ now).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/744c0cbbb91d339e5a207c411cea1c9424aecc2a

commit 744c0cbbb91d339e5a207c411cea1c9424aecc2a
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Jan 18 07:23:19 2017

UPSTREAM: seccomp: dump core when using SECCOMP_RET_KILL

The SECCOMP_RET_KILL mode is documented as immediately killing the
process as if a SIGSYS had been sent and not caught (similar to a
SIGKILL).  However, a SIGSYS is documented as triggering a coredump
which does not happen today.

This has the advantage of being able to more easily debug a process
that fails a seccomp filter.  Today, most apps need to recompile and
change their filter in order to get detailed info out, or manually run
things through strace, or enable detailed kernel auditing.  Now we get
coredumps that fit into existing system-wide crash reporting setups.

From a security pov, this shouldn't be a problem.  Unhandled signals
can already be sent externally which trigger a coredump independent of
the status of the seccomp filter.  The act of dumping core itself does
not cause change in execution of the program.

BUG= chromium:676357 
TEST=`minijail0 -n -S /tmp/an-empty-file /bin/ls` triggered a coredump
Signed-off-by: Mike Frysinger <vapier@chromium.org>
Acked-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I50e2354d6550b0381103e29d42b16c134f5ec1f6
Reviewed-on: https://chromium-review.googlesource.com/431400
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>

[modify] https://crrev.com/744c0cbbb91d339e5a207c411cea1c9424aecc2a/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/744c0cbbb91d339e5a207c411cea1c9424aecc2a

commit 744c0cbbb91d339e5a207c411cea1c9424aecc2a
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Jan 18 07:23:19 2017

UPSTREAM: seccomp: dump core when using SECCOMP_RET_KILL

The SECCOMP_RET_KILL mode is documented as immediately killing the
process as if a SIGSYS had been sent and not caught (similar to a
SIGKILL).  However, a SIGSYS is documented as triggering a coredump
which does not happen today.

This has the advantage of being able to more easily debug a process
that fails a seccomp filter.  Today, most apps need to recompile and
change their filter in order to get detailed info out, or manually run
things through strace, or enable detailed kernel auditing.  Now we get
coredumps that fit into existing system-wide crash reporting setups.

From a security pov, this shouldn't be a problem.  Unhandled signals
can already be sent externally which trigger a coredump independent of
the status of the seccomp filter.  The act of dumping core itself does
not cause change in execution of the program.

BUG= chromium:676357 
TEST=`minijail0 -n -S /tmp/an-empty-file /bin/ls` triggered a coredump
Signed-off-by: Mike Frysinger <vapier@chromium.org>
Acked-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I50e2354d6550b0381103e29d42b16c134f5ec1f6
Reviewed-on: https://chromium-review.googlesource.com/431400
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>

[modify] https://crrev.com/744c0cbbb91d339e5a207c411cea1c9424aecc2a/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/471d6d8dc0dfbc06a93b0f13e39fb7a9f4f7ce85

commit 471d6d8dc0dfbc06a93b0f13e39fb7a9f4f7ce85
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat May 13 20:34:26 2017

UPSTREAM: seccomp: dump core when using SECCOMP_RET_KILL

The SECCOMP_RET_KILL mode is documented as immediately killing the
process as if a SIGSYS had been sent and not caught (similar to a
SIGKILL).  However, a SIGSYS is documented as triggering a coredump
which does not happen today.

This has the advantage of being able to more easily debug a process
that fails a seccomp filter.  Today, most apps need to recompile and
change their filter in order to get detailed info out, or manually run
things through strace, or enable detailed kernel auditing.  Now we get
coredumps that fit into existing system-wide crash reporting setups.

From a security pov, this shouldn't be a problem.  Unhandled signals
can already be sent externally which trigger a coredump independent of
the status of the seccomp filter.  The act of dumping core itself does
not cause change in execution of the program.

BUG= chromium:676357 
TEST=`minijail0 -n -S /tmp/an-empty-file /bin/ls` triggered a coredump
Signed-off-by: Mike Frysinger <vapier@chromium.org>
Acked-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I50e2354d6550b0381103e29d42b16c134f5ec1f6
Reviewed-on: https://chromium-review.googlesource.com/431436
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>

[modify] https://crrev.com/471d6d8dc0dfbc06a93b0f13e39fb7a9f4f7ce85/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/6ada8c5aee1adb200d00ed3f28a1600caec57ea9

commit 6ada8c5aee1adb200d00ed3f28a1600caec57ea9
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat May 13 20:34:27 2017

UPSTREAM: seccomp: dump core when using SECCOMP_RET_KILL

The SECCOMP_RET_KILL mode is documented as immediately killing the
process as if a SIGSYS had been sent and not caught (similar to a
SIGKILL).  However, a SIGSYS is documented as triggering a coredump
which does not happen today.

This has the advantage of being able to more easily debug a process
that fails a seccomp filter.  Today, most apps need to recompile and
change their filter in order to get detailed info out, or manually run
things through strace, or enable detailed kernel auditing.  Now we get
coredumps that fit into existing system-wide crash reporting setups.

From a security pov, this shouldn't be a problem.  Unhandled signals
can already be sent externally which trigger a coredump independent of
the status of the seccomp filter.  The act of dumping core itself does
not cause change in execution of the program.

BUG= chromium:676357 
TEST=`minijail0 -n -S /tmp/an-empty-file /bin/ls` triggered a coredump
Signed-off-by: Mike Frysinger <vapier@chromium.org>
Acked-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I50e2354d6550b0381103e29d42b16c134f5ec1f6
Reviewed-on: https://chromium-review.googlesource.com/431456
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>

[modify] https://crrev.com/6ada8c5aee1adb200d00ed3f28a1600caec57ea9/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/333750fbd8cbed6daeab534f559559e360593717

commit 333750fbd8cbed6daeab534f559559e360593717
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat May 13 20:34:28 2017

UPSTREAM: seccomp: dump core when using SECCOMP_RET_KILL

The SECCOMP_RET_KILL mode is documented as immediately killing the
process as if a SIGSYS had been sent and not caught (similar to a
SIGKILL).  However, a SIGSYS is documented as triggering a coredump
which does not happen today.

This has the advantage of being able to more easily debug a process
that fails a seccomp filter.  Today, most apps need to recompile and
change their filter in order to get detailed info out, or manually run
things through strace, or enable detailed kernel auditing.  Now we get
coredumps that fit into existing system-wide crash reporting setups.

From a security pov, this shouldn't be a problem.  Unhandled signals
can already be sent externally which trigger a coredump independent of
the status of the seccomp filter.  The act of dumping core itself does
not cause change in execution of the program.

BUG= chromium:676357 
TEST=`minijail0 -n -S /tmp/an-empty-file /bin/ls` triggered a coredump
Signed-off-by: Mike Frysinger <vapier@chromium.org>
Acked-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I50e2354d6550b0381103e29d42b16c134f5ec1f6
Reviewed-on: https://chromium-review.googlesource.com/430857
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>

[modify] https://crrev.com/333750fbd8cbed6daeab534f559559e360593717/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/df085da3cdf9c22f4a46b22ba8584f5b478d4d37

commit df085da3cdf9c22f4a46b22ba8584f5b478d4d37
Author: Kees Cook <keescook@chromium.org>
Date: Tue May 16 03:51:39 2017

UPSTREAM: seccomp: Only dump core when single-threaded

The SECCOMP_RET_KILL filter return code has always killed the current
thread, not the entire process. Changing this as a side-effect of dumping
core isn't a safe thing to do (a few test suites have already flagged this
behavioral change). Instead, restore the RET_KILL semantics, but still
dump core when a RET_KILL delivers SIGSYS to a single-threaded process.

BUG= chromium:676357 
TEST=precq passes
Fixes: b25e67161c29 ("seccomp: dump core when using SECCOMP_RET_KILL")
Signed-off-by: Kees Cook <keescook@chromium.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I38bed8dd7a8babd29f4b6275b47143b994ddaf8c
Reviewed-on: https://chromium-review.googlesource.com/505710
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Kees Cook <keescook@chromium.org>

[modify] https://crrev.com/df085da3cdf9c22f4a46b22ba8584f5b478d4d37/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/886bea5a7ebf66858a81c931c76da3c9da87f860

commit 886bea5a7ebf66858a81c931c76da3c9da87f860
Author: Kees Cook <keescook@chromium.org>
Date: Tue May 16 03:51:38 2017

UPSTREAM: seccomp: Only dump core when single-threaded

The SECCOMP_RET_KILL filter return code has always killed the current
thread, not the entire process. Changing this as a side-effect of dumping
core isn't a safe thing to do (a few test suites have already flagged this
behavioral change). Instead, restore the RET_KILL semantics, but still
dump core when a RET_KILL delivers SIGSYS to a single-threaded process.

BUG= chromium:676357 
TEST=precq passes
Fixes: b25e67161c29 ("seccomp: dump core when using SECCOMP_RET_KILL")
Signed-off-by: Kees Cook <keescook@chromium.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I38bed8dd7a8babd29f4b6275b47143b994ddaf8c
Reviewed-on: https://chromium-review.googlesource.com/505749
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Kees Cook <keescook@chromium.org>

[modify] https://crrev.com/886bea5a7ebf66858a81c931c76da3c9da87f860/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2e1bcf2452f99059f8fbb50321ae74eb2b8f34d3

commit 2e1bcf2452f99059f8fbb50321ae74eb2b8f34d3
Author: Kees Cook <keescook@chromium.org>
Date: Tue May 16 07:09:25 2017

UPSTREAM: seccomp: Only dump core when single-threaded

The SECCOMP_RET_KILL filter return code has always killed the current
thread, not the entire process. Changing this as a side-effect of dumping
core isn't a safe thing to do (a few test suites have already flagged this
behavioral change). Instead, restore the RET_KILL semantics, but still
dump core when a RET_KILL delivers SIGSYS to a single-threaded process.

BUG= chromium:676357 
TEST=precq passes
Fixes: b25e67161c29 ("seccomp: dump core when using SECCOMP_RET_KILL")
Signed-off-by: Kees Cook <keescook@chromium.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I38bed8dd7a8babd29f4b6275b47143b994ddaf8c
Reviewed-on: https://chromium-review.googlesource.com/505709
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Kees Cook <keescook@chromium.org>

[modify] https://crrev.com/2e1bcf2452f99059f8fbb50321ae74eb2b8f34d3/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/cb4d184804a129a318145df789c4cbd43c1bf0db

commit cb4d184804a129a318145df789c4cbd43c1bf0db
Author: Kees Cook <keescook@chromium.org>
Date: Tue May 16 07:09:23 2017

UPSTREAM: seccomp: Only dump core when single-threaded

The SECCOMP_RET_KILL filter return code has always killed the current
thread, not the entire process. Changing this as a side-effect of dumping
core isn't a safe thing to do (a few test suites have already flagged this
behavioral change). Instead, restore the RET_KILL semantics, but still
dump core when a RET_KILL delivers SIGSYS to a single-threaded process.

BUG= chromium:676357 
TEST=precq passes
Fixes: b25e67161c29 ("seccomp: dump core when using SECCOMP_RET_KILL")
Signed-off-by: Kees Cook <keescook@chromium.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I38bed8dd7a8babd29f4b6275b47143b994ddaf8c
Reviewed-on: https://chromium-review.googlesource.com/505552
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Kees Cook <keescook@chromium.org>

[modify] https://crrev.com/cb4d184804a129a318145df789c4cbd43c1bf0db/kernel/seccomp.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/kernel/+/dc195c94c1ed3c45329d27ee540186809c962b1f

commit dc195c94c1ed3c45329d27ee540186809c962b1f
Author: Kees Cook <keescook@chromium.org>
Date: Thu May 18 02:06:19 2017

UPSTREAM: seccomp: Only dump core when single-threaded

The SECCOMP_RET_KILL filter return code has always killed the current
thread, not the entire process. Changing this as a side-effect of dumping
core isn't a safe thing to do (a few test suites have already flagged this
behavioral change). Instead, restore the RET_KILL semantics, but still
dump core when a RET_KILL delivers SIGSYS to a single-threaded process.

BUG= chromium:676357 
TEST=precq passes
Fixes: b25e67161c29 ("seccomp: dump core when using SECCOMP_RET_KILL")
Signed-off-by: Kees Cook <keescook@chromium.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: James Morris <james.l.morris@oracle.com>

Change-Id: I38bed8dd7a8babd29f4b6275b47143b994ddaf8c
Reviewed-on: https://chromium-review.googlesource.com/505570
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Kees Cook <keescook@chromium.org>

[modify] https://crrev.com/dc195c94c1ed3c45329d27ee540186809c962b1f/kernel/seccomp.c

all our kernels should support dumping core for single threaded processes using seccomp.  multithreaded doesn't work, but i don't think that impacts any of our daemons.  going to close this out as i don't have plans to work on this.

bulk Verify of Chromad V1 bugs