Splitting off the XNU issue details from  bug 756102  into here to keep the details restricted. I just submitted this report to Apple product security.

Follow-up: 671348000
--------------------

*Summary:*
A local, unprivileged, sandboxed process can trivially cause a local denial
of service/kernel panic by issuing one of several system calls.

*Impact:*
This issue affects macOS 10.11.6, 10.12.6, and 10.13 beta. macOS 10.10.5
and 10.9.5 are not affected. iOS was untested but is likely also affected.

*Root Cause:*
Several Mach traps make a call to waitq_link_reserve(). In 10.12.4
xnu-3789.1.32, this function calls wql_ensure_free_space(), which calls
ltable_grow(). If the waitq link table exceeds its maximum element size,
the system will panic with "No more room to grow table". Because
waitq_link_reserve() is not expected to fail, it is called from trap
handlers without any capacity checks, which means that by issuing a system
call, an unprivileged process can flood the waitq link table and cause a
kernel panic.

Examining xnu-3789.1.32, there are several userspace-accessible traps that
may be vulnerable to this:
- mach_port_allocate for MACH_PORT_RIGHT_PORT_SET
- mach_port_allocate_full for MACH_PORT_RIGHT_PORT_SET
- mach_port_insert_right
- mach_port_move_member
- kevent for EVFILT_MACHPORT
- select

I've attached two small test programs for the port set operations to
demonstrate the problem. Running either on the affected macOS versions will
cause a kernel panic within seconds. This issue is triggerable from within
the confines of the sandbox, including the web process sandboxes used by
Safari and Chrome.

The waitq-link-pset.c program merely allocates port sets in an unbounded
loop, until the waitq link table is exhausted and the system panics. The
waitq-link-pset-insert.c program is more interesting, in that it races
amongst threads the use of mach_port_insert_member(). While
mach_port_insert_member() does call waitq_link_release() at the end of the
routine, other threads (or tasks) may cause the waitq link table to become
exhausted if they are modifying port sets concurrently.

Note that the above is not an exhaustive list of calls to
waitq_link_reserve(), and others within xnu should be evaluated.

*Remediation:*
At minimum, the waitq_link_reserve() function should be able to return an
error if a link reservation fails, rather than causing a panic. This would
allow system call handlers to report up KERN_RESOURCE_SHORTAGE in this
scenario. If waitq-link-pset.c is run on macOS 10.10, for example,
KERN_NO_SPACE is returned.

More broadly, the introduction of this system-wide limit on waitq links
effectively puts a cap on the number of port sets that may be allocated
within the system. Even if waitq_link_reserve() were to return an error
when the link table is full, an unprivileged process would still be able to
cause denial-of-service to other privileged processes by preventing the
creation of new waitq links (e.g., port sets, kevents, etc.). Prior to the
introduction of the waitq link table in macOS 10.11, the limit on port sets
was a per-task resource.

*Comments:*
For port sets, the system-wide limit of 262142 is too low. This issue was
discovered because the kernel panic was reported by users who were
experiencing it on systems with moderate-to-heavy process usage. If
possible, it would be best to remove the system-wide limit on port sets and
restore the previous behavior in which they were a per-task resource. We
have observed both the mach_port_insert_member() race and the
mach_port_allocate() exhaustion in reports from users.

*Disclosure deadline:*
This bug is subject to a 90 day disclosure deadline. If 90 days elapse
without a broadly available patch, then the bug report will automatically
become visible to the public.
 

Deleted: Kernel_2017-08-17-185745_Ten-Twelve-VM.panic 5.0 KB

Kernel_2017-08-17-185745_Ten-Twelve-VM.panic 5.0 KB Download

Deleted: Kernel_2017-08-17-190120_Ten-Twelve-VM.panic 5.2 KB

Kernel_2017-08-17-190120_Ten-Twelve-VM.panic 5.2 KB Download

Deleted: waitq-link-pset.c 515 bytes

waitq-link-pset.c 515 bytes View Download

Deleted: waitq-link-pset-insert.c 1.6 KB

waitq-link-pset-insert.c 1.6 KB View Download