On Nexus5X enable ScreenCapture in chrome://flags and start screen capture.
Nexus5X will crash with stack like below:

12-16 09:48:21.692 12134 12134 F DEBUG   : backtrace:
12-16 09:48:21.694 12134 12134 F DEBUG   :     #00 pc 00049db4  /system/lib/libc.so (tgkill+12)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #01 pc 00047553  /system/lib/libc.so (pthread_kill+34)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #02 pc 0001d8a5  /system/lib/libc.so (raise+10)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #03 pc 000193f1  /system/lib/libc.so (__libc_android_abort+34)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #04 pc 00017034  /system/lib/libc.so (abort+4)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #05 pc 0031a5f1  /system/lib/libart.so (_ZN3art7Runtime5AbortEPKc+328)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #06 pc 000b5205  /system/lib/libart.so (_ZN3art10LogMessageD2Ev+1132)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #07 pc 00239b35  /system/lib/libart.so (_ZN3art9JavaVMExt8JniAbortEPKcS2_+1664)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #08 pc 00239d9b  /system/lib/libart.so (_ZN3art9JavaVMExt9JniAbortFEPKcS2_z+66)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #09 pc 0027f5c9  /system/lib/libart.so (_ZN3art3JNI19NewDirectByteBufferEP7_JNIEnvPvx+92)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #10 pc 000c8009  /system/lib/libart.so (_ZN3art8CheckJNI19NewDirectByteBufferEP7_JNIEnvPvx+484)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #11 pc 0001ed5d  /system/lib/libmedia_jni.so
12-16 09:48:21.694 12134 12134 F DEBUG   :     #12 pc 019ea64d  /system/framework/arm/boot-framework.oat (offset 0x158b000) (android.media.ImageReader$SurfaceImage.nativeCreatePlanes+104)
12-16 09:48:21.694 12134 12134 F DEBUG   :     #13 pc 019eac69  /system/framework/arm/boot-framework.oat (offset 0x158b000) (android.media.ImageReader$SurfaceImage.getPlanes+108)

Switching the default image format in ScreenCapture from YUV_420 to RGBA_8888 will stop the crash and start screen capture on N5X successfully.

The reason should be that the VirtualDisplay infrastructure doesn't provide very many strong guarantees about image format support. RGBA_8888 works everywhere, and beyond that, it gets dicey. We have a fallback-to-RGBA_8888 to catch the format-unsupported exception. But N5x just crashes before we can do anything.

A feature request to support YUV in VirtualDisplay has been filed to Android team. So before that we'd better to stay with RGBA_8888.