Just to add some detail, I think there's at least two issues

1) The Sommelier master has a hardcoded string for the dynamic linker which only works for x86
  - if we launch with sommelier manually then wayland apps appear to work properly
2) Through experimentation it appears that a 32-bit X server cannot handle a 64-bit client properly over the unix socket, and we are explicitly disabling the TCP socket right now.

I also tried starting a local 32-bit Xwayland server without the -nolisten tcp flag to see if I could get a 64-bit client to work properly over a local TCP socket but was having issues getting the server to open the TCP socket.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/79950584ee4ad0d41b69e7e405170384c3fecf6d

commit 79950584ee4ad0d41b69e7e405170384c3fecf6d
Author: Stephen Barber <smbarber@chromium.org>
Date: Fri May 11 02:40:39 2018

common-mk: add arm USE flag to common.gypi

BUG= chromium:841641 
TEST=follow-on CL works on ARM

Change-Id: I3bd1622b5de67d55123a79e6da49fd3da22651cd
Reviewed-on: https://chromium-review.googlesource.com/1053488
Commit-Ready: Stephen Barber <smbarber@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: David Reveman <reveman@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/79950584ee4ad0d41b69e7e405170384c3fecf6d/common-mk/common.gypi

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/e511352dc1984f2a3e87ba5af1d055a8eeeece6f

commit e511352dc1984f2a3e87ba5af1d055a8eeeece6f
Author: Stephen Barber <smbarber@chromium.org>
Date: Fri May 11 02:40:39 2018

vm_tools: sommelier: fix peer_cmd_prefix on ARM

peer_cmd_prefix must invoke the dynamic linker correctly on ARM.

BUG= chromium:841641 
TEST=emerge-tael vm_guest_tools

Change-Id: Ida6338831860ba17c31e841b1d2afff67767f464
Reviewed-on: https://chromium-review.googlesource.com/1053489
Commit-Ready: Stephen Barber <smbarber@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: David Reveman <reveman@chromium.org>

[modify] https://crrev.com/e511352dc1984f2a3e87ba5af1d055a8eeeece6f/vm_tools/sommelier/sommelier.gyp

smbarber, is this fixed?

Issue 1 is fixed, I don't believe  issue 2  is fixed -- reveman@ said he'd try to take a look this week

Looks like we're missing the following libraries in /opt/google/cros-containers/lib:

libnss_compat.so.2
libnss_nis.so.2
libnss_nsl.so.1
libnss_files.so.2

They are dynamically loaded when the first client connects so not too surprising that we missed them. Not sure why they are not needed on x86. Is our xwayland build exactly the same on arm and x86?

Here's how I came to the above conclusion:

1. Initially after a fresh container install, it was failing with the "No protocol" error we saw last week.
2. I built arm64 version of Xwayland inside the container to verify that it worked (it did).
3. Then I thought it be useful to try reproduce the error using my own 32 bit build of Xwayland inside the container.
4. I added armhf toolchain and installed the packages needed for building 32 bit version of Xwayland.
5. Resulting binary had no issues connecting to arm64 clients.
6. I then tried /opt/google/cros-containers/bin/Xwayland.elf again a realized that it had also started working.
7. "LD_DEBUG=libs DISPLAY=:1 /opt/google/cros-containers/bin/Xwayland $DISPLAY && xterm" was showing that we load the above mentioned libraries from /lib/arm-linux-gnueabihf/

Sonny, I'll assign this back to you and let you decide if we can adjust the build to not require these libraries or just include them.

x86 is 64-bit for client and server, so maybe they end up using the 64-bit versions of those libraries that are provided by Debian?  On ARM we have to provide all of the necessary 32-bit libraries ourselves.

Steve probably understands how these pieces are built and packaged much better than me -- Steve, could you take a look?

Ah, so we should probably includes these for x86 too then to avoid having to rely on packages inside the container.

In progress: https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1068620

the set of nss files needed is determined by the /etc/nsswitch.conf file.  do we have control over that ?  i'm guessing not.

if the container installs custom nss modules (e.g. nss_ldap or nss_mdns or other stuff), then changes nsswitch.conf to refer to them, we're going to be in the same situation where our copy of glibc tries to dlopen these nss modules and fails.

i don't think our daemons care about network accesses right ?  why is xwayland hitting nss ?  is it to look up some user/group accounts to resolve them to uid/gid ?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/a1131e0d6c1ee2427667dab3f7da23b0a06e1eb5

commit a1131e0d6c1ee2427667dab3f7da23b0a06e1eb5
Author: Stephen Barber <smbarber@chromium.org>
Date: Wed May 23 04:57:11 2018

termina_container_tools: add libnss for Xwayland

glibc will dlopen() libnss, and for some reason that is necessary for
Xwayland work.

BUG= chromium:841641 
TEST=sommelier -X works on kevin

Change-Id: I73b27aab7f346bd24ce6c8ace08752a3ba57d447
Reviewed-on: https://chromium-review.googlesource.com/1068620
Commit-Ready: Stephen Barber <smbarber@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>

[rename] https://crrev.com/a1131e0d6c1ee2427667dab3f7da23b0a06e1eb5/chromeos-base/termina_container_tools/termina_container_tools-0.0.1-r9.ebuild
[modify] https://crrev.com/a1131e0d6c1ee2427667dab3f7da23b0a06e1eb5/chromeos-base/termina_container_tools/termina_container_tools-0.0.1.ebuild

@smbarber is this done? Anything that needs to be in for M69?

Yep this is done. Verified on M69.