cgroupfs mount is not good.
This is the mountinfo for systemd pid.

50 76 0:55 / /sys/fs/cgroup rw,nosuid,nodev,noexec - tmpfs tmpfs rw,mode=755,uid=1000000,gid=1000000
51 50 0:33 /lxc/stretch /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,name=systemd
52 50 0:29 /lxc/stretch /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpuset
53 50 0:30 /lxc/stretch /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,blkio
54 50 0:27 /lxc/stretch /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,devices
55 50 0:32 /lxc/stretch /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,pids
56 50 0:31 /lxc/stretch /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,memory
58 50 0:26 /lxc/stretch /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,freezer

Is it one of those mounts causing the problem or all of them?

Inside of the lxd container, (after systemd doing whatever) it looks like this:
drwxrwxr-x 2 nobody root  0 Mar 19 21:51 blkio
drwxr-xr-x 2 root   root 40 Mar 19 21:51 cpu,cpuacct ----- empty
drwxrwxr-x 2 nobody root  0 Mar 19 21:51 cpuset
drwxr-xr-x 2 root   root 40 Mar 19 21:51 debug
drwxrwxr-x 5 nobody root  0 Mar 19 21:51 devices
drwxrwxr-x 2 nobody root  0 Mar 19 21:51 freezer
drwxrwxr-x 2 nobody root  0 Mar 19 21:51 memory
drwxr-xr-x 2 root   root 40 Mar 19 21:51 net_cls  ----- empty 
drwxrwxr-x 5 nobody root  0 Mar 19 21:51 pids
drwxrwxr-x 5 nobody root  0 Mar 19 21:51 systemd


While docker expect cpu,cpuacct mounted correctly and symlink cpu->cpu,cpuacct and cpuacct->cpuacct.


Apparently, lxd use cgroup namespace (/cgroup/<controller>/lxc/<container-name>). lxd does not work well with current cgroupfs mounting in termina and thus cgroup is not mounted good in the container.
I am trying to mimic what ubuntu is doing and testing it.

BTW, it's worth mentioning that I could not mount net_prio in termina.

Is the kernel config not set for the guest kernel?

If it's using src/third_party/kernel/v4.14/arch/x86/configs/chromiumos-container-vm-x86_64_defconfig

Then no, the flag is not set. Though this should not block us from running docker.

I modified maitred to this:
dr-xr-xr-x 4 root root  0 Apr  4 21:45 blkio
lrwxrwxrwx 1 root root 26 Apr  4 21:45 cpu -> /sys/fs/cgroup/cpu,cpuacct
dr-xr-xr-x 4 root root  0 Apr  4 21:45 cpu,cpuacct
lrwxrwxrwx 1 root root 26 Apr  4 21:45 cpuacct -> /sys/fs/cgroup/cpu,cpuacct
dr-xr-xr-x 4 root root  0 Apr  4 21:45 cpuset
dr-xr-xr-x 4 root root  0 Apr  4 21:45 devices
dr-xr-xr-x 4 root root  0 Apr  4 21:45 freezer
dr-xr-xr-x 4 root root  0 Apr  4 21:45 memory
lrwxrwxrwx 1 root root 31 Apr  4 21:45 net_cls -> /sys/fs/cgroup/net_cls,net_prio
dr-xr-xr-x 4 root root  0 Apr  4 21:45 net_cls,net_prio
dr-xr-xr-x 4 root root  0 Apr  4 21:45 pids
dr-xr-xr-x 4 root root  0 Apr  4 21:45 systemd



This is pretty close to what it looks like in Ubuntu:
dr-xr-xr-x 6 root root  0 Mar 19 14:52 blkio
lrwxrwxrwx 1 root root 11 Mar 19 14:52 cpu -> cpu,cpuacct
lrwxrwxrwx 1 root root 11 Mar 19 14:52 cpuacct -> cpu,cpuacct
dr-xr-xr-x 6 root root  0 Mar 19 14:52 cpu,cpuacct
dr-xr-xr-x 3 root root  0 Mar 19 14:52 cpuset
dr-xr-xr-x 6 root root  0 Mar 19 14:52 devices
dr-xr-xr-x 4 root root  0 Mar 19 14:52 freezer
dr-xr-xr-x 3 root root  0 Mar 19 14:52 hugetlb
dr-xr-xr-x 7 root root  0 Mar 19 14:52 memory
lrwxrwxrwx 1 root root 16 Mar 19 14:52 net_cls -> net_cls,net_prio
dr-xr-xr-x 3 root root  0 Mar 19 14:52 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Mar 19 14:52 net_prio -> net_cls,net_prio
dr-xr-xr-x 3 root root  0 Mar 19 14:52 perf_event
dr-xr-xr-x 6 root root  0 Mar 19 14:52 pids
dr-xr-xr-x 3 root root  0 Mar 19 14:52 rdma
dr-xr-xr-x 7 root root  0 Mar 19 14:52 systemd


cpu, cpuacct are still not mounted properly in the container. in the container: 
49 74 0:56 / /sys/fs/cgroup rw,nosuid,nodev,noexec - tmpfs tmpfs rw,mode=755,uid=1000000,gid=1000000
50 49 0:34 /lxc/stretch /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,name=systemd
51 49 0:33 /lxc/stretch /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,pids
52 49 0:29 /lxc/stretch /sys/fs/cgroup/net_cls rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,net_cls
53 49 0:31 /lxc/stretch /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,blkio
54 49 0:32 /lxc/stretch /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,memory
55 49 0:28 /lxc/stretch /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,devices
56 49 0:27 /lxc/stretch /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,freezer
57 49 0:30 /lxc/stretch /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpuset


I do find /sys/fs/cgroup/cpu/lxc stretch in Termina, (and user/group is root, 1000000), but it's not in the lxd container. I checked lxd code, looks like it passes everything to lxc. Starting to look into lxc code. Please comment with any suggestions.

Actually, I have a theory now. According to systemd code: (/src/core/mount-setup.c), it's trying to mount "join_controller" to cpu,cpuacct. 
Based on this: http://man7.org/linux/man-pages/man7/cgroups.7.html
I should be able to mount like this:
           mount -t cgroup -o cpu,cpuacct none /sys/fs/cgroup/cpu,cpuacct
But when I do this in termina, I got:
(termina) localhost /sys/fs/cgroup # mount -t cgroup cgroup -o cpu,cpuacct cpu 
mount: cgroup is already mounted or /sys/fs/cgroup/cpu,cpuacct busy
       cgroup is already mounted on /sys/fs/cgroup/freezerp -o cpu,cpuacct /sys/f
       cgroup is already mounted on /sys/fs/cgroup/devices
       cgroup is already mounted on /sys/fs/cgroup/net_cls,net_prio
       cgroup is already mounted on /sys/fs/cgroup/cpuset
       cgroup is already mounted on /sys/fs/cgroup/blkio
       cgroup is already mounted on /sys/fs/cgroup/memory
       cgroup is already mounted on /sys/fs/cgroup/pids
       cgroup is already mounted on /sys/fs/cgroup/systemd


systemd might failed at this step. Maybe we want to fix this first.

And, in the lxd container, I can run :
mount -t cgroup cgroup -o cpu cpu,cpuacct
mount -t cgroup cgroup -o cpuacct cpu,cpuacct 

But when I run:
mount -t cgroup cgroup -o cpu,cpuacct cpu,cpuacct
I got permission denied

chirantan can help with the initial cgroup setup as he wrote our termina init.

are we missing another kernel option to allow the joint mounting of cpu and cpuacct

Don't find anything related to that. 

As it complained "cgroup is already mounted on /sys/blah-blah", I umounted everything in /sys/fs/cgroup/* and have this(in termina):

(termina) localhost /sys/fs/cgroup # mount -t cgroup cgroup -o cpu,cpuacct cpu
mount: cgroup is already mounted or /sys/fs/cgroup/cpu,cpuacct busy


AND:
(termina) localhost /sys/fs/cgroup # cat /proc/mounts 
/dev/root / ext4 rw,relatime,block_validity,delalloc,nojournal_checksum,barrier,user_xattr,acl 0 0
devtmpfs /dev devtmpfs rw,nosuid,noexec,relatime,size=506036k,nr_inodes=126509,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
tmp /tmp tmpfs rw,nosuid,nodev,noexec,relatime 0 0
run /run tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
shmfs /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
var /var tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
none /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
tmpfs /var/lib/lxd/shmounts tmpfs rw,relatime,size=100k,mode=711 0 0
tmpfs /var/lib/lxd/devlxd tmpfs rw,relatime,size=100k,mode=755 0 0
/dev/vdb /mnt/stateful btrfs rw,relatime,noacl,space_cache,subvolid=5,subvol=/ 0 0
/dev/vdb /var/lib/lxd/storage-pools/default btrfs rw,relatime,noacl,space_cache,subvolid=262,subvol=/lxd/storage-pools/default 0 0

Found an old thread: https://bugzilla.redhat.com/show_bug.cgi?id=612805,
KVM is involved. Looking into the thread.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/eebab4f9e9a1e54f70d74eb3fec6e5de42761fd6

commit eebab4f9e9a1e54f70d74eb3fec6e5de42761fd6
Author: Jingkui Wang <jkwang@google.com>
Date: Sat Apr 07 02:50:45 2018

vm_tools: Update cgroup mounting in maitred

Mounting cgroupfs differently and make it compatible with docker in lxd.

BUG= chromium:828694 
TEST=local build and test

Change-Id: I77ef42f81bc4fe293378de1df94e8616af5a49e4
Reviewed-on: https://chromium-review.googlesource.com/999018
Commit-Ready: Jingkui Wang <jkwang@google.com>
Tested-by: Jingkui Wang <jkwang@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>

[modify] https://crrev.com/eebab4f9e9a1e54f70d74eb3fec6e5de42761fd6/vm_tools/maitred/init.cc

This is interesting.
https://github.com/lxc/lxd/issues/2977