New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 895822 link

Starred by 2 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Device policy does not load in VM

Project Member Reported by pmarko@chromium.org, Oct 16

Issue description

We've received reports of device policy not loading when Chrome OS is run in VMs. chrome://policy displays "Last fetched: Never" for device policy

The task here is to investigate what is going on and fix any bugs found.

Repro steps:
(1) cros chrome-sdk --board=betty-arcnext --download-vm --clear-sdk-cache --log-level info
(2) Enroll into @managedchrome.com
(3) Check device policy status

 
Status: Assigned (was: Untriaged)
Was unable to reproduce on my machine with:

cros chrome-sdk --board=betty-arcnext --download-vm --clear-sdk-cache --log-level info

All policies were loaded fine and "Last fetched: 0 secs ago" was shown. 
I could repro the Last Fetched: Never again by using the same VM startup line voit@ used in Comment #2.

I see Last Fetched: Never for Device Policy in guest and user session.
Will investigate what's going on now. The only thing that comes to my mind is that the device policy refresh scheduler somehow fails, so I'll add logging there as a first attempt.

Status: Started (was: Assigned)
It seems that DeviceCloudPolicyInitializer::TryToCreateClient decides to exit early, becuase |state_keys_broker_->pending()| is true.

2018-10-22T07:52:37.829207+02:00 WARNING session_manager[738]: [WARNING:server_backed_state_key_generator.cc(123)] Machine serial number missing!
2018-10-22T07:52:37.829229+02:00 INFO session_manager[738]: [INFO:server_backed_state_key_generator.cc(126)] Stable device secret missing!
2018-10-22T07:52:37.829252+02:00 WARNING session_manager[738]: [WARNING:server_backed_state_key_generator.cc(216)] No device identifiers available, no state keys generated

session_manager gets machine info from platform2/login_manager/init/scripts/ui-init-late which sends it to session_manager through dbus, taking /run/session_manager/machine-info as the source.
That in turn is collected in platform2/login_manager/init/scripts/ui-collect-machine-info which runs dump_vpd_log --full --stdout . I can confirm that this fails on my VM:

localhost ~ # dump_vpd_log --stdout --full
( does not print anything useful :) )

Since https://chromium-review.googlesource.com/c/chromiumos/platform2/+/847935 we do have a way to specify a serial number for a VM (using --uuid passed to QEMU / even without this, a SN should be auto-generated if there is none for VMs). However, this ends up only in /tmp/machine-info, which is read by chrome but not sent to session_manager for state keys generation.

I'll fix that - it seems that device policy initializes for me then, even though we don't have a stable device secret, because we have a serial number + (qemu emulated) disk serial number.
A WIP fix is in review - https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1293609 .

BTW, the weird behavior here comes from the fact that the serial number is available to chrome, but not to session_manager (because shell scripts read the serial number for the two, and one of them supports "fake serials" for VMs, while the other one doesn't - my CL above is trying to use the same thing for both).

When this happens, enrollment is possible (because chrome sees the serial number) and in my case it seems that the initial device policy fetch (which happens during enrollment) also works. However, device policy gets stale because session_manager couldn't generate state keys, which blocks parts of the policy stack from initializing properly (see Comment #4).

Not sure why others didn't have any issues with this setup however.

There is some pushback on the CL I don't understand yet, so until that is resolved, here's a workaround assuming you can ssh into the VM as root:

Assuming the VM has started listening for SSH on port 9222.
When copy-pasting commands that say "Inside SSH", please make sure you're inside the SSH session - you don't want to reboot your workstation :-)
 
(1) SSH into the VM:
ssh -p 9222 root@127.0.0.1
(password tends to be empty or test0000)

(2) Inside SSH, execute:
/usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --force

(3) Inside SSH, execute:
reboot && exit
This will exit your SSH session.

(4) Wait a few seconds and SSH into the VM again:
ssh -p 9222 root@127.0.0.1

(5) Inside SSH, execute:
mount -o remount,rw /

(6) Inside SSH, execute:
export FAKE_SERIAL=$(uuidgen)

(7) Inside SSH, execute:
echo "echo \"\\\"serial_number\\\"=\\\"${FAKE_SERIAL}\\\"\" >> \"\${UI_MACHINE_INFO_FILE}\"" >> /usr/share/cros/init/ui-collect-machine-info

(7) Inside SSH, execute:
sed -i "51s/.*/echo \\\"serial_number\\\"=\\\"${FAKE_SERIAL}\\\"/" /usr/share/cros/init/write-machine-info

(8) Inside SSH, execute:
echo $FAKE_SERIAL
This will print the serial number this VM instance will have

(8) Inside SSH, execute:
reboot && exit

Wait a few seconds for the VM to come back up. Now you should be able to enroll.

Please tell me if this works for you. Thanks!
Yup, those commands appear to have fixed the issue on the VM running 71 that I have. I'll keep an eye on the CL, but this works. Thanks!
Cc: isandrk@chromium.org
Just chiming in to say - THANKS!!!

Sign in to add a comment