Saving motopho raw output may help to figure out the root cause, if it is the issue in parsing code or motopho issue or even production issue.

A week or so ago, I modified the latency computing script to dump the raw data it receives from the motopho. Since then, I haven't seen any of the huge outliers (those seem to be much more rare), but a number of single-digit latencies. I also just modified the script again to save a visual representation of the screen brightness and angular velocity over time to help get a better idea of what's going on. A few initial findings and thoughts:

1. There doesn't seem to be any correlation between when the problem occurs and the URL that is being tested, so this is unlikely to be related to anything on Chrome's end such as low FPS or buffer stuffing issues.

2. The raw data from the Motopho appears to be good, at least in the case of getting single digit latencies. We'll get a better idea if this is the case once we get graphs of the data.

3. It occurred to me that the way we're doing movement might actually be the cause of this by being too repeatable. The latency computing script essentially finds the delay that results in the highest correlation between screen brightness and movement.

When a human is doing the movement, there's going to naturally be variations in the speed and duration of the rotation, meaning that you're only ever going to get a high correlation when you get the correct offset.

However, when we're doing movements with a servo, there's going to be essentially no difference in velocity between samples, which could make it easy for the script to associate some velocity measurement with an older brightness measurement, resulting in very low calculated latencies. If this is the case, then we'll have to see about making the movements vary in speed to prevent this from happening.

I got some comparison graphs between good runs and the runs with really low latencies, and I think I may have found the issue. See the attached files for a visual example.

There has been a long-standing bug in the way we do the motopho patch that causes the patch to sometimes, at random, flash brightly. I believe the root cause of this is because we need angular velocity, but VrCore does not provide that information, so instead we get two predicted poses that are very close to each other and calculate angular velocity from that.

This works well most of the time, but can cause sensor noise to show up as actual movement in some cases, which results in the screen flash.

If you look at the three bad graphs, you'll see that one of these flashes appeared very early in the test (shown by the random spike in yellow/red), whereas the good graphs have this occur much later or not at all.

This is important because the Motopho needs to spend some amount of time (~2-3 seconds) at the beginning collecting data while completely still and without changes in brightness in order to calculate the bias of its various sensors. My guess is that if the flash happens early, it screws up the Motopho's calibration, which results in these bad results.

+klausw who added the Motopho patch code - has VrCore added a way to get angular velocity directly from them/can you think of a way to get rid of these flashes?

Deleted: 2_1ms.pdf 62.1 KB

2_1ms.pdf 62.1 KB Download

Deleted: 7_02ms.pdf 61.9 KB

7_02ms.pdf 61.9 KB Download

Deleted: 8_02ms.pdf 61.2 KB

8_02ms.pdf 61.2 KB Download

Deleted: good_55_13ms.pdf 63.0 KB

good_55_13ms.pdf 63.0 KB Download

Deleted: good_65_14ms.pdf 63.1 KB

good_65_14ms.pdf 63.1 KB Download

Deleted: good_66_19ms.pdf 62.7 KB

good_66_19ms.pdf 62.7 KB Download

After working with Klaus, it looks like we fixed the unreasonably small latency results. The cause was the that the latency computation script was restricting the window of valid offsets (and thus latencies) to [0, max(250, first peak)] (we aren't 100% sure why this is the case - it looked like some sort of optimization to not do unnecessary computations, but in the normal case, the valid offsets would always be [0, 250] anyways).

What was happening was that the random flashes were being detected as brightness peaks, and if one occurred very early on during the test (e.g. 8ms in), the valid offsets would be restricted to very few values. Since we have the minimum acceptable correlation fairly low (0.6?), this small offset was usually enough to get an acceptable correlation and be accepted as a valid result by the script.

I'll leave this open for a bit longer to make sure the issue doesn't pop up again, and that the huge outliers don't show up, either.

Everything seems good now. I'll open a separate bug if something similar pops up again.