Ryan, which of our metrics could we use to track the number of users that might be stuck in this state? Can you add a link here?

See metrics for host status:
- Histogram: [1]
- Timeline: [2]

Let's consider only users who have attempted to set a multidevice host. That includes the buckets:
- "Host set locally but waiting for backend confirmation": 626 unique users (uu)
- "Host set but not yet verified": 878 uu
- "Host verified": 5699 uu.

Failures: 626uu + 878uu = 1504uu.
Total attempts: 626uu + 878uu + 5699uu = 7203uu.

Percentage of users potentially stuck in "waiting for verification": 1504uu / 7203uu = .2088.

So up to 21% of users may be stuck in this state.


1) https://uma.googleplex.com/p/chrome/histograms?endDate=20181211&dayCount=7&histograms=MultiDevice.Setup.HostStatus&fixupData=true&uniqueUsers=true&showMax=true&filters=platform%2Ceq%2CC%2Cchannel%2Cone_of%2C2%2Cisofficial%2Ceq%2CTrue&implicitFilters=isofficial
2) https://uma.googleplex.com/p/chrome/timeline_v2/?sid=0ab2ddb5c43b99a9f2279a2a5b828d6e

Ryan, Kyle and I took some time to dig into this off-thread. In reality, the number of users stuck in this state is probably much lower than 21%. Here's why:

1) This histogram is emmitted every time that the device is unlocked or the state changes. This means that any user that goes through setup will emit *at least* the "Host set but not yet verified" counter before the "Host verified" counter.
2) As a result, the bigger aggregation slice you look at, the higher the percentage of "verified" users will be who have gone through setup during that aggregation period. This means more users in both "waiting" buckets.
3) If we narrow the period to a one day aggregation [a], we see this 21% drop to (172+104)/(1,853+172+104) = ~13%.
4) Even at 13%, this is a cap on the worst possible scenario. In reality, any user than went through setup today should be deducted from the error buckets. However, it seems that we don't have a trivial way to get a count of "number of users that went through setup today". We've got an android-side log [b] for something like that. However, thr problem right now is that the ChromeOS MultiDevice.Setup.HostStatus metric is only in M72 (dev channel), while the Android one also includes M71 (beta) users. Once 72 hits stable, we'll have a good way to measure this.
5) One way to get a good signal on "stuck" states might be to emmit something on the ChromeOS side at increasing time intervals while the device is in the pending verification state. For example, if 1 minute after setup, we're still waiting, emit a counter. If 5 minutes later, we're still waiting, emit a different counter. etc., etc., up to about an hour or so delay. This would give us a pretty good signal.

In general, this doesn't seem to be as big an issue as it might seem. We should keep an eye on it because users have indeed been reporting it ocassionally, but I don't think the world is on fire quite yet :-). 


[a] https://uma.googleplex.com/p/chrome/histograms?endDate=20181211&dayCount=1&histograms=MultiDevice.Setup.HostStatus&fixupData=true&uniqueUsers=true&showMax=true&filters=platform%2Ceq%2CC%2Cchannel%2Cone_of%2C2%2Cisofficial%2Ceq%2CTrue&implicitFilters=isofficial
[b] https://uma.googleplex.com/p/proximity_auth/histograms?endDate=latest&dayCount=1&histograms=BetterTogetherSetupResult&fixupData=true&uniqueUsers=true&showMax=true&filters=isofficial%2Ceq%2CTrue&implicitFilters=isofficial