Probably okay, may increase noise in a/b tests though since we no longer have a clear off surface-layer-off signal though...

The added noise is minimal as Picture-in-Picture represents pretty much no watch time compared to global watch time. The impact on our users using non-Stable builds is high as they may end up moved in and out of the controls group that as a side effect will shut down an entire feature.

That's fair, but not all metrics are proportional to watch time. I think we should wait until we have a decent sample from YouTube for at least beta w/ my latest changes. Per your own e-mail we've previously seen different results once we've reached that stage.

After beta data, I'd actually instead be more supportive of changing the default to "always" based on early data:
https://uma.googleplex.com/p/chrome/variations/?sid=6d3de79d862d0b308afe6e6c428d1232 (assuming it holds up in beta) -- otherwise we can land the changes as you propose and merge to 73.

We have been running OnDemand vs Disabled for a while and saw no difference. I'm not sure I follow why we would see a difference between Disable and OnDemand when compared with Enabled now. What kind of metrics do you think may be impacted? All the metrics we've been looking at so far will be somewhat dependent on watch time, all things behind equal.

My comment about metrics was when comparing Canary/Dev to Beta or Stable. Canary/Dev has been a bit all over the place metrics wise for this feature.

Okay, it's good you have that data. Can you link to it? I wasn't able to find it. From looking at the configs the PIP groups have weight=0, which would seem to prevent them from being used for the comparison we want here?

The PIP group used to be the experiment group when we launched Picture-in-Picture. We moved it to 0 recently.

Finch data are available here: https://uma.googleplex.com/p/chrome/variations/?sid=2cf617c4bee6f66f298598986c45bdac (Google internal)

Thanks for the link. UnderflowDuration keeps showing up as improved in the kOnDemand case, which is great, but unexpected I think? So I still prefer that we keep this as is until beta and then either land your change or switch it to always.

Since that's only ~2 weeks away, do you have any user reports complaining about this? Or otherwise some other need to get this in right away?

We are actually getting frequent feedback from developers/users that the feature doesn't work. fbeaufort@ CC'd regularly ping me about this because the steps to "enable Picture-in-Picture" are hard to explain. Even Chrome OS engineers are sometimes confused by the fact that the feature is disabled on our official channels. This is the only reason why I want to do this. I believe the benefit outweigh the potential noise in the data.