Should we use enums instead? I've always found our existing one pretty useless since we get poor bucketing with numeric values.

Do you mean a list like CGA, WXGA, UHD etc?

https://en.wikipedia.org/wiki/Display_resolution#/media/File:Vector_Video_Standards8.svg

This should be doable. And if a resolution is not a "standard" one, we can put them into the closest bucket (with some limit on the error). Then for some really uncommon video resolutions, we can put them in a "uncommon" bucket.

That said, I still feel a numeric UMA (e.g. video height) would be useful, e.g. to answer questions like "how many videos are over 720p".

Eh, or something like k720p etc :) You can still answer that question using the enum approach (roughly), but you can't with a numeric approach unless you are very careful with your bucket sizes. You'll end up being able to say something like, 10% of videos are > 687p or something since bucket ranges are programatically generated.

If we are willing to consume this data via dremel as opposed to the UMA UI, we should be able to answer specific questions like what % of videos exceed a given vertical resolution (or overall pixel count if we are going with that instead).

Since it hasn't been explicitly stated above, let's also log watch time for each resolution as opposed to just instantiations. IIUC, logging watch time would only be possible using the enum route, correct?

They are different problems, but it would make more sense if he had a common bucketing mechanism for the two.

1. height vs resolution

It seems height is simpler to understand and use. In our implementation we can check whether the video is vertical and we can use the width of a vertical video as the "height" in that case.

2. number vs enum (+isherman for comments)

I still like numeric UMA instead of enums given the height is really just a number. Also, with numbers we can do a bunch of things like sum or average. The fact that numbers are put in linear or logrithmtic buckets is an implementation detail and we have some ways to overcome the "687p" problem:

a. We can choose more buckets so the rounding error would be small enough, e.g.

   UMA_HISTOGRAM_CUSTOM_COUNTS("Media.VideoHeight.Initial.MSE", 100, 10000, 1000);

b. We can use UMA_HISTOGRAM_SPARSE_SLOWLY given the fact that:
  - The resolution UMA is only recorded once or twice per playback.
  - Most videos follow the common video heights, e.g. 480, 720, 1080. However, this will not be true for "average" video height due to adaptive streaming.

3. I am not sure whether we want to have WatchTime for each resolution. The number of combinations is just too big. I really wish we could record a "vector" of numbers for a playback instance so we can do all kinds of analysis.

But I agree we should have the same bucketing as WatchTime UMAs. Here are what I am thinking about right now:

Media.VideoHeight.Initial.EME
Media.VideoHeight.Initial.SRC
Media.VideoHeight.Initial.MSE
Media.VideoHeight.Initial.ALL

Media.VideoHeight.Average.EME
Media.VideoHeight.Average.SRC
Media.VideoHeight.Average.MSE
Media.VideoHeight.Average.ALL

Please do not record a histogram with a thousand buckets -- this is very expensive in terms of memory usage.

UMA_HISTOGRAM_SPARSE_SLOWLY is also not intended for histograms that can have thousands of buckets, and I'd imagine that across the Web, we would indeed see a wide variety.  That is, sparse histograms would use less memory client-side (good!), but would still be expensive server-side if they cover a vast range.  That said, if you're willing to forgo using the UMA dashboards, and instead query Dremel directly, we can simply exclude the histograms from the server-side processing.  In that case, sparse histograms would not be too expensive.  (We prefer not to do this very often, though, because it makes the histograms less discoverable, and also blacklists like this are an extra maintenance burden.)

isherman: Thanks for the comments. We don't want to blacklist this UMA as well for the same reasons.

1000 buckets is just an example. Given we only care about the range from 100 to 5000, 100 buckets should be good enough. I'll play with the histograms to see how common video heights are covered.

isherman: I did a bit research and a histogram with 100 bucket should be sufficient for our use case.

Or, I found that we can use UMA_HISTOGRAM_CUSTOM_ENUMERATION, with a given custom range that's less than 100 values. An example list of the custom range can be as follows (with all possible heights in https://en.wikipedia.org/wiki/List_of_common_resolutions). Note that this is an exhaustive list. I am sure we can easily trim it down to <50. If so, would UMA_HISTOGRAM_CUSTOM_ENUMERATION be a good fit here?

11
16
30
32
36
40
48
64
72
80
90
96
100
120
128
144
152
160
168
176
192
200
208
212
224
234
240
250
256
270
272
288
300
320
340
342
348
350
352
360
364
384
390
400
480
486
500
512
540
544
576
600
624
640
720
750
768
800
832
854
858
864
900
960
1000
1024
1050
1080
1152
1200
1280
1332
1344
1392
1400
1440
1536
1556
1600
1700
1714
1728
1800
1824
1920
2000
2048
2100
2160
2304
2400
2664
2880
3000
3072
3112
3160
3200
4096
4320
4608
4800
8192
9334

Yes, UMA_HISTOGRAM_CUSTOM_ENUMERATION sounds like a good fit =)

Great!

hbengali / dalecurtis: Any suggestions/comments on the list posted in #11? I think we don't care about not-so-common video resolutions, so we should drop some not-so-common-and-too-close-to-the-neighbor values.

Dale:

Here's an updated list, with 100 numbers. It's mostly from https://en.wikipedia.org/wiki/List_of_common_resolutions with some tweaks (e.g. I added 6000 since the gap between 4800 to 8192 is too big).

Now I wonder whether we need these many, especially the smaller ones. But part of me felt this list should be exhaustive; who knows what could be useful or not.

WDYT?


16
30
32
36
40
48
64
72
80
90
96
100
120
128
144
152
160
168
176
192
200
208
212
224
234
240
250
256
270
272
288
300
320
340
352
360
364
384
390
400
480
500
512
540
576
600
624
640
720
750
768
800
832
854
858
864
900
960
1000
1024
1050
1080
1152
1200
1280
1332
1344
1392
1400
1440
1536
1556
1600
1700
1714
1728
1800
1824
1920
2000
2048
2100
2160
2304
2400
2664
2880
3000
3072
3112
3160
3200
3600
4096
4320
4608
4800
6000
8192
9334

The nice thing about an enum based version is we can add more later if we find the current enums don't match reality. So I think starting with a smaller subset with a overflow bucket would let us craft this with the minimum required set of values.

I think things like 16p, 140p, 240p, 360p, 480p, 720p, 900p, 1080p, 1440p, 2160p would be a good starting set.

With matching into those buckets within +/- 16p

Enum based version has it's own value, but it has a few issues:
- Not suitable for "average resolution" over the entire playback session.
- Not easy to get average resolution for all users etc.
- Uncommon resolution will not be reported, making it hard to cross reference this data with other playback UMAs.

How about we add two UMAs
1. Media.CommonVideoHeights.*  // enum based
2. Media.VideoHeights.*  // value based

Then we can simplify Media.VideoHeights* to just use UMA_HISTOGRAM_CUSTOM_COUNTS with 100 buckets.

(2) will also be helpful for use to improve (1).

What is useful about average resolution? We already get something similar with the existing metric and we've never used it :)

That said I think your suggestion is good. We can have a kCrazyResolution bucket on the enum version which we fill for non-matches and then record _only_ those values to a counts based histogram to whittle down the enum version. We do this with some other metrics like FallbackHardwareParameters.

I think we'd want to shoot for having < 5% of entries in the kCrazyResolution bucket.

Average resolution would be useful to measure playback experience. Say, on LTE, the average resolution should be smaller than Wifi. Which existing metric are you referring to? I only found Media.VideoVisibleWidth which is only reported for SRC :(

I played with histograms a bit more. It seems UMA_HISTOGRAM_CUSTOM_COUNTS(name, sample, 64, 10000, 120) will give us pretty much what we need (for now). For example [476, 497) for 480p, [701, 732) for 720p, and [1075, 1122) for 1080p.

0, 64, 67, 70, 73, 76, 79, 82, 86, 90, 94, 98, 102, 106, 111, 116, 121, 126, 132, 138, 144, 150, 157, 164, 171, 178, 186, 194, 202, 211, 220, 230, 240, 251, 262, 273, 285, 297, 310, 324, 338, 353, 368, 384, 401, 419, 437, 456, 476, 497, 519, 542, 566, 591, 617, 644, 672, 701, 732, 764, 797, 832, 868, 906, 946, 987, 1030, 1075, 1122, 1171, 1222, 1276, 1332, 1390, 1451, 1515, 1581, 1650, 1722, 1797, 1876, 1958, 2044, 2134, 2228, 2326, 2428, 2534, 2645, 2761, 2882, 3008, 3140, 3278, 3422, 3572, 3729, 3892, 4063, 4241, 4427, 4621, 4823, 5034, 5255, 5485, 5725, 5976, 6238, 6511, 6796, 7094, 7405, 7730, 8069, 8423, 8792, 9178, 9580, 10000, 2147483647

I mean the VisibleWidth one, the same changes in height that you describe for LTE would show up in the width. Also most playback usage on Android is SRC, so that's the one we'd care about for your query.

Those values look good for a the counts based metric though. I suspect there will be some smearing around 140p, 240p, and 360p though.

My point for SRC is that the width is unlikely to change, so "average width" doesn't make too much sense. But for MSE, it would be a better measurement than the "initial width".

Why wouldn't it change? The width varies with adaptive content. I.e. 1280x720p, 854x480p, etc.

Oh I see, you just mean any SRC metric is unlikely to see resolution changes, I agree with that. MSE indeed would be better served by average metrics.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/60802656141458897882b4283908e75bb444b420

commit 60802656141458897882b4283908e75bb444b420
Author: xhwang <xhwang@chromium.org>
Date: Wed Apr 19 07:29:58 2017

media: Report initial video height to UMA

Also fix the issue where the initial video natural size is
not reported to MediaLog.

BUG= 709354 
TEST=Manually tested

Review-Url: https://codereview.chromium.org/2814043005
Cr-Commit-Position: refs/heads/master@{#465518}

[modify] https://crrev.com/60802656141458897882b4283908e75bb444b420/media/blink/webmediaplayer_impl.cc
[modify] https://crrev.com/60802656141458897882b4283908e75bb444b420/media/blink/webmediaplayer_impl.h
[modify] https://crrev.com/60802656141458897882b4283908e75bb444b420/tools/metrics/histograms/histograms.xml

The CL in #24 has been verified in trunk. Request to merge #24 to M59.

Then we'll keep discussing how to measure "average" and/or "all" resolutions.

Your change meets the bar and is auto-approved for M59. Please go ahead and merge the CL to branch 3071 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), gkihumba@(ChromeOS), Abdul Syed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8a036edb1efbc4d997b3e3fb62983e11d9404c29

commit 8a036edb1efbc4d997b3e3fb62983e11d9404c29
Author: Xiaohan Wang <xhwang@chromium.org>
Date: Thu Apr 20 18:56:57 2017

(merge m59) media: Report initial video height to UMA

Also fix the issue where the initial video natural size is
not reported to MediaLog.

BUG= 709354 
TEST=Manually tested

Review-Url: https://codereview.chromium.org/2814043005
Cr-Commit-Position: refs/heads/master@{#465518}
(cherry picked from commit 60802656141458897882b4283908e75bb444b420)

Review-Url: https://codereview.chromium.org/2834713002 .
Cr-Commit-Position: refs/branch-heads/3071@{#95}
Cr-Branched-From: a106f0abbf69dad349d4aaf4bcc4f5d376dd2377-refs/heads/master@{#464641}

[modify] https://crrev.com/8a036edb1efbc4d997b3e3fb62983e11d9404c29/media/blink/webmediaplayer_impl.cc
[modify] https://crrev.com/8a036edb1efbc4d997b3e3fb62983e11d9404c29/media/blink/webmediaplayer_impl.h
[modify] https://crrev.com/8a036edb1efbc4d997b3e3fb62983e11d9404c29/tools/metrics/histograms/histograms.xml

The "initial" video height CL has been merged into m59.

For next steps, here's a summary of what's discussed here, in the CL and in person about what we might want to do next:

1. Based on the current UMA, we can add a new enum based UMA, capturing common video heights (e.g. k240p, k360p, k480p, k720p, k1080p, with some error tolerance, e.g. +/- 16 pixels), with an "uncommon" bucket for everything else. We can refine this UMA over time to keep the "uncommon" bucket relatively low.

2. The "average" video height should be defined as "the average video resolution" across all playback instances". Ideally, we want to get something like 
  sum(resolution * resolution_playback_time) / sum(playback_time). 
One way to achieve this is by sampling, e.g. we report resolution every 1 or 10 minute. We can also consider some randomness in sampling time to combat fixed ABR patterns (e.g. switching on minute boundaries).

3. We could also report "all" video heights we saw, i.e. report the video height every time we see a new one. But it's not super clear how we are gonna use this metric.

4. (from hbengali) It'll be nice to have WatchTime-per-resolution metrics so that we can answer questions like "what's the watch-time on 480p vs 1080p".

Thanks!  Regarding the first point, what's the motivation for adding an error tolerance?  Is it (known to be) common for videos to have near-common resolutions?

I won't say it's common, but we have to look at the data to understand more. That's why I feel we need to "refine" it overtime. We could start with exact resolution first and see how it goes, and relax it if needed. But really, from our perspective, we don't really care about whether a resolution is an exact or approximate match; users don't care either. So I don't see any harm by adding some error tolerance.

Okay, as long as there aren't optimizations that care about exact resolutions (or other similar reasons to care about precise resolutions), then recording approximate matches makes sense.  Thanks!

The remaining work is not targeting M59. Move the milestone to M60.

dalecurtis: Do we report UKM for every resolution change? Then we can compute "watchtime weighted average video height" easily, and we don't need an additional UMA for that.

I believe the answer to the above is yes, and I have already added the duration weighted average video height to our dashboard. I think we can close this as won't fix.