You might want to look at URLRequestHttpJob::RecordPerfHistograms. There are already byte counter UMAs there and at least some (not sure about the proxy stuff) of what you're asking for could be put there.

The histograms proposal sgtm. 

Ryan, has there been any progress on this?

Add another request. For video, we would like to split/flag the video bytes by whether Chrome has requested an compressed-video or not. This can be done by checking the CPAT request header, where "Chrome-Proxy-Accept-Transform: compressed-video" means Chrome requests the optimized video.

UMA for measuring bypassed bytes and bytes received from the Flywheel proxy on a per-response basis already exist with the histograms DataReductionProxy.BypassedBytes.*, including DataReductionProxy.BypassedBytes.NotBypassed.

I think the only new histograms we need here are SSL vs. Non-SSL for all responses even when Flywheel is disabled, and video-breakdown histograms.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/049f4387ba7724fa3c5ac1117a438fabb504f563

commit 049f4387ba7724fa3c5ac1117a438fabb504f563
Author: ajo <ajo@chromium.org>
Date: Thu Apr 13 17:59:52 2017

Add UMA to break down data usage by http/https and video/non_video

BUG= 689659 

Review-Url: https://codereview.chromium.org/2804113004
Cr-Commit-Position: refs/heads/master@{#464465}

[modify] https://crrev.com/049f4387ba7724fa3c5ac1117a438fabb504f563/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc
[modify] https://crrev.com/049f4387ba7724fa3c5ac1117a438fabb504f563/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate_unittest.cc
[modify] https://crrev.com/049f4387ba7724fa3c5ac1117a438fabb504f563/tools/metrics/histograms/histograms.xml

AJ, I believe this is fixed in M-59, correct? Please close the issue if so.

Dude, this is fantastic - thanks for doing this!

A couple of questions:

1) Is Net.HttpContentLength the sum of the individual Net.HttpContentLength.{Http,Https,Video} breakdowns?

2) Any reason why we decided not to split HTTP/HTTPS and Video/non-Video separately? (With this breakdown it's not possible to see whether video is HTTP or HTTPS.)

3) I believe you want to extend the range of the video histogram if possible, since most counts are in the 1MB - 2GB bucket (meaning we will potentially undercount actual video data usage):

https://uma.googleplex.com/p/chrome/histograms/?endDate=20170511&dayCount=1&histograms=Net.HttpContentLength.Http%2CNet.HttpContentLength.Https%2CNet.HttpContentLength.Video&fixupData=true&showMax=true&filters=platform%2Ceq%2CA%2Cisofficial%2Ceq%2CTrue&implicitFilters=isofficial

Friendly ping. It would be good to get this for M59 so wanted to make sure we knew where we were at with this.

Per off-thread discussion with mdw@, I'm not sure that the existing breakdowns are sufficient.

sclittle@ mentioned above that direct/bypassed/proxy bytes are recorded in the DataReductionProxy.BypassedBytes histogram, but it's unclear if that makes it possible to figure out what % of overall video bytes were proxied/bypassed/etc.

One possibility is to record the following:
Net.HttpContentLength.HTTP.Direct
Net.HttpContentLength.HTTP.ViaDRP
Net.HttpContentLength.HTTP.BypassedDRP
Net.HttpContentLength.HTTPS.Direct
Net.HttpContentLength.HTTPS.ViaDRP
Net.HttpContentLength.HTTPS.BypassedDRP

It's possible that that would be overkill/redundant, however.

I wonder if it's possible to add a "Video" suffix to the DataReductionProxy and compute the Direct bytes by subtracting DataReductionProxy.Bypased+proxied from Net.HttpContentLength

#12, question (2): Do you mean do something like Net.HttpContentLength.HTTP.Video? This seems reasonable to me, though since we're dealing with an exponential explosion, I think we should nail down which combinations we need.

#12, question (3): Sounds like we currently have a bug. AJ, can you fix? For consistency, we'll either need to rename the historgram or merge the fix into M-59. 

I looked at the current Net.HttpContentLength.Video in M59 (current beta), I think the bucketing is less ideal. Most counts accumulates in the last a few buckets.

https://uma.googleplex.com/histograms?endDate=latest&dayCount=1&histograms=Net.HttpContentLength.Video&fixupData=true&otherVersions=true&showMax=true&filters=platform%2Ceq%2CA%2Cisofficial%2Ceq%2CTrue&implicitFilters=isofficial

It is using the same UMA_HISTOGRAM_COUNTS_1M bucketing as other responses. I think we need to the extend the max and maybe need more bucketing.

How about something like 

UMA_HISTOGRAM_CUSTOM_COUNTS(
  "Net.HttpContentLength.Video"                            
  received_content_length,
  1,
  16<<20,  // 16MB max
  100);  // 100 buckets

Also, is there plan to do similar break down for Net.HttpOriginalContentLength as suggested in #1?  It will help to compute the data saving for each case.

Re: #16, there's UMA_HISTOGRAM_COUNTS_10M, which is similar. Why not add a few buckets near the top of the range and add more buckets, e.g.,

UMA_HISTOGRAM_CUSTOM_COUNTS(...,1, 128<<20, 200)

Ping. I think this is pretty important for this milestone. It was labeled as M-59. Let's get it done for M-60, please.

I have an in-progress CL with the following suffixes to Net.HTTPContentLength:


  <suffix name="HTTP.Direct"/>
  <suffix name="HTTP.ViaDRP"/>
  <suffix name="HTTP.BypassedDRP"/>
  <suffix name="HTTP.Other"/>
  <suffix name="HTTPS.Direct"/>
  <suffix name="HTTPS.ViaDRP"/>
  <suffix name="HTTPS.BypassedDRP"/>
  <suffix name="HTTPS.Other"/>
  <suffix name="HTTP.Direct.Video"/>
  <suffix name="HTTP.ViaDRP.Video"/>
  <suffix name="HTTP.BypassedDRP.Video"/>
  <suffix name="HTTP.Other.Video"/>
  <suffix name="HTTPS.Direct.Video"/>
  <suffix name="HTTPS.ViaDRP.Video"/>
  <suffix name="HTTPS.BypassedDRP.Video"/>
  <suffix name="HTTPS.Other.Video"/>

mdw@, bolian@, are these combinations sufficient?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/301b00f0c3b91119545da8f1762dcba0f665697e

commit 301b00f0c3b91119545da8f1762dcba0f665697e
Author: ajo <ajo@chromium.org>
Date: Fri May 26 05:25:11 2017

increase bucket size and break down video by http/https
Effected existing histograms:
Net.HttpContentLength.Http
Net.HttpContentLength.Https
Net.HttpContentLength.Video
Net.HttpContentLength.Http.Video
Net.HttpContentLength.Https.Video

New Histograms:

Net.HttpContentLength.Http.Direct
Net.HttpContentLength.Http.ViaDRP
Net.HttpContentLength.Http.BypassedDRP
Net.HttpContentLength.Http.Other
Net.HttpContentLength.Https.Direct
Net.HttpContentLength.Https.ViaDRP
Net.HttpContentLength.Https.BypassedDRP
Net.HttpContentLength.Https.Other

Net.HttpContentLength.Http.Direct.Video
Net.HttpContentLength.Http.ViaDRP.Video
Net.HttpContentLength.Http.BypassedDRP.Video
Net.HttpContentLength.Http.Other.Video
Net.HttpContentLength.Https.Direct.Video
Net.HttpContentLength.Https.ViaDRP.Video
Net.HttpContentLength.Https.BypassedDRP.Video
Net.HttpContentLength.Https.Other.Video

BUG= 689659 

Review-Url: https://codereview.chromium.org/2888653003
Cr-Commit-Position: refs/heads/master@{#474922}

[modify] https://crrev.com/301b00f0c3b91119545da8f1762dcba0f665697e/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc
[modify] https://crrev.com/301b00f0c3b91119545da8f1762dcba0f665697e/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate_unittest.cc
[modify] https://crrev.com/301b00f0c3b91119545da8f1762dcba0f665697e/tools/metrics/histograms/histograms.xml

It appears this missed branch. Requesting merge. AJ, please coordinate with a committer to merge this.

Your change meets the bar and is auto-approved for M60. Please go ahead and merge the CL to branch 3112 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), josafat@(ChromeOS), bustamante@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I'm adding the breakdowns for OCL as well am planning on removing the Net.HttpOriginalContentLengthWithValidOCL histograms, since they should be captured by:

Net.HttpOriginalContentLength.Http.ViaDRP

NB: when fixing crbug.com/723853, Net.HttpOriginalContentLength might need to updated

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c9a6e48daa1618e9c65b17b872ec8ad1926400ab

commit c9a6e48daa1618e9c65b17b872ec8ad1926400ab
Author: AJ Ortega <ajo@chromium.org>
Date: Wed May 31 21:48:11 2017

Add new OCL histograms

Bug:  689659 
Change-Id: Ie18371febfdff041d6dea0df73e145328585c2ed
Reviewed-on: https://chromium-review.googlesource.com/517385
Commit-Queue: AJ Ortega <ajo@chromium.org>
Reviewed-by: Scott Little <sclittle@chromium.org>
Reviewed-by: Ilya Sherman <isherman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#476043}
[modify] https://crrev.com/c9a6e48daa1618e9c65b17b872ec8ad1926400ab/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc
[modify] https://crrev.com/c9a6e48daa1618e9c65b17b872ec8ad1926400ab/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate_unittest.cc
[modify] https://crrev.com/c9a6e48daa1618e9c65b17b872ec8ad1926400ab/tools/metrics/histograms/histograms.xml

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d63765809ea46b7f59a1f56a9719634d5fbf4864

commit d63765809ea46b7f59a1f56a9719634d5fbf4864
Author: Tarun Bansal <tbansal@google.com>
Date: Thu Jun 01 06:51:44 2017

increase bucket size and break down video by http/https Effected existing histograms: Net.HttpContentLength.Http Net.HttpContentLength.Https Net.HttpContentLength.Video Net.HttpContentLength.Http.Video Net.HttpContentLength.Https.Video

New Histograms:

Net.HttpContentLength.Http.Direct
Net.HttpContentLength.Http.ViaDRP
Net.HttpContentLength.Http.BypassedDRP
Net.HttpContentLength.Http.Other
Net.HttpContentLength.Https.Direct
Net.HttpContentLength.Https.ViaDRP
Net.HttpContentLength.Https.BypassedDRP
Net.HttpContentLength.Https.Other

Net.HttpContentLength.Http.Direct.Video
Net.HttpContentLength.Http.ViaDRP.Video
Net.HttpContentLength.Http.BypassedDRP.Video
Net.HttpContentLength.Http.Other.Video
Net.HttpContentLength.Https.Direct.Video
Net.HttpContentLength.Https.ViaDRP.Video
Net.HttpContentLength.Https.BypassedDRP.Video
Net.HttpContentLength.Https.Other.Video

BUG= 689659 

Review-Url: https://codereview.chromium.org/2888653003
Cr-Original-Commit-Position: refs/heads/master@{#474922}
Review-Url: https://codereview.chromium.org/2920703002 .
Cr-Commit-Position: refs/branch-heads/3112@{#82}
Cr-Branched-From: b6460e24cf59f429d69de255538d0fc7a425ccf9-refs/heads/master@{#474897}

[modify] https://crrev.com/d63765809ea46b7f59a1f56a9719634d5fbf4864/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc
[modify] https://crrev.com/d63765809ea46b7f59a1f56a9719634d5fbf4864/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate_unittest.cc
[modify] https://crrev.com/d63765809ea46b7f59a1f56a9719634d5fbf4864/tools/metrics/histograms/histograms.xml

Requesting a merge for c9a6e48daa1618e9c65b17b872ec8ad1926400ab, also necessary to fix this bug.

Your change meets the bar and is auto-approved for M60. Please go ahead and merge the CL to branch 3112 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), josafat@(ChromeOS), bustamante@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ab401110aa6fec87e1d5acde20dddddc2ee9cac4

commit ab401110aa6fec87e1d5acde20dddddc2ee9cac4
Author: Tarun Bansal <tbansal@google.com>
Date: Tue Jun 06 20:15:50 2017

Add new OCL histograms

Bug:  689659 
Change-Id: Ie18371febfdff041d6dea0df73e145328585c2ed
Reviewed-on: https://chromium-review.googlesource.com/517385
Commit-Queue: AJ Ortega <ajo@chromium.org>
Reviewed-by: Scott Little <sclittle@chromium.org>
Reviewed-by: Ilya Sherman <isherman@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#476043}
Review-Url: https://codereview.chromium.org/2926683002 .
Cr-Commit-Position: refs/branch-heads/3112@{#203}
Cr-Branched-From: b6460e24cf59f429d69de255538d0fc7a425ccf9-refs/heads/master@{#474897}

[modify] https://crrev.com/ab401110aa6fec87e1d5acde20dddddc2ee9cac4/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc
[modify] https://crrev.com/ab401110aa6fec87e1d5acde20dddddc2ee9cac4/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate_unittest.cc
[modify] https://crrev.com/ab401110aa6fec87e1d5acde20dddddc2ee9cac4/tools/metrics/histograms/histograms.xml

Is this fixed?

Is this fixed?

These new histograms are not showing up on the UMA dashboard. Seems to be a problem here. AJ - can you please take a look?

This is related to Issue 733697. Can you try selecting the checkbox "Show obsolete histogram names"?

There seems to be a bug in the implementation of the new histograms, in that the ".Direct" variant is only populated when the request_type is HTTPS:

https://codereview.chromium.org/2888653003/diff/410001/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc

Note that many users will never have the Data Reduction Proxy enabled, so we need to record the data use histograms for those users as well. I see that the implementation here is within DataReductionProxyNetworkDelegate, so perhaps this code path will only be invoked for DRP users?

The point here is that ".Direct" should apply to any case where a request *was not made via the DRP and should not have gone through the DRP*. This includes HTTPS, users with the DRP disabled, or anything that for some other reason is ineligible for DRP (e.g., bad proxy config, etc.)

It appears to me that there are missing OCL headers, similar to CL headers, we should have all of these but for OCL headers:

Net.HttpContentLength.Http.Direct.Video
Net.HttpContentLength.Http.ViaDRP.Video
Net.HttpContentLength.Http.BypassedDRP.Video
Net.HttpContentLength.Http.Other.Video
Net.HttpContentLength.Https.Direct.Video
Net.HttpContentLength.Https.ViaDRP.Video
Net.HttpContentLength.Https.BypassedDRP.Video
Net.HttpContentLength.Https.Other.Video

And Chrome does report to these missing headers:
https://cs.chromium.org/chromium/src/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc?gsn=RecordNewContentLengthHistograms&l=155

Look at the code, I think we are missing below section for OCL:
<histogram_suffixes name="NetHttpContentLengthType3" separator=".">
  <suffix name="Video" label="Bytes of video traffic"/>
  <affected-histogram name="Net.HttpContentLength.Http.BypassedDRP"/>
  <affected-histogram name="Net.HttpContentLength.Http.Direct"/>
  <affected-histogram name="Net.HttpContentLength.Http.Other"/>
  <affected-histogram name="Net.HttpContentLength.Http.ViaDRP"/>
  <affected-histogram name="Net.HttpContentLength.Https.BypassedDRP"/>
  <affected-histogram name="Net.HttpContentLength.Https.Direct"/>
  <affected-histogram name="Net.HttpContentLength.Https.Other"/>
  <affected-histogram name="Net.HttpContentLength.Https.ViaDRP"/>
</histogram_suffixes>

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/17685b60e80c66b7fb039bf376f7de3444744f84

commit 17685b60e80c66b7fb039bf376f7de3444744f84
Author: AJ Ortega <ajo@chromium.org>
Date: Wed Jul 12 20:57:18 2017

fixup data reduction proxy reporting.

When the data reduction proxy was disabled, the metrics code 
considered it to be a SHORT_BYPASS, making it impossible to report
on the content length for non-flywheel http traffic.

Additionally, add missing video metrics to histograms.xml for
the UMA UI.

Bug: 726411
Bug:  689659 
Change-Id: I4db55f6492291bbd0159f8126b6907e4a367b34c
Reviewed-on: https://chromium-review.googlesource.com/546863
Commit-Queue: AJ Ortega <ajo@chromium.org>
Reviewed-by: Ilya Sherman <isherman@chromium.org>
Reviewed-by: Ryan Sturm <ryansturm@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486081}
[modify] https://crrev.com/17685b60e80c66b7fb039bf376f7de3444744f84/components/data_reduction_proxy/core/browser/data_reduction_proxy_compression_stats.cc
[modify] https://crrev.com/17685b60e80c66b7fb039bf376f7de3444744f84/components/data_reduction_proxy/core/browser/data_reduction_proxy_metrics.cc
[modify] https://crrev.com/17685b60e80c66b7fb039bf376f7de3444744f84/components/data_reduction_proxy/core/browser/data_reduction_proxy_metrics.h
[modify] https://crrev.com/17685b60e80c66b7fb039bf376f7de3444744f84/components/data_reduction_proxy/core/browser/data_reduction_proxy_metrics_unittest.cc
[modify] https://crrev.com/17685b60e80c66b7fb039bf376f7de3444744f84/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate.cc
[modify] https://crrev.com/17685b60e80c66b7fb039bf376f7de3444744f84/components/data_reduction_proxy/core/browser/data_reduction_proxy_network_delegate_unittest.cc
[modify] https://crrev.com/17685b60e80c66b7fb039bf376f7de3444744f84/tools/metrics/histograms/histograms.xml

Ryan, would you mind taking over this issue? I think we just need to verify that it is fixed.

Ryan, what's the status?

I believe this is fixed:

We record all combinations of {HTTP|HTTPS}, {viadrp|direct|bypassed|other}, {OCL|CL}, {VIDEO|ALLTRAFFIC}.

Histograms without data are:
Net.HttpContentLengthV2.Http (not a fully expanded type, so this make sense, data is captured under the bypass, direct histograms. This is common with the way histograms.xml works)
Net.HttpContentLengthV2.Https (not fully expanded)
Net.HttpContentLengthV2.Https.BypassedDRP (we don't bypass HTTPS)
Net.HttpContentLengthV2.Https.BypassedDRP.Video (we don't bypass HTTPS)
Net.HttpContentLengthV2.Http.Other.Video (Other should be empty)
Net.HttpContentLengthV2.Https.Other (Other should be empty)
Net.HttpContentLengthV2.Https.Other.Video (Other should be empty)
Net.HttpContentLengthV2.Https.ViaDRP (no HTTPS DRP traffic)
Net.HttpContentLengthV2.Https.ViaDRP.Video (no HTTPS DRP traffic)
Net.HttpContentLengthV2.Video (not a fully expanded type)
Net.HttpOriginalContentLengthV2.Http (not a fully expanded type)
Net.HttpOriginalContentLengthV2.Http.Other.Video (shouldn't have data as it's other)
Net.HttpOriginalContentLengthV2.Https (not a fully expanded type)
Net.HttpOriginalContentLengthV2.Https.BypassedDRP (we don't bypass DRP on HTTPS)
Net.HttpOriginalContentLengthV2.Https.BypassedDRP.Video (we don't bypass DRP on HTTPS)
Net.HttpOriginalContentLengthV2.Https.Other (other should be empty)
Net.HttpOriginalContentLengthV2.Https.Other.Video (other should be empty)
Net.HttpOriginalContentLengthV2.Https.ViaDRP (we don't use DRP on HTTPS)
Net.HttpOriginalContentLengthV2.Https.ViaDRP.Video (we don't use data on HTTPS)
Net.HttpOriginalContentLengthV2.Video (not a fully expanded type)


A concern I have is that Net.HttpOriginalContentLengthV2.Http.Other has some limited data (it should have none) on 62 stable, which is not great. I'm not totally sure what that means, but it seems like it should be it's own bug.

Thanks for looking into this. So you're saying ".Other" shouldn't have data because the requests is either ".ViaDRP" or ".BypassedDRP." Where would a request be logged that doesn't use the DRP because another proxy is in use? 

I suspect "direct". I'll update later.

I believe that for HTTP traffic, we treat another proxy used as "Other" which explains the Net.HttpOriginalContentLengthV2.Http.Other data.

For HTTPS, we treat all traffic as direct, however, this could theoretically be split to other for proxy use.

I'm satisfied with that result, and we can see for UMA users, we have very little "other" proxy usage (none for video), which seems to be feasible.

Fixing, as I am satisfied with the results of this conversion.