New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 710474 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Jun 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug



Sign in to add a comment

[autofill] UMA Autofill.Quality.* does not capture false positives and true negatives.

Project Member Reported by rogerm@chromium.org, Apr 11 2017

Issue description

The Autofill.Quality.* UMA logging does not capture enough cases to allow correct analysis of precision and recall as described at https://en.wikipedia.org/wiki/Precision_and_recall

The cases missing are:

  - the field was not autofillable but it was predicted as such
  - the field was not autofillable and no type was predicted

 

Comment 1 by f...@chromium.org, Apr 12 2017

Cc: f...@chromium.org

Comment 2 by f...@chromium.org, Apr 12 2017

One way to look at this is that the histograms are currently recorded by the actual type. Meaning, if the "actual" type was NAME then it records NAME_[MATCH, MISMATCH, UNKNOWN]. You also need the reverse: if the "predicted" type was NAME then record NAME_[MATCH, MISMATCH, UNKNOWN]. If you had that then you'd have the full logic table for all possible combinations.
Project Member

Comment 3 by bugdroid1@chromium.org, Apr 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8f71139f8c5982d83fcdcc840a5275698b71438c

commit 8f71139f8c5982d83fcdcc840a5275698b71438c
Author: rogerm <rogerm@chromium.org>
Date: Wed Apr 12 20:09:37 2017

[autofill] Expand Autofill.Quality.* to log false positives/true negatives.

This CL adds Autofill.Quality.* logging for the following cases:

 * the field was not confirmed as autofillable (either the user
   left the field empty or filled the field with data that was
   not recognized); and,

   - A: the field was predicated as autofillable
        (false positive)
   - B: the field was not predicted as autofillable
        (true negative)

These round out the logging required to compute precision, recall and
other performance metrics as defined at
https://en.wikipedia.org/wiki/Precision_and_recall.

BUG= 710474 

Review-Url: https://codereview.chromium.org/2809213002
Cr-Commit-Position: refs/heads/master@{#464127}

[modify] https://crrev.com/8f71139f8c5982d83fcdcc840a5275698b71438c/components/autofill/core/browser/autofill_metrics.h
[modify] https://crrev.com/8f71139f8c5982d83fcdcc840a5275698b71438c/components/autofill/core/browser/autofill_metrics_unittest.cc
[modify] https://crrev.com/8f71139f8c5982d83fcdcc840a5275698b71438c/components/autofill/core/browser/form_structure.cc
[modify] https://crrev.com/8f71139f8c5982d83fcdcc840a5275698b71438c/tools/metrics/histograms/histograms.xml

Project Member

Comment 4 by bugdroid1@chromium.org, Jun 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3c6caa287dacd94ee5e64eedaf44ea987f287bf8

commit 3c6caa287dacd94ee5e64eedaf44ea987f287bf8
Author: rogerm <rogerm@chromium.org>
Date: Fri Jun 02 19:58:48 2017

Log Autofill.Quality.*.ByFieldType by predicted type for unknown data.

When the actual data type of an field cannot be determined (because it
was left empty or was populated with unknown data) log the ByFieldType
version of the metrics by the type predicted by autofill. This will
allow analysis of the false-positive rate broken down by type.

The previous version of the code was logging all of these under
Empty/Unknown => AMBIGUOUS.

The CL also fleshes out the ByFieldType enumeration in histograms.xml

BUG= 710474 
R=sebsg

Review-Url: https://codereview.chromium.org/2870163003
Cr-Commit-Position: refs/heads/master@{#476766}

[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/components/autofill/core/browser/autofill_metrics.cc
[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/components/autofill/core/browser/autofill_metrics.h
[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/components/autofill/core/browser/autofill_metrics_unittest.cc
[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/components/autofill/core/browser/autofill_type.cc
[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/components/autofill/core/browser/field_types.h
[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/components/autofill/core/browser/form_structure.cc
[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/tools/metrics/histograms/enums.xml
[modify] https://crrev.com/3c6caa287dacd94ee5e64eedaf44ea987f287bf8/tools/metrics/histograms/histograms.xml

Labels: M-60 Merge-Request-60
Project Member

Comment 6 by sheriffbot@chromium.org, Jun 6 2017

Labels: -Merge-Request-60 Hotlist-Merge-Approved Merge-Approved-60
Your change meets the bar and is auto-approved for M60. Please go ahead and merge the CL to branch 3112 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), josafat@(ChromeOS), bustamante@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Status: Verified (was: Untriaged)
Project Member

Comment 8 by bugdroid1@chromium.org, Jun 6 2017

Labels: -merge-approved-60 merge-merged-3112
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8

commit a7bf9ee09dc8641a1385ac2c48edfbd6400555f8
Author: Roger McFarlane <rogerm@chromium.org>
Date: Tue Jun 06 19:15:28 2017

Log Autofill.Quality.*.ByFieldType by predicted type for unknown data.

When the actual data type of an field cannot be determined (because it
was left empty or was populated with unknown data) log the ByFieldType
version of the metrics by the type predicted by autofill. This will
allow analysis of the false-positive rate broken down by type.

The previous version of the code was logging all of these under
Empty/Unknown => AMBIGUOUS.

The CL also fleshes out the ByFieldType enumeration in histograms.xml

BUG= 710474 
R=sebsg

Review-Url: https://codereview.chromium.org/2870163003
Cr-Original-Commit-Position: refs/heads/master@{#476766}
Review-Url: https://codereview.chromium.org/2923123005 .
Cr-Commit-Position: refs/branch-heads/3112@{#198}
Cr-Branched-From: b6460e24cf59f429d69de255538d0fc7a425ccf9-refs/heads/master@{#474897}

[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/components/autofill/core/browser/autofill_metrics.cc
[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/components/autofill/core/browser/autofill_metrics.h
[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/components/autofill/core/browser/autofill_metrics_unittest.cc
[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/components/autofill/core/browser/autofill_type.cc
[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/components/autofill/core/browser/field_types.h
[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/components/autofill/core/browser/form_structure.cc
[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/tools/metrics/histograms/enums.xml
[modify] https://crrev.com/a7bf9ee09dc8641a1385ac2c48edfbd6400555f8/tools/metrics/histograms/histograms.xml

Sign in to add a comment