Reported suggestion position in UMA is wrong |
||||
Issue descriptionThe position reported to UMA in various metrics (suggestion shown, opened, dismissed, context menu, action item pressed) is currently set at the moment a suggestion is created, but its position can change over its lifetime since other suggestions can be removed or added. So this number is not accurate by any mean nor guaranteed to be monotonically increasing for a given instance of the NTP. Link for where we set the value: https://cs.chromium.org/chromium/src/chrome/android/java/src/org/chromium/chrome/browser/ntp/cards/SectionList.java?q=sectionlist&sq=package:chromium&dr=CSs&l=200&rcl=1482135521 What exactly do we want to track with these metrics? (NewTabPage.ContentSuggestions.Shown, Opened, etc) Is it okay if we start using the index of the item in the NTP instead of trying to get the index of a suggestion? We already have this for internal housekeeping. The difference would be that an item can be a status card or a header, while we don't report any position for these. (Note: we do report a position for the MORE button. It's currently the number of suggestions in the section.)
,
Dec 19 2016
In any case: If we change the semantics of UMA histograms, we have to rename them.
,
Dec 19 2016
A case where it's quite wrong: - 3 sections are loaded, suggestions are labelled 0-29 - User taps "Fetch More" on the middle section - the labels are: Section 1: 0-9, Section 2: 10-29, Section 3: 20-29 What I'm suggesting is just using the plain index in the UI. We would be able to compute it when the UMA is reported instead of having it stored at initialisation and be wrong when there are changes. Alternatively we could do something about calculating the absolute suggestion index, but I don't know what we would do about the action items then. Example NTP with 4 sections Plain index in UI | Current UMA | Sugg index ---------------------+-------------+-------- 0 - Above the fold | | 1 - header | | 2 - suggestion | 0 | 0 3 - suggestion | 1 | 1 4 - action item | 2 | ? 5 - header | | 6 - status card | | 7 - action item | 0 | ? 8 - progress item | | 9 - Header | | 10 - status card | | 11 - action item | 0 | ? 12 - header | | 13 - suggestion | 2 | 2 14 - suggestion | 3 | 3 15 - action item | 2 | ? 16 - footer | | 17 - spacer | |
,
Dec 22 2016
Indeed, this doesn't make much sense. Using the UI index sounds reasonable. But should clicks on actions items even emit to NewTabPage.ContentSuggestions.Shown, Opened, etc? I'd say that these histograms are meant to track interactions with *content suggestions* and not with UI elements.
,
Dec 22 2016
The click on the action item is reported as NewTabPage.ContentSuggestions.MoreButtonClicked, but we still report its position. I don't feel strongly about using the UI index or the global suggestion index in the reported metric, as long as it's consistent with reality and the signification of each reported value is clear. Currently the action item reports its position in the current section (excluding the header), which is fine. It is not coherent with what would be reported by a suggestion being clicked, but as long as it is made clear and that is what is intended, there is no issue with that. We will just know that we're not supposed to compare these two positions.
,
Jan 6 2017
The NewTabPage.ContentSuggestions.<Event>.<Category> metrics (e.g. NewTabPage.ContentSuggestions.Shown.Articles) also has issues in some different way. It currently assumes that the max number of suggestions per category is 10. That is a wrong assumption when you factor in the "Fetch More" feature. Also, because of the way we set the position/rank of the incoming suggestions (we set it to the position in the new bundle rather than the position in the section itself) additional suggestions are assigned to the <10 position anyway. This also affects the global ranking of the suggestion (NewTabPage.ContentSuggestions.<Event>), which is calculated from the section specific one. One consequence of that is that earlier buckets will be over represented and we don't have much visibility on what happens with user request additional snippets. AFAICT we just have More button interactions like NewTabPage.ContentSuggestions.MoreButtonClicked.Articles, which bundles interactions on 10+ together anyway.
,
Jan 6 2017
Additional issue: The reported position of the More button in the section is currently dynamic. It's its position in the section at the moment it is tapped, while suggestions report their positions at the moment they are created.
,
Jan 16 2017
To #6, increasing the max number of suggestions per category is discussed in issue 678644.
,
Jan 16 2017
,
Jan 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5549f302eeb5cc43d3e13435c3826587721ddee8 commit 5549f302eeb5cc43d3e13435c3826587721ddee8 Author: dgn <dgn@chromium.org> Date: Wed Jan 18 14:55:54 2017 [NTP Client] Tweak the suggestion ranks for UMA to handle fetchMore There are 2 types of suggestion positions we report to UMA: - Within a given section: because we were attributing ranks in the incoming bundle, it was not taking into account the suggestions that had been pushed before. This number is now always increasing, so after fetching more suggestions 5 times, the ranks for the 5th bundle can be in the 40 to 49 range. - Global rank: we also now attribute the rank as suggestions are added to the section, and keep track of that number. It would be the position of the suggestion if we kept adding suggestions always at the end of the list rather than grouping them by category. This way for a given NTP, there can be no overlap of global rank or local rank within a section. BUG= 675623 Review-Url: https://codereview.chromium.org/2618893003 Cr-Commit-Position: refs/heads/master@{#444354} [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/NewTabPage.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/NewTabPageView.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/cards/ActionItem.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/cards/SectionList.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/cards/SuggestionsSection.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/snippets/SnippetArticle.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/snippets/SnippetArticleViewHolder.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/ntp/snippets/SnippetsBridge.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/suggestions/ContentSuggestionsActivity.java [add] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/suggestions/SuggestionsMetricsReporter.java [add] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java/src/org/chromium/chrome/browser/suggestions/SuggestionsRanker.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/java_sources.gni [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/javatests/src/org/chromium/chrome/browser/ntp/cards/NewTabPageRecyclerViewTest.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/javatests/src/org/chromium/chrome/browser/ntp/snippets/ArticleSnippetsTest.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/junit/src/org/chromium/chrome/browser/ntp/cards/ContentSuggestionsTestUtils.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/junit/src/org/chromium/chrome/browser/ntp/cards/NewTabPageAdapterTest.java [add] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/junit/src/org/chromium/chrome/browser/ntp/cards/SectionListTest.java [modify] https://crrev.com/5549f302eeb5cc43d3e13435c3826587721ddee8/chrome/android/junit/src/org/chromium/chrome/browser/ntp/cards/SuggestionsSectionTest.java
,
Jan 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ecb7bd1cd2294ead17cb71265c98a9574dd5f342 commit ecb7bd1cd2294ead17cb71265c98a9574dd5f342 Author: dgn <dgn@chromium.org> Date: Thu Jan 19 10:08:13 2017 [NTP Client] Use first seen position in suggestion UMA Updates the description and calculation to be more consistent with what we want to track and more accurate across the different modifications the suggestion surface can go through. This means that instead of tracking the position at the moment the item is loaded, we track the position when it is bound to the view and the user has a chance to see it. BUG= 675623 Review-Url: https://codereview.chromium.org/2640443003 Cr-Commit-Position: refs/heads/master@{#444693} [modify] https://crrev.com/ecb7bd1cd2294ead17cb71265c98a9574dd5f342/chrome/android/java/src/org/chromium/chrome/browser/ntp/cards/SuggestionsSection.java [modify] https://crrev.com/ecb7bd1cd2294ead17cb71265c98a9574dd5f342/chrome/android/java/src/org/chromium/chrome/browser/suggestions/SuggestionsRanker.java [modify] https://crrev.com/ecb7bd1cd2294ead17cb71265c98a9574dd5f342/chrome/android/junit/src/org/chromium/chrome/browser/ntp/cards/SectionListTest.java [modify] https://crrev.com/ecb7bd1cd2294ead17cb71265c98a9574dd5f342/tools/metrics/histograms/histograms.xml
,
Jan 19 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by treib@chromium.org
, Dec 19 2016