Could it be that the de-duplication logic is broken? Would you mind taking a look?

The client-side deduplication is host-based and the hosts of both tiles are different. This means:
https://top.com/x == http://top.com/y
but 
sub.top.com != top.com

There are several possibilities to change that but every solution has different problems:


1). We could change that to check whether one domain is sub domains of the other:

Top domains would consume mobile domains but www.* and m.* domains wouldn't be deduplicated.
m.youtube.com == youtube.com
www.youtube.com == youtube.com
but:
m.youtube.com != www.youtube.com

2.) Deduplicate by tile name:
Every site with "Welcome!" as title would be identical and everything that changes title's frequently would appear multiple times. Quite brittle.

3.) Check for common higher level domain:
espn.com == www.espn.com == m.espn.com
www.a.blogspot.com == a.blogspot.com
but:
news.google.co.uk == google.co.uk
a.blogspot.com == b.blogspot.com

4.) Special case mobile.*, m.*, www.* hosts. This will break for some pages as well but might still be preferable over what we use today.


Any preferences/other ideas?

(I think this is less of a bug and more of an improvement/feature request. Therefore, I do not think we really need this change for M-61.)

Issue 703628 has been merged into this issue.

I prepared a CL for 4) because it improves the situation at least slightly: https://crrev.com/c/628879

It would still be nice to know whether we want those special cases.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b1733404dbb36a76e44206e57431a62458b38ac3

commit b1733404dbb36a76e44206e57431a62458b38ac3
Author: Friedrich Horschig <fhorschig@chromium.org>
Date: Mon Aug 28 17:06:11 2017

Improve deduplication for ntp tiles

This CL introduces a simple blacklist so that popular sites don't
produce duplicates with similar mobile sites.
If ML or TS deliver such duplicate, it will not be prevented.

Bug:  752928 
Change-Id: Ie7222ecb6468d4d727343f7e5416a16be02b380c
Reviewed-on: https://chromium-review.googlesource.com/628879
Commit-Queue: Friedrich Horschig <fhorschig@chromium.org>
Reviewed-by: Chris Pickel <sfiera@chromium.org>
Cr-Commit-Position: refs/heads/master@{#497782}
[modify] https://crrev.com/b1733404dbb36a76e44206e57431a62458b38ac3/components/ntp_tiles/most_visited_sites.cc
[modify] https://crrev.com/b1733404dbb36a76e44206e57431a62458b38ac3/components/ntp_tiles/most_visited_sites.h
[modify] https://crrev.com/b1733404dbb36a76e44206e57431a62458b38ac3/components/ntp_tiles/most_visited_sites_unittest.cc

Verified on:

App version: 62.0.3202.40 beta
Devices: iPhone 6 Plus, iPad Air
iOS Versions: 10.3.3, 11.0

NTP contains only ‘Most Visited’ tile when navigated to a site which is already listed as ‘Popular Site’. Issue is fixed.