I have a hard time reproducing it. I think the issue is coming from what ChromeReader sends us. Can you please use chrome://snippets-internal to send me examples of duplicates?

Also, did you have --ntp-snippets-dont-restrict set? (that's also visible from chrome://snippets-internal)

I'll report a case as soon as I see it.

No, didn't have that flag enabled. What is it for?

I have this case again, but chrome://snippets-internal does not exist in the latest Canary.

Duplicates:
http://edition.cnn.com/2016/04/18/asia/philippines-rodrigo-duterte-gang-rape-comment-apology/
http://m.wdsu.com/national/philippine-candidate-sorry-over-gang-rape-joke/39096116

May, I think this should get resolved when your change regarding content info lands:

When there are multiple URLs with the same content, then contentInfo.url points to just one of them. But the source_corpus_info will have the correct url based on the restrict.

Do you have a bug on file? If yes, we may want to dupe this one against it.

We don't have a bug for it specifically, I had as part of my favicon change. 

Note that if there are multiple sources within your host restrict that have the article (e.g. you have both nytimes.com and cnn.com and they both carry the same story), duplicates could still happen unless the source_corpus_info ordering is deterministic. Right now we use the first source_corpus_info we find as the source of the article, but if that order changes when we get another set of data from ChromeReader, we could have dupes.

It's worth keeping around this bug to fix for that case (e.g., look at all the available sources and de-dupe based on that).

Starting on this after 596410 lands, as it depends on it.

Also make sure DiscardSnippet() uses the same de-duping logic.

 Issue 596410  is fixed. I guess this is next now? :)

Well,  Issue 609112  has higher pri on my list. In fact, I'll unassign for now to allow others to take this if they want as it's a pretty independent change. I'll pick it up later if no one else has.

When fixing this bug, make sure to make a pass over SnippetsService and fix all the places that rely only on the main URL as the identifier of a snippet.

I have found another repro of this - two "identical" snippets where the main URL differs only by a query param. Screenshot attached.

Deleted: 2016-05-10.png 337 KB

	2016-05-10.png 337 KB View Download

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ab78ef3824631e1ef5fffba7c21946d66b9ad84f

commit ab78ef3824631e1ef5fffba7c21946d66b9ad84f
Author: tschumann <tschumann@chromium.org>
Date: Wed May 18 20:42:27 2016

Use multiple IDs when discarding or merging snippets.

Changes the discarding and de-duplication logic to take IDs of all snippet
sources (these are always URLs in our cases) into account.

R=treib@chromium.org
BUG= 603884 

Review-Url: https://codereview.chromium.org/1992803002
Cr-Commit-Position: refs/heads/master@{#394539}

[modify] https://crrev.com/ab78ef3824631e1ef5fffba7c21946d66b9ad84f/components/ntp_snippets/ntp_snippets_service.cc
[modify] https://crrev.com/ab78ef3824631e1ef5fffba7c21946d66b9ad84f/components/ntp_snippets/ntp_snippets_service_unittest.cc

verified in M52-52.0.2743.8