Before adding any persistence, please make sure to include me in the review. As presently implemented, it may represent privacy concerns, so we should resolve those first.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6

commit b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6
Author: eranm <eranm@chromium.org>
Date: Fri Jul 08 14:10:44 2016

Certificate Transparency: MerkleTreeLeaf, MerkleAuditProof improvements.

Per comments on another code review, I've documented MerkleTreeLeaf and
changed the Hash function name.

This change also adds the tree size to the MerkleAuditProof, as each
proof ties to a particular tree size (and it can't directly be
deduced from the number of nodes in the proof).

BUG= 613495 

Review-Url: https://codereview.chromium.org/2107423004
Cr-Commit-Position: refs/heads/master@{#404356}

[modify] https://crrev.com/b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6/net/cert/merkle_audit_proof.cc
[modify] https://crrev.com/b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6/net/cert/merkle_audit_proof.h
[modify] https://crrev.com/b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6/net/cert/merkle_tree_leaf.cc
[modify] https://crrev.com/b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6/net/cert/merkle_tree_leaf.h
[modify] https://crrev.com/b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6/net/cert/merkle_tree_leaf_unittest.cc
[modify] https://crrev.com/b38ec0a4aaf590b21c1344a5962bcfeb65f3e3d6/net/test/ct_test_util.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b03a8a77ead185c85a6c55303ac20176eaa20a56

commit b03a8a77ead185c85a6c55303ac20176eaa20a56
Author: eranm <eranm@chromium.org>
Date: Mon Jul 25 14:20:31 2016

Measure how often SCTs can't be checked because they're too new

Signed Certificate Timestamps (SCTs) designate Merkle tree leaves that can
be checked for inclusion with a CT Log's Signed Tree Head (STH).
However, inclusion should only be checked against an STH that is newer
than the SCTs timestamp by at least 24 hours - this is because Logs'
have a Maximum Merge Delay of 24 hours, which is the time they have
to produce a new STH that incorporates a given SCT.

STHs are delivered periodically out of band. If there isn't a new enough
STH, then SCTs will need to be marked as pending inclusion check,
waiting for a new STH to be delivered.

To determine how frequently an STH should be delivered (currently,
daily) and how big the SCT queue list should be (that is, how frequently
clients encounter a brand new SCT that is newer than any STH),
measure how often an SCT can't be checked for inclusion immediately
after it's been seen.

BUG= 613495 

Review-Url: https://codereview.chromium.org/2153123002
Cr-Commit-Position: refs/heads/master@{#407471}

[modify] https://crrev.com/b03a8a77ead185c85a6c55303ac20176eaa20a56/components/certificate_transparency/single_tree_tracker.cc
[modify] https://crrev.com/b03a8a77ead185c85a6c55303ac20176eaa20a56/components/certificate_transparency/single_tree_tracker_unittest.cc
[modify] https://crrev.com/b03a8a77ead185c85a6c55303ac20176eaa20a56/tools/metrics/histograms/histograms.xml

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1b5a833bca72a9f89e8c6e8c592037540eaf1411

commit 1b5a833bca72a9f89e8c6e8c592037540eaf1411
Author: eranm <eranm@chromium.org>
Date: Mon Jan 23 22:21:52 2017

Audit CT logs by requesting inclusion proofs for
observed Signed Certificate Timestamps from the
CT log that issued them.

To verify that a CT log behaves correctly and indeed
publishes all the certificates it committed to publishing,
it is necessary to verify that each observed log entry
(as denoted by an SCT and the corresponding
certificate) is included in the log.

Check for inclusion of observed SCTs by finding out
the leaf index of each observed SCT and requesting
an inclusion proof from the CT log for it over DNS
(using the LogDnsClient).

Note that no action is taken based on the inclusion
check results in this change. Follow-up changes
will add telemetry.

BUG= 613495 

Review-Url: https://codereview.chromium.org/2017563002
Cr-Commit-Position: refs/heads/master@{#445513}

[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/components/certificate_transparency/BUILD.gn
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/components/certificate_transparency/single_tree_tracker.cc
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/components/certificate_transparency/single_tree_tracker.h
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/components/certificate_transparency/single_tree_tracker_unittest.cc
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/components/certificate_transparency/tree_state_tracker.cc
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/components/certificate_transparency/tree_state_tracker.h
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/net/cert/merkle_audit_proof.cc
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/net/cert/merkle_audit_proof.h
[modify] https://crrev.com/1b5a833bca72a9f89e8c6e8c592037540eaf1411/tools/metrics/histograms/histograms.xml

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bbf5af70f469a5f7807ec4573a65c3710cfbb29a

commit bbf5af70f469a5f7807ec4573a65c3710cfbb29a
Author: eranm <eranm@chromium.org>
Date: Thu Feb 02 16:06:18 2017

Wire NetLog into the TreeStateTracker

Pass a NetLog instance from the IOThread/ProfileIOData into the
TreeStateTracker instance created in each, so that CT log auditing
events can be logged into the NetLog and DNS queries related to log
auditing are also logged.

A new NetLog source was created to track NetLog events related to
CT log auditing, since they happen independently of the SSL
connections in which the  certificates and SCTs were observed.

BUG= 613495 

Review-Url: https://codereview.chromium.org/2650803004
Cr-Commit-Position: refs/heads/master@{#447772}

[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/chrome/browser/io_thread.cc
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/chrome/browser/profiles/profile_io_data.cc
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/components/certificate_transparency/BUILD.gn
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/components/certificate_transparency/single_tree_tracker.cc
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/components/certificate_transparency/single_tree_tracker.h
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/components/certificate_transparency/single_tree_tracker_unittest.cc
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/components/certificate_transparency/tree_state_tracker.cc
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/components/certificate_transparency/tree_state_tracker.h
[add] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/components/certificate_transparency/tree_state_tracker_unittest.cc
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/net/log/net_log_event_type_list.h
[modify] https://crrev.com/bbf5af70f469a5f7807ec4573a65c3710cfbb29a/net/log/net_log_source_type_list.h

Re-assigning to Rob since he's taken over that code.

This has been done, bar the persistence of Merkle tree leaves. I don't think we actually want to persist them though. The DNS inclusion checking privacy changes require that we know whether the network has changed, and this information would not be available between Chrome restarts.