New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 780654 link

Starred by 6 users

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: 2018-02-11
OS: Linux , Android , Windows , Chrome , Mac , Fuchsia
Pri: 3
Type: Bug



Sign in to add a comment

Certificate Transparency - Cloudflare "nimbus2018" Log Server Inclusion Request

Reported by n...@cloudflare.com, Nov 1 2017

Issue description

Contact Information for the Log Operator
* An email or e-mail alias that is continuously monitored by the Log Operator: ct-logs@cloudflare.com
* A phone number: +1 (424) 353-4399
* A list of person(s) authorized to represent the Log Operator:
** Brendan McMillion (brendan@cloudflare.com)
** Nick Sullivan (nick@cloudflare.com)
** Patrick Donahue (pat@cloudflare.com)
** Zi Lin (zi@cloudflare.com)
** Ivan Babrou (ivan@cloudflare.com)

A public HTTP endpoint that responds to all Log Client Messages indicated in RFC 6962, Section 4:
https://ct.cloudflare.com/logs/nimbus2018

Log ID: 23Sv7ssp7LH+yj5xbSzluaq7NveEcYPHXZ1PN7Yfv2Q=

nimbus2018 is an open and free log. Certificates that are anchored by a root that is included in root store from major browsers and operating systems such as those operated by Microsoft, Apple, and Mozilla will be trusted. This trust store will be managed on Github at https://github.com/cloudflare/cfssl_trust.

* The Nimbus logs are sharded based on the leaf certificate’s expiration date
** Nimbus2018 will only accept certificates that expire between Jan 01 2018 00:00:00Z inclusive to Jan 01 2019 00:00:00Z exclusive
* Revoked and expired certificates will be accepted if their dates fall within the accepted range and they chain up to a trusted root at the time of submission and the trust chain is composed of unexpired and unrevoked CA certificates
* We reserve the right to rate limit submissions by
** IP address
** Trusted root
** An overall maximum throughput, as dictated by operational requirements
* Rate limited requests will be denied with an HTTP error status code
* The Maximum Merge Delay (MMD) of the Log is 24h
* All of the Accepted Root Certificates of the Log
** (attached)

We will freeze nimbus2018 once its inclusion expiry window has passed and close it for new submissions as of Jan 01 2019 00:00:00Z. We will then request that trust be withdrawn from this log by Chromium as all the certificates it contains will have expired.
 
pubkey.nimbus2018.der
91 bytes Download
ca-bundle.pem
528 KB Download
Components: Internals>Network>CertTrans
Owner: asymmetric@chromium.org
Status: Untriaged (was: Unconfirmed)
Cc: robpercival@chromium.org
NextAction: 2018-02-11
Owner: katjoyce@google.com
Status: Assigned (was: Untriaged)
The log application looks good and it meets all the criteria for inclusion. Assigning to begin the monitoring window. 

Note @Nick: Would it be possible to describe your rate limiting mechanism by IP, root, and overall throughput?
Cc: certific...@googlegroups.com

Comment 4 by katjoyce@google.com, Nov 14 2017

Thank you for your request, we have started monitoring your log server.
Should no issues be detected, the initial compliance monitoring phase
will be complete on the 12th of February 2018 and we will update this bug
shortly after that date to confirm.

Comment 5 by n...@cloudflare.com, Nov 15 2017

@Devon We return a 403 Forbidden if there are more than a maximum number of unsequenced leaves. Currently, this is 500k, which is <1hr at 200ops. We reserve the right to implement per-IP and per-root rate limiting with a similar mechanism when under unexpected operational duress.

Comment 6 by katjoyce@google.com, Nov 20 2017

Status: Started (was: Assigned)
My monitor has detected over 1,000 certificates in this log that do not expire in 2018 (see attached file for a list of entry indexes).

When logs do not adhere to their stated certificate expiry range, it poses problems for monitors (since they can no longer ignore a log on the basis that all of its certificates ought to be expired), and poses an ecosystem risk (since removing the log from TLS clients on schedule may cause SCTs for unexpired certificates to stop working).

Therefore, I recommend this log be rejected and replaced with a new log that adheres to its stated certificate expiry range.
nimbus2018_illegal_entries.txt
5.9 KB View Download
Andrew,

Thanks for the report. This log has adhered to its stated policy for as long as the policy has been stated. The certificates you identified were added to the log before the policy was finalized and before the log was submitted for inclusion. Furthermore, all of the identified certificates have already expired, so this should not pose an ecosystem risk.

Nick
If the certificates all expire before the expiry range, then there is no risk to monitors or the ecosystem that I can see.

That said, I do not agree that a log should be allowed to accept certificates outside the expiry range before it is submitted for inclusion.  First, it poses the risks I mentioned if the certificates happen to expire after the expiry range rather than before.  Second, it makes it hard for monitors to verify that the log is complying with the range.
I just sent the following to ct-logs@cloudflare.com (subject "Nimbus 2018: Unable to verify STH 37070971"):

At 2018-01-09 22:56:17.949 UTC, Nimbus 2018 produced the following STH ("STH 1"), which my monitor verified as corresponding to the first 37,068,980 entries in the log:

{
        "tree_size": 37068980,
        "timestamp": 1515538577949,
        "sha256_root_hash": "n0LBs7P9hcDrW1tun46Qg023VtzafYsubF+oFRnfPgA=",
        "tree_head_signature": "BAMASDBGAiEAv/cyHo5fMH5TYn/uPCUW5L0vjdz3OCgC0/5jWQiGfFMCIQCicvMpvGCwk2UvCoKf5gAwqYxwngWW4N2gXR4lGcyzlA=="
}

At 2018-01-09 23:00:17.606 UTC, Nimbus 2018 produced this STH ("STH 2"):

{
        "tree_size": 37070971,
        "timestamp": 1515538817606,
        "sha256_root_hash": "8lTjzoJhtQ1uLzB1WfeCcDrEfIj5benCanGQmi5ZmaU=",
        "tree_head_signature": "BAMARzBFAiEArwF0wOV5CfdRGs7P1fZ76hoPkxK1w9ZvjlWjO6S1B0wCIGboxzUdsPRZoXkNqZS+EryDv+hKTdAqcYdlzN2+EEJ8"
}

As of 2018-01-10 03:00:08.430 UTC, Nimbus 2018 has been unable to produce a sequence of entries in the range [37068980,37070971) that when appended to the tree of STH 1 produce the tree of STH 2.  Instead, the sequence of entries produce a tree with root hash moeYrFR2kq/UtDgAFjnfMBNWFtspxheFko2T6P+PGig=.

Interestingly, Nimbus 2018 *is* able to produce a valid consistency proof between STH 1 and STH 2.  Nevertheless, the failure to produce the corresponding entries is (potentially recoverable) misbehavior.

Note that Graham Edgecombe's monitor is also experiencing problems verifying STHs from Nimbus 2018: https://ct.grahamedgecombe.com/logs/52
...and my email to ct-logs@cloudflare.com bounced with the following message:

"We're writing to let you know that the group you tried to contact (ct-logs) may not exist, or you may not have permission to post messages to the group. A few more details on why you weren't able to post:"
Thank you for the report. We have fixed our email settings and subsequent messages to ct-logs@cloudflare.com should no longer bounce. We're looking into reproducing the issue.
I've determined that there is a problem with entries 37070048 and 37070049.

How to reproduce the problem with entry 37070048:

1. Retrieve the entry at 37070048 with https://ct.cloudflare.com/logs/nimbus2018/ct/v1/get-entries?start=37070048&end=37070048

2. Calculate the leaf hash of the returned leaf_input; when I do this I get aCoDlLbB27LrDgM/v740otM3rq59DVdUpN7Vrv2xcSY=

3. Try to obtain an inclusion proof for this leaf hash to the STH at 37070971 with https://ct.cloudflare.com/logs/nimbus2018/ct/v1/get-proof-by-hash?hash=aCoDlLbB27LrDgM%2Fv740otM3rq59DVdUpN7Vrv2xcSY%3D&tree_size=37070971 - you will get a "Not Found" error which is clearly wrong because a leaf with this hash is in the log according to get-entries
Hi all.

I apologize, we were serving different data from get-entries than we signed the STHs over. This should be better now, so if you re-crawl from entry 37068980 all the STHs you observed should verify. The affected entries were 37070048, 37070049, 37083753; the leaves that were originally there were put back and the incorrect leaves were moved to 37328014, 37324261, 37329726 resp. We also detected that we issued 5 SCTs without sequencing the corresponding certificates; this has been resolved as well (within the MMD).

Internally, we store all leaf data in Kafka. My (limited) understanding is that routine maintenance on our Kafka cluster caused us to sign an STH over one view of the data, but afterwards the cluster came to consensus that a slightly different view of the data was canonical. I will adjust how we configure Kafka's write durability to try to prevent this from happening in the future.
I've been able to fetch entries that correspond to the published STHs.  Thanks!

Is there a chance you signed any STHs over the other view of the log?
No, all the STHs we signed are consistent with each other.
Will Cloudflare provide a detailed post mortem of this incident?  Other log operators have provided detailed post mortems following issues, and post mortems are helpful for reassuring relying parties and educating other log operators.
The NextAction date has arrived: 2018-02-11
Owner: pphaneuf@google.com
This log has passed the initial 90 day compliance period and we will start the process to add this to Chrome.
Sorry, upon closer inspection, there seems to be possible issues with this log:

https://ct.grahamedgecombe.com/logs/52

We're investigating, and will report back shortly.
This issue was reported in comment 10 of this ticket, and resolved in comment 14. Graham just hasn't updated his monitor.
My apologies, that's absolutely correct, I just replied immediately to have an update (as it was reaching the end of the day, and I didn't want to rush this), and this is indeed already resolved.

I'll be resuming the process to add this log to Chrome.
Project Member

Comment 24 by bugdroid1@chromium.org, Mar 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/79966f2ee55749a3d9494f8beb1bcd9a5dcca373

commit 79966f2ee55749a3d9494f8beb1bcd9a5dcca373
Author: Devon O'Brien <asymmetric@chromium.org>
Date: Thu Mar 01 19:33:09 2018

Add Nimbus and Argon to Trusted CT Logs

The following CT Logs have passed their monitoring period and
are being added as trusted Logs in Chrome:

Google Argon2018, Argon2019, Argon2020, Argon2021
Cloudflare Nimbus2018, Nimbus2019, Nimbus2020, Nimbus2021

Bug:  756814 ,  756817 ,  756818 ,  756819 ,  780654 ,  780655 ,  780656 ,  780657 
Change-Id: I6b8671db0dc7ba34b666345049934ed3e2b5705a
Reviewed-on: https://chromium-review.googlesource.com/942688
Reviewed-by: Ryan Sleevi <rsleevi@chromium.org>
Commit-Queue: Ryan Sleevi <rsleevi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#540254}
[modify] https://crrev.com/79966f2ee55749a3d9494f8beb1bcd9a5dcca373/net/data/ssl/certificate_transparency/log_list.json

Labels: Merge-Request-65
Project Member

Comment 26 by sheriffbot@chromium.org, Mar 1 2018

Labels: -Merge-Request-65 Merge-Review-65 Hotlist-Merge-Review
This bug requires manual review: We are only 4 days from stable.
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Pls apply appropriate OSs label. Thank you.
Labels: OS-Chrome
Owner: asymmetric@chromium.org
Labels: OS-Linux
Labels: OS-Mac
Labels: OS-Windows
Labels: OS-Android
Labels: OS-Fuchsia
Cc: awhalley@chromium.org cma...@chromium.org
+awhalley@ for M65 merge review.

+cmasso@ as FYI.
Status: Fixed (was: Started)
Labels: -Merge-Review-65
Tracking any merges for this in  issue 756814 
Project Member

Comment 37 by bugdroid1@chromium.org, Mar 2 2018

Labels: merge-merged-3325
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a293012d1d566826faba24c33e52343453fcedbd

commit a293012d1d566826faba24c33e52343453fcedbd
Author: Ryan Sleevi <rsleevi@chromium.org>
Date: Fri Mar 02 18:06:44 2018

Add Nimbus and Argon to Trusted CT Logs

The following CT Logs have passed their monitoring period and
are being added as trusted Logs in Chrome:

Google Argon2018, Argon2019, Argon2020, Argon2021
Cloudflare Nimbus2018, Nimbus2019, Nimbus2020, Nimbus2021

TBR=asymmetric@chromium.org

(cherry picked from commit 79966f2ee55749a3d9494f8beb1bcd9a5dcca373)

Bug:  756814 ,  756817 ,  756818 ,  756819 ,  780654 ,  780655 ,  780656 ,  780657 
Change-Id: I6b8671db0dc7ba34b666345049934ed3e2b5705a
Reviewed-on: https://chromium-review.googlesource.com/942688
Reviewed-by: Ryan Sleevi <rsleevi@chromium.org>
Commit-Queue: Ryan Sleevi <rsleevi@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#540254}
Reviewed-on: https://chromium-review.googlesource.com/946568
Cr-Commit-Position: refs/branch-heads/3325@{#647}
Cr-Branched-From: bc084a8b5afa3744a74927344e304c02ae54189f-refs/heads/master@{#530369}
[modify] https://crrev.com/a293012d1d566826faba24c33e52343453fcedbd/net/data/ssl/certificate_transparency/log_list.json

(M65 merge approval granted in 756814)
We are adding the ACCVRAIZ1 root certificate which belongs to the Government of Spain, Autoritat de Certificació de la Comunitat Valenciana (ACCV). Attached.
ACCVRAIZ1.crt
2.8 KB Download
To be clear, I'm adding the root to all shards, but only updating this ticket.

Comment 41 by cmasso@google.com, Mar 29 2018

I hope all of this is in M66 as well.
Labels: -Hotlist-Merge-Review
We are adding the A-Trust-Root-05 root certificate which belongs to Austrian CA A-Trust. Again, we're adding the root to all log shards but only updating this ticket. Attached.
A-Trust-Root-05.crt
2.0 KB Download
We are adding the Szafir Root CA2 which belongs to the National Clearing House of Poland. Attached.
szafir_root_ca2.crt
1.2 KB Download
We are adding SSL.com's 4 publicly trusted roots. Attached.
SSLcom-RootCA-EV-RSA-4096-R2.pem
2.3 KB Download
SSLcomEVRootCertificationAuthorityECC.pem
956 bytes Download
SSLcomRootCertificationAuthorityECC.pem
944 bytes Download
SSLcomRootCertificationAuthorityRSA.pem
2.0 KB Download
We are adding two roots which belong to Network Solutions. Attached.
100961666.crt
956 bytes Download
100961690.crt
2.1 KB Download
We are adding two roots which belong to the Hellenic Academic and Research Institutions Certificate Authority (HARICA). Attached.
HaricaRootCA2015.pem
2.1 KB Download
HaricaECCRootCA2015.pem
1017 bytes Download
Is there any chance you have updated the Nimbus root set in the past couple of days?
Yes. The CA that requested the change is not happy yet, so I'm still working with them to determine the root set they need added.
Okay, just checking as our monitor picked up new roots but we saw no announcement, which is rare for you guys!  The explanation that you're still working on getting the root set into the correct state makes sense - we look forward to you posting what it settles on :)
We've added a number of roots that belong to the Camerfirma CA. These are attached, concatenated.
camerfirma.pem
23.0 KB Download

Sign in to add a comment