Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Issue 664177 Out of date Chrome results in ERR_CERTIFICATE_TRANSPARENCY_REQUIRED for Symantec operated sites
Starred by 86 users Project Member Reported by rsleevi@chromium.org, Nov 10 Back to list
Status: Verified
Owner:
Closed: Nov 18
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 0
Type: Bug

Restricted
  • Only users with EditIssue permission may comment.



Sign in to add a comment
Beginning with Chrome 53, Certificate Transparency was required for Symantec sites (as announced at https://security.googleblog.com/2015/10/sustaining-digital-certificate-security.html and implemented in Issue 620178)

The goal of such a policy is "Only trust Symantec if we're confident in CT" - that is, that the default policy would have been to remove Symantec, but that Symantec remains trusted, because we trust CT.

However, the CT information has a built-in build-time bomb of 10 weeks - after 10 weeks from build time, the CT code no longer believes it can trust in CT information, because logs may have been added or removed. This is to ensure that an old Chrome client doesn't blindly trust logs known to be untrustworthy. For EV, this was an acceptable behaviour - as it means out of date Chrome versions no longer show the EV bar. This is also similar to the build-time bombs of HSTS and HPKP - except that they disable their security functionality after 10 weeks, and due to them being additive policies rather than subtractive, 'fail open' (read: make Chrome less secure).

When these two implementations were combined with another change - https://chromium.googlesource.com/chromium/src/+/37be86cd32e1ccde5b193f53d7f568cf183880f9 - which changed the default response for CT from being an affirmative "Yes, it complies" to "No, I don't know" - a fail-closed timebomb was unintentionally set, where, after 10 weeks from build time, Symantec sites (such as www.amazon.com), fail to operate.
 
Submitting a Finch CL to provide relief for the out-of-date users, tweaking the current Chrome trunk to be more forgiving, and then we'll see how we wan to merge (given the Finch option)
Blockedon: 664205
Owner: rsleevi@chromium.org
Status: Started
This affects Geotrust too, as the same measures were applied against them. Any others? It might be just worth pointing this out as not everyone with a cert understands the ownership between brands.

Not a lot to be done here is there for all the people with affected certs? Nothing remotely that you can do to enable CT on those affected builds since in your own words it is a 'built-in build-time bomb'?


The set of certificates is in https://chromium.googlesource.com/chromium/src/+/master/net/data/ssl/symantec

And we're exploring solutions for out of date users - but of course, patches won't reach out of date users.
Might be good as reference to list when possible the builds & CT bomb date. 

Example - 53.0.2785.89 was released on 31st as per the blog, but it was actually built on 30th. Makes a difference to those operating sites and tracking the effect. 
Project Member Comment 7 by bugdroid1@chromium.org, Nov 12
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec8e431e9a0f80ace76368ce7edce006f3d409f2

commit ec8e431e9a0f80ace76368ce7edce006f3d409f2
Author: rsleevi <rsleevi@chromium.org>
Date: Sat Nov 12 00:32:28 2016

Defang the CT Timebomb

A timebomb existed that if the set of known CT logs goes stale,
no certificate can comply with the Certificate Transparency
policy, beause the logs may be out of date and not trustworthy. The
set of known logs goes stale 10 weeks after the build date.

While this was acceptable to cause EV certificates to downgrade to
DV certificates, this also means that certificates issued by CAs
that MUST be CT compliant also fail to work - they fail closed,
rather than fail open. In particular, certificates issued by
Symantec fail to work if it's more than 10 weeks after the build
date - in effect, the default behaviour is to distrust Symantec.

While the proper behaviour is debated - to either fail open (like
HSTS and HPKP do), treating CT as additive, or to fail closed,
treating CT as a restrictive policy that must be made - change
the code to allow an out of date build to skip the CT checks,
failing open.

BUG= 664177 

Review-Url: https://codereview.chromium.org/2495583002
Cr-Commit-Position: refs/heads/master@{#431707}

[modify] https://crrev.com/ec8e431e9a0f80ace76368ce7edce006f3d409f2/net/quic/chromium/crypto/proof_verifier_chromium.cc
[modify] https://crrev.com/ec8e431e9a0f80ace76368ce7edce006f3d409f2/net/socket/ssl_client_socket_impl.cc
[modify] https://crrev.com/ec8e431e9a0f80ace76368ce7edce006f3d409f2/net/spdy/spdy_session.cc

Labels: -ReleaseBlock-Beta M-55
Forgot to set mstone target. Letting bake on 56, but will want to get this to 55 by EOW.
Issue 664813 has been merged into this issue.
Cc: mef@chromium.org
Issue 663883 has been merged into this issue.
Issue 664945 has been merged into this issue.
Issue 664798 has been merged into this issue.
Hello Chromium devs, what are the source code changes that one can make in order to fix the issue with building an older custom build of Chromium 53 as well as keeping the Certificate Transparency checks?
Issue 664913 has been merged into this issue.
Issue 664938 has been merged into this issue.
re: Comment 13: Please update to a newer version of Chrome. Keeping Chrome 53 is strongly discouraged because it will not receive any security updates and is out of the range that we can reasonably keep secure.
re: Comment 13: That being said, the patch in comment 7 seems to be relatively simple changes.
But it is not recommended. Please update Chrome.
For some reason the new build is not available through yum in ReadHat
re: comment 16 How about Chromium 53 instead of Chrome 53?
On ubuntu Chromium 53 *is* the latest version which installs from the standard repositories.
re: Comment 21: there is already a Fixed Committed bug in the Ubuntu bug tracker here: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1641380 (marked yours as duplicate)

The updated package should be released soon, and in the meantime Ubuntu users can use builds from this PPA: https://launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage (or switch to Chrome instead of Chromium)
Issue 665170 has been merged into this issue.
Upgraded to ubuntu 16.10 - solved (53.0.2785.143 now)
re: comment 21:
I installed 53.0.2785.143 from the staging repositories and now problems are gone.
Strangely enough, I originally also had 53.0.2785.143 which gave the errors.

Note that Google doesn't support old versions, so odds are that the version numbers are the same because that's based on Chromium 53.0.2785.143, with some additional unofficial patches on top of it.
re: comment 25: Whether you see this error depends on the build date of the Chromium build.
Issue 664825 has been merged into this issue.
Any resolution to this issue on Chromium version 53.0.2785.143 Built on Ubuntu , running on Ubuntu 16.04.1 (64-bit)?

Issue 665910 has been merged into this issue.
Labels: Merge-Request-55
Labels: -Merge-Request-55 Merge-Approved-55 Hotlist-Merge-Approved
Your change meets the bar and is auto-approved for M55 (branch: 2883)
I don't understand, guys, I have Chromium, why was it merged into a Chrome issue?
#33 - Chrome and Chromium share the same code base. Chrome is a Chromium based browser.
where is the CT information stored? is there a way we can keep our old Chromium builds updated with the latest Certificate Transparency logs?
re: Comment 29: Up to the distro.
Cc: krajshree@chromium.org
Issue 664790 has been merged into this issue.
Cc: ligim...@chromium.org
Issue 665806 has been merged into this issue.
Project Member Comment 40 by bugdroid1@chromium.org, Nov 16
Labels: -merge-approved-55 merge-merged-2883
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/98538c32c96bed2d6d68d89cb4d06c49b8f5e0b5

commit 98538c32c96bed2d6d68d89cb4d06c49b8f5e0b5
Author: Ryan Sleevi <rsleevi@chromium.org>
Date: Wed Nov 16 22:32:10 2016

Defang the CT Timebomb

A timebomb existed that if the set of known CT logs goes stale,
no certificate can comply with the Certificate Transparency
policy, beause the logs may be out of date and not trustworthy. The
set of known logs goes stale 10 weeks after the build date.

While this was acceptable to cause EV certificates to downgrade to
DV certificates, this also means that certificates issued by CAs
that MUST be CT compliant also fail to work - they fail closed,
rather than fail open. In particular, certificates issued by
Symantec fail to work if it's more than 10 weeks after the build
date - in effect, the default behaviour is to distrust Symantec.

While the proper behaviour is debated - to either fail open (like
HSTS and HPKP do), treating CT as additive, or to fail closed,
treating CT as a restrictive policy that must be made - change
the code to allow an out of date build to skip the CT checks,
failing open.

BUG= 664177 

Review-Url: https://codereview.chromium.org/2495583002
Cr-Commit-Position: refs/heads/master@{#431707}
(cherry picked from commit ec8e431e9a0f80ace76368ce7edce006f3d409f2)

Review URL: https://codereview.chromium.org/2511583002 .

Cr-Commit-Position: refs/branch-heads/2883@{#594}
Cr-Branched-From: 614d31daee2f61b0180df403a8ad43f20b9f6dd7-refs/heads/master@{#423768}

[modify] https://crrev.com/98538c32c96bed2d6d68d89cb4d06c49b8f5e0b5/net/quic/chromium/crypto/proof_verifier_chromium.cc
[modify] https://crrev.com/98538c32c96bed2d6d68d89cb4d06c49b8f5e0b5/net/socket/ssl_client_socket_impl.cc
[modify] https://crrev.com/98538c32c96bed2d6d68d89cb4d06c49b8f5e0b5/net/spdy/spdy_session.cc

Issue 665783 has been merged into this issue.
Issue 665692 has been merged into this issue.
Issue 665484 has been merged into this issue.
Updating to Chromium to version 54 resolves the issue.
For linux mint:
sudo add-apt-repository ppa:canonical-chromium-builds/stage
sudo apt-get update
sudo apt-get install chromium-browser
I can also verify that comments and suggestion made in comment 44 also fixes the issue in Ubuntu 16.04.1 but it upgrades chromium to 53.0.2785.143.
The problem doesn't fixed by comment 44 in 53.0.2785.143 Ubuntu 16.04.1
Issue 666177 has been merged into this issue.
Status: Verified
Marking this as Verified
Issue 665430 has been merged into this issue.
Issue 666650 has been merged into this issue.
There are a bunch of recent comments and issue merges mentioning this issue currently as afflicting 53.0.2785.143. 

However 53.0.2785.143 was released just over 7 weeks ago. Is there really a 3 week gap from build to release for that version which would mean it is 10 weeks since build and therefore post-Timebomb?

Apologies if I am missing something but as per comment 6 I'm just trying to keep track of when the major releases of Chrome in the market are coming up to their Timebomb date.
I have noticed also that some https connections do not update. They will load but not update from the last good download of the page.

Case: https://www.txarmymars.org/resources/solarweather.php
The first graphic will not update but the rest do.

Not noticed on FireFox.
Hello,The problem was solved with an update received . THX .
 

    On Friday, November 18, 2016 6:24 PM, ad… via monorail <monorail+v2.3387730482@chromium.org> wrote:
Well the version 54.0.2840.99 m does not solve the problem. I just checked.
#54 - that does not make a lot of sense, so you might have a different issue...
I did a software update and Chromium is working again no more "Out of date
Chrome results in ERR_CERTIFICATE_TRANSPARENCY_REQUIRED for Symantec
operated sites".

Thanks everyone for fixing this.
I continue to check for a new version of Chromium, but none exists. I now have an error on my computer that states:
 ERror: Please run package manager from the right click menu or apt; get in a terminal to find out what's wrong. The error message was "unknown error <class 'systemclass'> (E: the package google chrome needs to be reinstalled but i can't find an archive for it - there are probably unmet dependencies..

guess this 65 year old lady is stuck now.. 

I used chromium from ubuntu staging repositories. It solved the certificates problem but broke youtube videos.
I removed the repository, purged chromium and removed it from /var/cache/apt/archives and installed it again from default repositories.
Problems are now gone. Everything works again, including youtube.
Thanks for all the nerds fixing this, where would the world be without you.
Issue 667777 has been merged into this issue.
Issue 668078 has been merged into this issue.
@Ryan & Team - Is there any reason why a previously affected copy of 53.0.2785.89 would have suddenly started working? I have this use-case and I'm a bit stumped as following the thread this was not expected as my understanding is  there has been no remote fix applied (the only fix being to keep on a timely build by updating).
#Comment 61

Same here, some of the websites having issues on 53.0.2785.143 started working now.
Seems to be very random for me. About 2 weeks ago I saw this once or twice, then it appeared to work normally for about a week. Today its like half of the https sites I browse have the error. 

 

If video streaming is fixed on a future release then I will update (firefox also has this streaming issue, so im assuming its not the chromium code & its not flash player as I didn’t change the version on the windows machine when checking it) for now, im kinda stuck between a rock and hard place. 

 

From: vaikoov… via monorail <monorail+v2.1666032050@chromium.org>
Reply-To: <chromium@monorail-prod.appspotmail.com>
Date: Friday, 25 November 2016 at 11:16 am
To: Daniel Morgan <morgsdc@gmail.com>
Subject: Issue 664177 in chromium: Out of date Chrome results in ERR_CERTIFICATE_TRANSPARENCY_REQUIRED for Symantec operated sites
shit 
Labels: Merge-Request-54
Going to mark this as Merge-Request for M54 as well, to ensure robust WebView handling.
Labels: -Merge-Request-54 Merge-Review-54 Hotlist-Merge-Review
[Automated comment] Request affecting a post-stable build (M54), manual review required.
For TPMs reviewing: Behaviour is already live for Finch-enabled releases for M54 (and M55 all releases). This is just making sure to explicitly enable the functionality for non-Finch enabled releases.
This will go out in M55 Stable Any Day Now(tm), but we're not doing any more M54 stable updates (modulo a massively critical security vulnerability)
Labels: -Merge-Review-54 Merge-Rejected-54
awhalley@ is correct, no more planned M54 releases.
After posting my issue to my tech friends, one shared this article: http://www.pcworld.com/article/3146718/security/chrome-bug-triggered-errors-on-websites-using-symantec-ssl-certificates.html which led me to this Issue Forum. 

still experiencing the issue on all sites with the Symantec cert after updating to v55, clearing cache and cookies, deleting the verisign G5 certificate from my keychain (On Mac 10.9.5 Mavericks), and restarting. My system time is correct and no antivirus software is running.  

Knowing at this point that it's not an actual safety error but a bug, I'm comfortable disabling it in flags...except that has been removed with the new version of Chrome. 

Halp.

This is incredibly frustrating. Does anyone have a solution that I've not tried? 
Issue 671988 has been merged into this issue.
RE comment #70: Can you provide us with a NetLog (https://dev.chromium.org/for-testers/providing-network-details) captured while reproducing the problem?

In particular, try loading https://symbeta.symantec.com/ while capturing the NetLog.

Another experiment to try is launching Chrome with this flag, and seeing if it changes the outcome:
   --force-fieldtrials=EnforceCTForProblematicRoots/disabled/
@eroman, .json file is attached. Loaded https://symbeta.symantec.com/ while capturing the NetLog as requested which kicked back the same privacy error. 

Can try launching Chrome with the flag but need a few more directions on how to do so. 
net-internals-log (1).json
153 KB View Download
Google searched how to launch with flag command line in terminal and did so - still no dice. 

Thank you for the support btw. <3
@persepsion0: Thanks for the log!

You are getting a different error, ERR_CERT_AUTHORITY_INVALID.

In comment #70 you noted:
   "deleting the verisign G5 certificate from my keychain"

I expect this is the source of your (current) problem. G5 should be the trust anchor for these chains, so if it was removed from the system trust store that would be a problem.

Let's move this to a separate bug thread; please create a new bug and email me the link, I will follow-up with you there.

Cheers.
Cc: hdodda@chromium.org
Issue 665176 has been merged into this issue.
Chromium version 53.0.2785.143 started working about a week ago (Dec 3,
2016) and is now allowing me to log into Yahoo, Wellsfargo, etc.. Looks
like the issue was resolved.
Chromium version 53.0.2785.143 Built on Ubuntu, running on Ubuntu 16.10 (64-bit)

Getting the error: NET::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED on some websites; Amazon etc.

I reinstalled without effect.
Hi,

I reproduced this problem accessing https://www.amazon.co.uk/ with Chromium 53.0.2785.143 built on Debian stretch/sid. net-internals attached.

net-internals-log.json
387 KB View Download
This issue will occur in Chromium 53 and 54. Please update to Chrome 55 which also includes a goodly number of security fixes (see https://googlechromereleases.blogspot.com/2016/12/stable-channel-update-for-desktop.html)
My Debian system just updated to Chromium 55.0.2883.75-1~deb8u1 and all is well. Much thanks to the devs who keep the world running!!
There doesn't appear to be a current PPA for Chromium stable on Ubuntu. Have I missed something?
#82 - every distribution publishes Chromium on its own pace, unfortunately. You should ask the maintainer of the Ubuntu repository, as the Chromium project does not maintain it.
Question out of curiosity related to WebView: 

Say I have a WebView-based Android app, and HTTPS fails due to this CT issue. Would it be possible to embed the affected Symantec-issue cert inside an app, create a custom TrustManager/SSLContext and make that cert trusted?

Or would WebView ignore it always / would it ignore it in this particular scenario of CT bug?
No, it wouldn't work. Beyond WebView not supporting that custom TrustManager/SSLContext, because you would be distributing a CA known to be publicly trusted by the OS (and thus subject to the same set of industry requirements to ensure certificates are not misissued), the CT policy would still apply.
#82 - I have found this testing site PPA with Chromium 55 that was able to fix this issue for me: https://launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage
Issue 673888 has been merged into this issue.
@rsleevi thanks for the answer, I have now two last questions, unrelated to the Symantec/CT topic (sorry for hijacking the thread, but it's hard to find reliable info on the topic)

Are those two statements below correct?

1. WebView uses the device CA store and nothing more; it's impossible to pass it a custom TrustManager/SSLContext/etc. and make it trust a custom cert (for example, if I wanted to embed in the app - for compatibility - a root cert that is not available in CA store of older Androids)
2. The only way to override WebView behavior with respect to SSL is to override onReceivedSslError() and call handler.proceed() if certain conditions are met

Thanks!
Please don't hijack bugs. Ask on net-dev@chromium.org
Issue 674634 has been merged into this issue.
Cc: rsleevi@chromium.org jkarlin@chromium.org
Issue 660646 has been merged into this issue.
Same here. Funny is: https://amazon.de fails, my own self built el cheapo site https://binky.tuxfriends.net/ works fine.
Heh, I'm on Chromium 56.0.2889.0 and it fails for amazon...
Of course it works on Chromium 45 (45.0.2448.0). I'll upgrade to a newer version soon to see if it was fixed, otherwise I'll look if another bug on the subject is open.
I get this error on chromium OS 55.0.2853.0 (Netflix/Amazon). It says it's up to date.
re comment 95: Please file a new bug, and include a net-internals log ( https://dev.chromium.org/for-testers/providing-network-details ). That will ensure prompt triage and diagnosing. Once you've done so, feel free to mention that bug number here with a link, just in case.
I'm also seeing this on 54.0.2840.71. @95 did you file a new bug?
Issue 676849 has been merged into this issue.
I am pretty confused here. I'd really appreciate if one of the developers could post a quick response.

53.0.2785.89 should fail to load Symantec sites 10 weeks after its build date. We are well past that now for the official build.

I was one of the people who first reported this bug and I did so as I operate a site which uses a Symantec EV certificate and was therefore affected. I was able to reproduce the issue in a separate VM with a separate install of 53.0.2785.89.

The Chrome team have consistently communicated the need to upgrade as a workaround to this issue to stay within the 10 week zone where CT works. This is completely understood but of course there are always edge cases to this and as site owners we thought it sensible to monitor that our user-base was updating in a timely fashion and we didn't have a batch of users (like myself) stuck on an affected version of Chrome.

As mentioned back in Comment 61 however, at some point this issue at least on my test machines was resolved without an update. The same test hosts with 53.0.2785.89 now work with the same site on a Symantec EV. ct_compliance_status is still "BUILD_NOT_TIMELY" but SSL_CONNECT is now working wheras before net_error -214 was flagged. 

How can this be possible?



#99 - are you sure this is the same binary as before? Was it not patched (but the version stayed intact)?
Chromium distributions sometimes issue updates without increasing the version number, because those are distribution-specific patches and not a patch that made it to the base Chromium branch of the version.
#100 - It was painful at the time to find the binary, so I kept it. ‎Signed by Google's cert 31 ‎August ‎2016 03:06:24.

On one test machine I've done a fresh reinstall in fresh VM off-network, only joining it to the network right before visiting the site it should have an issue with.

chrome://version shows:
Google Chrome	53.0.2785.89 (Official Build) m (64-bit)
Revision	a905cffb2d18baa410d0a19cc9d3941349f2d4b6-refs/branch-heads/2785@{#796}

If there is a remote patch being pushed to older clients, then that's not clear from my reading of this thread. That's why I'm stumped. 
#101 - oh, you are using Google Chrome - sorry, that was not obvious (most of the cases here were Chromium users, I think). You can disregard my comment, then.
I'm using Chromium Version 55.0.2883.87 Built on Ubuntu , running on
LinuxMint 18 (64-bit) with no issues. I've been using this version for
about a month now.
#103 - yep, totally expected, ten weeks have not passed anyway.
Labels: Restrict-AddIssueComment-EditIssue
I'm going to go ahead and close comments on this, because it's a resolved issue with a lot of stars. If you are encountering any further technical issues, please report with a new bug and include a chrome://net-internals

Regarding Comment #99: The situation for Chrome was resolved with an out-of-band update that applied once you restarted Chrome, independent of fully updating. The remainder of issues (e.g. Android WebView, Chromium distributions, other browsers based on Chromium source) required to upgrade. Upgrading Chrome itself is also the right thing to do to ensure online safety, so you/your users still should, but what changed is that the security protection was turned off for the older Chrome versions (like yours) until you upgrade.
Cc: sleevi@google.com eranm@chromium.org
Issue 682470 has been merged into this issue.
Issue 691256 has been merged into this issue.
Issue 696225 has been merged into this issue.
Sign in to add a comment