An Evaluation of the Effectiveness of Chrome's CRLSets

ZDNet's Larry Seltzer: As a Google spokesperson said to me, “...[W]hen we identify a bad cert, we update our revocation list, and Chrome users are automatically protected.”

This page answers the question: Is that true?

For Google's response to this page, among others, please see the end of this page.

What is Chrome's CRLSet?A “CRL” is a Certificate Revocation List, a list of the serial numbers of unexpired security certificates which have been revoked by their issuer and should no longer be trusted. Each issuing “certificate authority” (CA) maintains and publishes its own CRL containing the current list of certificates it once signed and vouched for, but which should no longer be trusted. CRLs are in a continual state of change as old certificates expire off the lists (no longer trusted anyway due to their age) and serial numbers of newly revoked certificates are added.

The Chromium project calls its custom implementation of a CRL a “CRLSet” because it contains a carefully selected collection of revoked-certificate serial numbers published by many different certificate authorities. In other words, the CRLSet consists of a set of individually curated CA CRLs. The Chromium project defines the intent and goals for their CRLSet clearly on this page. To keep the CRLSet manageable, its size is strictly limited to 256k bytes. Consequently, an oft repeated admission by Google's engineers and spokespeople is that their CRLSet “does not contain all revoked certificates”. This page carefully examines that admission.

We believe that this examination is important because the topic of certificate revocation is quite complex. In a factual vacuum sound-bites are powerful. This creates a great deal of confusion within the industry, resulting in articles such as “Chrome does certificate revocation better”, recently written by industry veteran Larry Seltzer, published April 21st, 2014 by ZDNet. We'll come back to Larry's article, and the Google spokesperson he quotes, once we've established some context for evaluating his and Google's claims. (Larry also wrote another piece following up after this page was published. See the end of this page for more.)

What, exactly, does Chrome's CRLSet contain?In order to evaluate the real world performance of Chrome's CRLSet, we need to understand exactly what CRLSets are, and how they operate. When you reach the end of this page you will understand everything about Chrome's CRLSets.

Chrome's CRLSet file format is well documented, straightforward and easily understood by non-programmers. It is divided into two parts:

1. The first part is a short JSON-format file header identifying the file version, content type, the CRLSet's sequence number, the count of certificate authority CRL sets appearing after the header, a short list of explicitly blocked certificates, and an expiration date for the data. You can see these data fields in the box below.
2. The balance of the CRLSet consists of repeating sets of carefully curated CA CRLs. The first identifier in each set is the signature of the issuing authority's certificate whose key was used to sign each of the revoked certificates making up the balance of the set. Think of it as a parent followed by all of its children. Then a different parent and all of its children, and so on. This pattern repeats for the number of sets specified in the “NumParents” header field (which is 53 in this instance).

Below is Chrome's CRLSet #1596, the first 1,000 lines of 24,275, retrieved from the “clients2.google.com” server on April 23rd, 2014. (The entire file is available, contained within our DIY Kit referred to below.) It is displayed within a scrolling window so you may browse around to explore the file's contents.

What do these characteristics mean for the CRLSet's effectiveness?

The header indicates that the body of the file contains 53 sets of revoked certificates (NumParents:53). In other words, the entire current CRLSet contains sets of revoked certificates signed by 53 certificate authorities.

Let's look at some numbers:Apple's site provides their list of 211 root certificate authorities which are built into and trusted by Apple products, and Microsoft's site provides their listings (on four pages) of 353 root certificate authorities built into and trusted by Microsoft products.

If you are using Chrome on an Apple product, which trusts the certificates signed by 211 certificate authorities, the limitations inherent in Chrome's CRLSet blinds it to ALL of the certificates which have been revoked by 158 certificate authorities trusted by Apple's products. To rephrase that: Chrome will blindly trust every revoked certificate that was originally signed by three quarters of the certificate authorities Apple trusts.

If you are using Chrome on a Microsoft system, which trusts the certificates signed by 353 certificate authorities, the limitations inherent in Chrome's CRLSet blinds it to ALL of the certificates which have been revoked by 300 certificate authorities trusted by Microsoft's systems. To rephrase that: Chrome will blindly trust every revoked certificate that was originally signed by more than four fifths of the certificate authorities Microsoft trusts.

Now we know that Chrome's CRLSet ignores the revocation warnings published by a large majority of the Internet's certificate issuers.

How does it perform in sheer number of revoked certificates?Examining CRLSet #1596, we find 24,259 lines of certificate identifiers, one per line. Since we know that 53 of those lines are the certificate signatures of each section's “parent,” we subtract 53 from the total body line count to obtain 24,206 revoked serial numbers contained within CRLSet #1596.

But . . . compared to what?  That's the question.

In July, 2013, Websense harvested data from their “ThreatSeeker Intelligence Cloud” as described in detail in this Websense Security Labs Blog article titled “Digging Into Certificate Revocation Lists.” During approximately one hour of sampling the Internet for certificate traffic, Websense gathered 10,000 records. From those 10,000 records, 9,066 contained the URL of a certificate authority's online certificate revocation list (CRL). Within that set they found 819 unique URLs (there were many repeats).

Here is Websense's graph of the Top 30 requested CRL URLs:

Credit: Websense blog, July 10th, 2013

Websense notes that the Globalsign CRL (the lower line on the chart above) is the most popular, having been referenced almost twice as often as most of the other top 30, not to mention the other 481 URLs that didn't make it into the Top 30 list. We'll come back to Globalsign a bit later.

Websense then compiled the certificate revocation list content from 511 of the 819 URLs that responded to their queries. Given the nature of the ad hoc URL collection, and their unknown provenance, Websense was not surprised by non-responses. Other facets of Websense's research is further discussed on our OCSP Must-Staple page.

What's significant here, is that by pulling from the Internet's readily available certificate revocation lists, Websense gathered records of a total of 2,232,845 revoked certificates. This means that Chrome's CRLSet, which currently lists the serial numbers of 24,206 revoked certificates, reflects only 1.08% of the revoked certificates collected by Websense in one hour.

The 2,232,845 number should probably be regarded as a worst-case high-end count. Certificates are used and revoked for purposes other than protecting web traffic. So for the sake of argument and scrupulous fairness, lets assume that only half of the revoked certificates collected by Websense are relevant to web security. That's probably very conservative, as additional data will shortly demonstrate. But it allows us to very conservatively conclude . . .

Chrome's CRLSet contains at most 2% of
the Internet's currently revoked certificates,
leaving Chrome's users 98% unprotected.

So that we're not basing our entire case upon a single research study, here are two additional data points from Internet monitoring companies to further characterize the limited reach of Chrome's list of 24,206 revoked serial numbers:

• Websense:
  Starting in the year 2000, Websense has tracked the rate of certificate revocations. This chart clearly illustrates the trend:

  
  Credit: Websense blog, July 10th, 2013

  The apparent decline in 2013 is due to the fact that data for only the first five months of 2013 was available and shown. Websense notes that during those first five months, 766,451 certificate revocations were seen. A calculation of the average monthly revocation rate based upon those first five months will significantly underestimate the year's total, because the historical trend has clearly been increasing. But it can give us a working minimum estimate.

  766,451 revocations in five months means an average of 153,290 revocations per month through those first five months. Extrapolating that for the year, which, again, won't account for any rate increase, gives us a low-end estimate of 1,839,482 newly published certificate revocations occurring only during 2013.

  Standard web certificates have a lifetime of either two or three years (24 or 36 months) though most are issued for three years to obtain a discounted price. If we assume that revocation events are uniformly (randomly) spread across the lifetime of certificates, this means that the average revoked certificate will remain on its CRL for half its lifetime, or 18 months. Then it will be expired off the list due to age. For the sake of obtaining a scrupulously fair low-end estimate, we will ignore the effects of an accelerating revocation rate which would cause more certificates to be added every month than expire. At any given time, assuming a low-end revocation rate of 153,290 revocations per month, the Internet's collected revocation lists would contain an average of 18 months of certificate revocations, or 2,759,220 listed revoked certificates. Remember that this figure assumes a constant revocation rate based upon the observed revocation rate during the first five months of 2013. We know the real number is significantly higher.

  From this data, even after making every conservative assumption, Chrome's list of 24,206 revoked certificates, accounts for just 0.877% of the Internet's currently published non-expired and possibly fraudulent certificates.

  Update: In a communication with Netcraft engineers attempting to verify our back-of-the-envelope estimates, they wrote: “In our data, from 1800 GMT yesterday, there were 2.85M revocations in CRLs we’ve fetched after seeing them in publicly-trusted certificates. This is very close to your estimated 2.76M.” — This additional data directly confirms an aggregate global CRL size of well over two million.

• Netcraft:
  Early in the Heartbleed adventure, Netcraft, a well known and respected Internet monitoring company, wondered aloud in a posting when the expected Heartbleed “tsunami” of certificate revocations was going to show up. Most sites took longer to respond than many industry observers expected, with a great many occurring the following week. Quoting form Netcraft's post:

  Activity on certificate revocation lists peaked at a rate of 3,900 revocations per hour on the day the Heartbleed bug was announced (Monday April 7, 2014). On a typical Monday, we would expect to see a total of around 22,000-30,000 SSL certificates being revoked over the course of the day. On the Monday that the Heartbleed bug was announced to the public, there were 29,000 revocations. On the next day (Tuesday), 33,000 certificates were revoked, followed by 32,000 on Wednesday. These were both above average, suggesting that around 5,000 certificates were revoked in direct response to the Heartbleed bug each day. Note that fewer revocations usually take place over weekends.

  The crucial takeaway from Netcraft's rather matter-of-fact baseline, is that they state that on a typical Monday before the Heartbleed incident, they would expect to see between 22,000 and 30,000 certificates revoked. With most certificates having either a three year (36 month) life, we'd expect the average certificate to remain on revocation lists for half that time, 18 months or 548 days. Some will expire sooner, and some later. But Chrome's CRLSet #1596 contains a grand total of 24,206 revoked certificate serial numbers. Given Netcraft's observations, that implies that Chrome's CRLSet might be able to handle a single day's Internet certificate revocations. But given the average revoked certificate CRL presence of 548 days, Chrome can only retain approximately 1/548^th of the Internet's revoked CRLs.

  Update: In a communication with Netcraft engineers, they explained that the numbers obtained during this particular work were fetched from all of the CRLs they have access to, including some published by certificate authorities not trusted by mainstream web browsers. So this somewhat horrifying number is reduced somewhat.

The remaining question to answer is:

Which certificate authorities are
represented among the chosen 53?

Since signatures are made from hash functions which are deliberately “one-way,” it's not possible to take a signature hash and “decode it” to recover the value that was originally hashed. The only way to determine which certificate authorities are represented in Chrome's CRLSet is to take candidate CA signing certificates, generate their signature, and see whether that value matches any of the values in Chrome's CRLSet.

Let's first take a look at Globalsign:As we saw above, during the single hour of Internet traffic capture which Websense analyzed last summer (long before anyone knew about the critical vulnerability in OpenSSL and had imagined or named “Heartbleed”), references to Globalsign's CRL were made nearly twice as often as most of the other CRL's among the Top 30, to say nothing of the remaining 481 responding CRLs in the data set. In other words, as reflected by this sample . . .

Internet users are actively establishing connections secured by Globalsign's certificates much more often than any other certificate authority.

In other words, the integrity of Globalsign's certificates matters to Internet users. This makes the importance of intercepting and blocking any attempted abuse of these certificates clear.

As a consequence of the Heartbleed incident, hundreds of thousands of possibly-compromised certificates were revoked and replaced across the Internet. This chart, compiled by Netcraft, shows the regular daily cycle of Internet certificate revocations per hour in the days following the Heartbleed revelation:

Credit: Netcraft Security Blog, April 18th, 2014

It also shows, separately in green, the April 16th to 17th period during which the large web performance and security company, CloudFlare, which was vulnerable to Heartbleed as were so many sites, revoked all of its web certificates issued by Globalsign.

To obtain an estimate of the total number of CloudFlare's certificates revoked by Globalsign, we next turn to the SANS Technology Institute's certificate revocation tracker:

Credit: SANS Technology Institute, Internet Storm Center, SSL CRL Activity monitor

At the time of this writing, the SANS Internet Storm Center is monitoring the CRLs of 81 Internet certificate authorities including Globalsign's. Although this is only a subset of all trusted certificate authorities, it is perfect for our purposes, because their monitoring of fewer means that skewing of the Globalsign data will be reduced.

The two data points of interest occur on April 16th and 17th. Examining the chart, we note that the midpoint of the line connecting the data for the two days falls almost exactly across the 70,000 certs-per-day scale line. This means that the value for April 16th is as far above 70,000 as the value for the 17th is below 70,000. In other words, the average of the two days is slightly below 70,000, so the total for the two days is slightly less than twice that, or 140,000.

Therefore, we can confidently conclude that just short of 140,000 possibly-compromised and now revoked certificates were originally issued by Globalsign, the most heavily used certificate authority (based upon Websense's traffic analysis).

What does Chrome's CRLSet have
to say about that 140,000?

The Globalsign CRL in question, which was monitored by Netcraft and SANS for their reports, is http://crl.globalsign.com/gs/gsorganizationvalg2.crl.  This handy page provides the intermediate CA certificate whose private key was used to sign the Organization Validated (OV) certificates whose subsequent revocations were listed in the Globalsign OV CRL. Since this certificate is the “parent” certificate of all those revoked, its signature will introduce a set of revoked serial numbers within Chrome's CRLSet.

We generate any certificate's “signature” with the following concatenation of OpenSSL commands:

openssl x509 -pubkey -noout -in gsov.cer | openssl base64 -d | openssl dgst -sha256

We provide a complete DIY Crypto Kit to allow anyone to duplicate and verify this research and findings.

This extracts the Subject's Public Key Info (SPKI) record, converts it from base64 into binary, obtains its SHA256 hash and outputs the final result in hexadecimal, which is the format Chrome's CRLSet uses for each of its certificate set's headers. When this command is run against the Globalsign OV Intermediate certificate we obtain this signature:

Now we perform a simple text search for that string of characters in the ASCII-converted CRLSet. And it is not found. In other words . . .

Following the Heartbleed incident, none of the approximately 140,000 certificates revoked by Globalsign after the confirmed possibility that they could have been compromised, are present in Chrome's CRLSet.

But . . .

What we DID discover in Chrome's CRLSet
may surprise you. It certainly surprised us.

First, a brief bit of history to set the stage . . .Certificate revocation has not been receiving sufficient attention from the Internet community. And it seemed clear that the news of the “Heartbleed” vulnerability would result in a “tsunami” of revocations of possibly-compromised certificates. So the day after Heartbleed went public, work on GRC's revocation testing “https://revoked.grc.com” website began. It was secured by a certificate that had been revoked before the site was ever made public. Its intent was clearly stated: To allow users to test the “revocation awareness” of their various browsers, devices, and computing platforms.

As expected, the revoked.grc.com site immediately and vividly demonstrated that “the revocation problem” has been a well-kept secret among industry insiders. The Internet's many millions of web surfers have been encouraged to believe that revocation is working and that their web browsers are protecting them from fraudulent web sites. (They're not.)

Because few people understand the reality of today's broken revocation system, Chrome's users assumed that their web browser's willing display of the revoked.grc.com demonstration page must be a bug . . . and reported it as such to the Chromium developers: Issue 362696: Missing warning on revoked certificate. (For additional background, read the thread there. It's enlightening.)

And sure enough, though it took a few days, we started receiving reports that Chrome was now “properly” blocking the special revoked.grc.com test site . . . as expected.

Wanting to take a closer look, we grabbed a copy of Adam's Langley's CRLSet fetch & dump utility from his github repository. (This is how we originally obtained and created the complete CRLSet listing above.) Knowing how the CRLSet worked, we simply searched it for the serial number of the certificate of our now-blocked revoked.grc.com site . . .

0519bb07ad6f78216b1a001fa72223af

 . . . assuming, of course, that it would be there somewhere.  But it wasn't!

The serial number of GRC's “revoked.grc.com”
certificate wasn't in the CRLSet file anywhere.

How could that be? Chrome had begun blocking access to revoked.grc.com once people obtained the updated CRLSet. Adam Langley himself, answering in the dialog following the Issue 362696 bug report, wrote: “CRLSets push 1576 (I think) contains the revocation for revoked.grc.com. (See chrome://components.)” (Adam's reference to “chrome://components” is a means to have Chrome display the versions of its various components, among them the last retrieved and loaded CRLSet.)

So we had a mystery: We have Adam Langley publicly affirming that “CRLSet push 1576 contains the revocation for revoked.grc.com.” And, sure enough, after loading any CRLSet at or after #1576, Chrome knows not to allow the revoked.grc.com page to display. Yet we know how the CRLSets operate . . . and our serial number is definitely NOT listed anywhere in the file!

Was our CA, Digicert, missing the same way Globalsign was?We decided to see whether our parent's (Digicert) intermediate certificate was present as one of the headers for a set of its revoked certificates. So just as we did with the Globalsign certificate, we ran the OpenSSL incantation on the certificate for the key that signed “revoked.grc.com”:

openssl x509 -pubkey -noout -in digi.cer | openssl base64 -d | openssl dgst -sha256

It dutifully dumped out the signature of the certificate for the key that signed “revoked.grc.com”:

And, once again, we searched the CRLSet for that string . . . and it was not present in the file. That meant that no certificates signed by Digicert's High Assurance CA-3 intermediate certificate were present in the file, including ours.

But it had to be since Chrome had started blocking the site.

A bunch of us in the grc.securitynow newsgroup, who had been collaborating on this research, puzzled over this intractable mystery for a day or two. Finally, one of the brighter bulbs in the box decided to pursue a wild hunch . . . which shocked us all, but instantly solved the mystery:

“revoked.grc.com” was in the CRLSet file header.

openssl x509 ‑pubkey ‑noout ‑in rgrc.cer | openssl base64 ‑d | openssl dgst ‑binary ‑sha256 | openssl base64 ‑e

Rather than using just our certificate's serial number, our entire certificate is fed into this invocation. It extracts the Subject's Public Key Info (SPKI) record and returns a base64 encoding of it's SHA256 hash:

 . . . which is exactly what's listed as the last entry in the list of certificates to be blocked by the certificate's public key (not its serial number). You'll note that we also discovered that one other high profile recently revoked domain certificate had also recently been added into the CRLSet header: cloudflarechallenge.com

We have no idea what certs those first three “BlockedSPKI” entries are blocking, and no one (except the Chrome Devs) would ever know what those last two were, if this mystery hadn't been solved. Now we all know.

In order to block the two known high profile revoked sites, which Chrome would have otherwise continued to display, the Chromium developers added a special-case to Chrome's CRLSet. Not because Chrome's CRLSet works . . . but because it doesn't.

Let's bring it home . . .

What, exactly, do we now know?

• We know that Chrome's CRLSet currently includes certificates from only 53 different certificate authorities. ALL of the other many hundreds of CAs are publishing their own CRLs of revoked certificates which Chrome will blindly trust.
• We know that Chrome's CRLSet includes at most 2% of the revoked certificates currently published by the Internet's certificate authorities. Chrome will blindly trust the remaining 98% of the Internet's revoked and not-yet-expired certificates. And, of course, new revocations are being published every day, 98% of which Chrome will never be aware of.
• We know that Chrome is blind to every one of the approximately 140,000 possibly-compromised certificates recently revoked by Globalsign on behalf of CloudFlare. Chrome trusts them all.
• We know that the appearance of the entirely benign revoked.grc.com test, and news that the certificate for the cloudflarechallenge.com website had been revoked, threatened to reveal that Chrome would not automatically block those revoked certificates. (It's not a bug . . . it's a feature.)
• We know that the Chromium developers (secretly) know all of the above, and that they don't rely upon their own CRLSet system to block sites that users are aware of and expect to be blocked.
• We know that when they need embarrassing certificates to be blocked, they make special cases of them, adding them manually, as static entries, into the header of Chrome's CRLSet, because that's the only thing that works.

Thus, by any reasonable definition of “broken” . . .

• We know that Chrome's CRLSet certificate revocation system is completely broken. It cannot, and should not, be relied upon to protect the security interests of its users.

“ Chrome does certificate revocation better ”

You'll recall that near the top of this page we referred to Larry Seltzer's recent ZDNet column with that title. We now have sufficient context, truth and fact to evaluate Larry's conclusion. In his column, Larry quotes an unnamed Google spokesperson saying:

As a Google spokesperson said to me, “...[W]hen we identify a bad cert, we update our revocation list, and Chrome users are automatically protected.”

Somebody, somewhere, has been drinking some serious Google Kool-Aid.

As another point of interest, it's worth noting that in practice, updates to Chrome's CRLSet appear to often take several days and may be unreliable. In the online dialog following the developers' addition of the cloudflarechallenge.com and revoked.grc.com certificates, participants can be seen continuing to report that the sites were not being blocked. They were jumping through hoops, rebooting and restarting their browsers, forcing update refreshes, and doing everything they could think of to get the blocking to work. The actual process seemed to go a bit less smoothly in practice than in theory.

Larry then goes on to quote Adam Langley:

“That's why I claim that online revocation checking is useless — because it doesn't stop attacks. Turning it on does nothing but slow things down. You can tell when something is security theater because you need some absurdly specific situation in order for it to be useful.”

Let's address Adam's last sentence first: If anything has ever been deserving of the title “security theater” it's a certificate revocation detection system that is completely blind to all of the certificates revoked by 4/5ths of the Internet's certificate authorities, at least 98% of revoked certificates overall, doesn't even attempt to deal with the consequences of major Internet events such as Heartbleed, and must have known-revoked certificates, which would be embarrassing to not block, manually added by its maintainers.  As we all now know, that describes Chrome's CRLSet to a tee. “Security theater” . . . by design.

What about Adam's first assertion?:“That's why I claim that online revocation checking is useless — because it doesn't stop attacks. Turning it on does nothing but slow things down.”

Unfortunately, in our view, Adam is allowing the perfect to be the enemy of the good. He is absolutely correct in asserting that using most of today's technology there are situations in which online revocation checking can fail and/or be blocked by a determined and properly positioned attacker. But we believe he's wrong to completely ignore the many situations in which it cannot be blocked and is guaranteed to succeed in protecting its user from online fraud.

One example is Firefox's available option to refuse any secure connection whose certificate cannot be positively verified as valid. Although this can result in Firefox blocking when it should not, it provides extremely good protection to users who have enabled it. (We have always had it enabled and have never received a single false-positive block.) This page is not the place for a detailed pro and con discussion of this issue, but we do have such a page: For a thorough discussion, including the examination of worst-case attack scenarios, please see our OCSP Must-Staple page.

Given Adam's continual assertions (here and here and here) that “revocation checking is useless” one quick example of certificate revocation working valuably and perfectly is in order. For brevity and clarity, and because we haven't discussed the newer OCSP protocol here, we'll look at the case of classic CRL lists which have been supported for decades:

• A typical Internet user visits secured sites such as Amazon, eBay, PayPal, their bank, and so on. With revocation checking enabled, every time they connect their web browser will query the CRL pointed to by each institution's security certificate to verify that the certificate is NOT currently listed as revoked. CRL's are typically updated by their issuing authorities daily, with a week-long expiration, so the most recently obtained CRL covering a given certificate will be retained (cached) by the browser and/or operating system and updated as needed.
• Then, at some point, something happens to cause one of the secured institutions to worry that their certificate might have been compromised. So they ask their CA to revoke and replace their no-longer-trusted certificate. The replacement will be from the same CA, of the same class and have the same intermediate signing certificate, so it will almost certainly be covered by the same CRL.
• Now imagine that the institution was correct, and that bad guys got control of the revoked certificate. At some point before the revoked certificate expires they create a fraudulent clone of the target institution's website and arrange, perhaps through internet router, DNS poisoning, or traffic hijack, to divert the institution's traffic to their fake site.
• If revocation checking is enabled, before accepting the connection the user's operating system and/or web browser will examine the pending connection's certificate to find its CRL. Since this will likely be the same CRL listed for the institution's replaced and non-revoked certificate, which they've been using, they probably already have a locally cached copy which lists the previously revoked certificate as such . . . thus absolutely preventing them from trusting the fraudulent website. They are protected.

In the entirely feasible real world scenario described above, you'll note that there was absolutely nothing any attacker could do at the time of the attack to block the browser's awareness of the revoked certificate. Thanks to background CRL fetching and caching, the user's device already knew of the certificate's revoked status. And note that since no additional Internet queries and retrievals were required, in this instance nothing was slowed down . . . except for the attacker, who was stopped cold.

It is unfortunate that anyone would advise everyone to disable this long-standing feature of Internet security using the argument that there are some scenarios in which it can be defeated. No one argues that. But that certainly doesn't render it useless, and it appears to be far more generally useful than Google's own CRLSet which presumes to replace it.

The system known as “OCSP Must-Staple” is
the solution . . . and we're close to having it.

With OCSP Must-Staple, the same web server that serves the security certificate also “staples” to that certificate a freshly signed assertion from the issuing certificate authority attesting to the certificate's current status. The web browser doesn't need to make any additional fetches, so no delays or performance penalty, ever. The web server periodically obtains a fresh OCSP assertion from its certificate's authority and provides that to every browser.

“Stapling” already exists and it's supported by all major web servers (Apache, nginx, LiteSpeed, IIS and others):

• Microsoft IIS: enabled in all web servers by default for years.
• Apache: https://httpd.apache.org/docs/current/mod/mod_ssl.html#sslusestapling
• nginx: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_stapling

Firefox has supported stapling since version 26 and Microsoft with Internet Explorer has been well ahead of the curve.

The final piece of the puzzle is the enforcement of stapling, known is must-staple, and we're close to having that too.

Let's get this fixed!

1. If you haven't already, please read and absorb our The Case for OCSP Must-Staple page. You'll come away with a complete and comprehensive understanding and appreciation of the whole problem.
2. You'll learn and understand why “OCSP Must Staple” is where we need to be.
3. Then use Digicert's excellent help page to check whether the various websites you need to trust currently have OCSP Stapling enabled. (GRC does, Google and most others do not). All major web servers have supported OCSP stapling for some time, but with the exception of Microsoft's servers which enabled it by default, it is disabled by default and most have not seen any need to turn it on. Help them to see the need. Let them know you need them to care more about their (and your!) security.
4. It appears that Firefox will be the first browser to offer support for server response-header “Must Staple”—they're very close and the Mozilla guys “get it.” Internet Explorer has all of the plumbing and foundation required to quickly follow. So until Google catches up, either of those two browsers will likely offer the best protection against Internet certificate abuse. There are guys at Google who also “get it” and who know what needs to be done. They're just not as close to being there as they would like to be today.

Note: Our next and final page in this series will describe and characterize the current “revocation awareness” state of the industry's various OS and web browser platforms . . . stay tuned!

News, Updates, Additions, Errata & Corrections

• Google's Adam Langley's blog response to this page: Revocation still doesn't work.
• The official Certificate Authority (CA) Security Council industry group has also weighed-in on the issue of Google's claims and the proven CRLSet ineffectiveness.
• The Google Chromium team have responded by disabling all external certificate revocation checking (CRL and OCSP) and then removing the checkbox to optionally allow the user to turn it on, using the justification that it confuses users. (Even though it was already well hidden under the Advanced Settings. See near the bottom of this “bug” page, and the comments in this code commit page: “Remove the confusing certificate revocation checkbox.”
• ArsTechnica's coverage and interpretation: Google Chrome protection for Heartbleed-hacked sites called “completely broken”
• SlashGear's coverage offers a sane and coherent conclusion: What we can gauge from this is that a security standard — a real standard — must be arrived at soon. The issue with CRLSet revolves around transport layer security (TLS), and if one of the major browsers is only identifying 2% of compromised sites, that’s just plain inadequate. When it comes to security, there simply shouldn’t be disparate parties solving things their own way. There needs to be a call to arms, and a dissolving of egos.
• ZDNet's Larry Seltzer followed up his earlier piece, quoted above, with: “Certificate revocation controversy heats up” Within his column, this part came as welcome news:

  After my last article praising CRLSets I was approached by the CA Security Council, a CA industry group. They made some of the same arguments that Gibson makes about CRLSets, but they didn't try very hard to defend the current regime.I did ask them about what could work in the long term, and we agreed that a combination of a new standard called OCSP Must Staple and hard failing would be effective, particularly in combination with short OCSP validity periods.

  Yep. That's exactly where we need to go. See: The case for OCSP Must-Staple.
• Softpedia covers this issue with: GRC: Google Chrome Only Blocks 3% of Sites Compromised by Heartbleed