Why Certificates Lifecycle Automation Issues
Posted: Tue, 30 January 2024
| permalink
|
No comments
In the event you’ve perused the ActivityPub feed of certificates whose keys are known to be compromised, and clicked on the “Present Extra” button to see the title of the certificates issuer, you will have seen that some issuers appear to return up time and again.
This would possibly make sense – in any case, if a CA is issuing a big quantity of certificates, they’ll be seen extra usually in an inventory of compromised certificates.
In an try and see if there may be something that we are able to study from this information, although, I did a little bit of digging, and got here up with some illuminating outcomes.
I began off by discovering all of the unexpired certificates logged in Certificate Transparency (CT) logs which have a key that’s within the pwnedkeys database as having been publicly disclosed.
From this listing of certificates, I eliminated duplicates by matching up issuer/serial quantity tuples, after which lowered the set by counting the variety of distinctive certificates by their issuer.
This gave me an inventory of the issuers of those certificates, which seems a bit like this:
/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G4 /C=GB/ST=Larger Manchester/L=Salford/O=Sectigo Restricted/CN=Sectigo RSA Area Validation Safe Server CA /C=GB/ST=Larger Manchester/L=Salford/O=Sectigo Restricted/CN=Sectigo RSA Group Validation Safe Server CA /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Safe Certificates Authority - G2 /C=US/ST=Arizona/L=Scottsdale/O=Starfield Applied sciences, Inc./OU=http://certs.starfieldtech.com/repository//CN=Starfield Safe Certificates Authority - G2 /C=AT/O=ZeroSSL/CN=ZeroSSL RSA Area Safe Website CA /C=BE/O=GlobalSign nv-sa/CN=GlobalSign GCC R3 DV TLS CA 2020
Reasonably than attempt to work with uncooked issuers (as a result of, as Andrew Ayer says, The SSL Certificate Issuer Field is a Lie), I mapped these issuers to the organisations that handle them, and summed the counts for these grouped issuers collectively.

The tip results of this work is the next desk, sorted by the rely of certificates which have been compromised by exposing their non-public key:
Issuer | Compromised Depend |
---|---|
Sectigo | 170 |
ISRG (Let’s Encrypt) | 161 |
GoDaddy | 141 |
DigiCert | 81 |
GlobalSign | 46 |
Entrust | 3 |
SSL.com | 1 |
In the event you’re accustomed to the CA ecosystem, you’ll most likely recognise that the organisations with massive numbers of compromised certificates are additionally those that situation a whole lot of certificates.
To this point, nothing notably shocking, then.
Let’s look extra intently on the relationships, although, to see if we are able to get extra helpful insights.
Utilizing the issuance volume report from crt.sh, we are able to examine issuance volumes to compromise counts, to provide you with a “compromise price”.
I’m utilizing the “Unexpired Precertificates” colume from the issuance quantity report, as I really feel that’s the quantity that greatest matches the certificates inhabitants I’m inspecting to search out compromised certificates.
To keep up parity with the earlier desk, this one continues to be sorted by the rely of certificates which have been compromised.
Issuer | Issuance Quantity | Compromised Depend | Compromise Fee |
---|---|---|---|
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
ISRG (Let’s Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
Entrust | 23,166 | 3 | 1 in 7,722 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
If we now kind this desk by compromise price, we are able to see which organisations have probably the most (and least) leakiness happening from their prospects:
Issuer | Issuance Quantity | Compromised Depend | Compromise Fee |
---|---|---|---|
Entrust | 23,166 | 3 | 1 in 7,722 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
ISRG (Let’s Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
By grouping by order-of-magnitude within the compromise price, we are able to establish three “bands”:
-
The Tremendous Leakers: Clients of Entrust and GlobalSign appear to like to lose management of their non-public keys.
For Entrust, no less than, although, the small volumes concerned make the numbers considerably untrustworthy.
The three compromised certificates might very nicely belong to only one buyer, for example.
I’m not conscious of something that GlobalSign does that may make them such an outlier, both, so I’m inclined to suppose they only bought unfortunate with one or two prospects, however as CAs don’t embody buyer IDs within the certificates they situation, it’s not potential to say whether or not that’s the precise trigger or not. -
The “Common” Leakers: Clients of SSL.com, GoDaddy, and Sectigo all have compromise charges within the 1-in-hundreds-of-thousands vary.
Once more, the low volumes of SSL.com make the numbers considerably unreliable, however the different two organisations on this group have massive sufficient numbers that we are able to depend on that information pretty nicely, I feel. -
The Low Leakers: Clients of DigiCert and Let’s Encrypt are no less than thrice much less possible than prospects of the “common leakers” to lose management of their non-public keys.
Good for them!
Now we now have some helpful insights we are able to take into consideration.

The entire organisations on the listing, aside from Let’s Encrypt, are what one would possibly time period “conventional” CAs.
To a primary approximation, it’s cheap to imagine that the overwhelming majority of the shoppers of those conventional CAs most likely handle their certificates the identical method they’ve for the previous 20 years or extra.
That’s, they generate a key and CSR, add the CSR to the CA to get a certificates, then copy the cert and key… someplace.
Since people are dealing with the keys, there’s a better threat of the people utilizing both dangerous practices, or making a mistake, and exposing the non-public key to the world.
Let’s Encrypt, alternatively, points all of its certificates utilizing the ACME (Automatic Certificate Management Environment) protocol, and the entire Let’s Encrypt documentation encourages the usage of software program instruments to generate keys, situation certificates, and set up them to be used.
Provided that Let’s Encrypt has 161 compromised certificates at the moment within the wild, it’s clear that the automation in use is much from good, however the considerably decrease compromise price suggests to me that lifecycle automation no less than reduces the speed of key compromise, although it doesn’t eradicate it fully.
It’s true that all of the organisations on this evaluation additionally present ACME issuance workflows, ought to prospects want it.
Nevertheless, the “conventional CA” firms have been round rather a lot longer than ACME has, and they also most likely acquired lots of their prospects earlier than ACME existed.
Provided that it’s extremely onerous to get people to vary the way in which they do issues, as soon as they’ve a method that “works”, it appears cheap to imagine that many of the certificates issued by these CAs are dealt with in the identical human-centric, error-prone method they at all times have been.
If organisations wish to refute this assumption, although, by sharing their information on ACME vs legacy issuance charges, I’m certain we’d all be extraordinarily .
Explaining the Outlier
The distinction in presumed issuance practices would appear to elucidate the numerous distinction in compromise charges between Let’s Encrypt and the opposite organisations, if it weren’t for one outlier.
It is a largely “conventional” CA, with the manual-handling points that means, however with a compromise price near that of Let’s Encrypt.
We’re, in fact, speaking about DigiCert.
The factor about DigiCert, that doesn’t present up within the uncooked numbers from crt.sh, is that DigiCert manages the issuance of certificates for a number of of the largest “hosted TLS” suppliers, equivalent to CloudFlare and AWS.
When these providers receive a certificates from DigiCert on their buyer’s behalf, the non-public key’s stored locked away, and no human can (we hope) get entry to the non-public key.
That is supported by the truth that no certificates identifiably issued to both CloudFlare or AWS seem within the set of certificates with compromised keys.
After we ask for “all certificates issued by DigiCert”, we get each the certificates issued to those huge suppliers, that are superb at conserving their keys underneath management, in addition to the certificates issued to everybody else, whose key dealing with practices might not be fairly so stringent.
It’s potential, although not trivial, to account for certificates issued to those “hosted TLS” suppliers, as a result of the certificates they use are issued from intermediates “branded” to these firms.
With the crt.sh psql interface we are able to run this question to get the entire variety of unexpired precertificates issued to those managed providers:
SELECT SUM(sub.NUM_ISSUED[2] - sub.NUM_EXPIRED[2]) FROM ( SELECT ca.title, max(coalesce(coalesce(nullif(trim(cc.SUBORDINATE_CA_OWNER), ''), nullif(trim(cc.CA_OWNER), '')), cc.INCLUDED_CERTIFICATE_OWNER)) as OWNER, ca.NUM_ISSUED, ca.NUM_EXPIRED FROM ccadb_certificate cc, ca_certificate cac, ca WHERE cc.CERTIFICATE_ID = cac.CERTIFICATE_ID AND cac.CA_ID = ca.ID GROUP BY ca.ID ) sub WHERE sub.title ILIKE '%Amazon%' OR sub.title ILIKE '%CloudFlare%' AND sub.proprietor="DigiCert";
The quantity I get from working that question is 104,316,112, which ought to be subtracted from DigiCert’s whole issuance figures to get a extra correct view of what DigiCert’s “common” prospects do with their non-public keys.
After I do that, the compromise charges desk, sorted by the compromise price, seems like this:
Issuer | Issuance Quantity | Compromised Depend | Compromise Fee |
---|---|---|---|
Entrust | 23,166 | 3 | 1 in 7,722 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
“Common” DigiCert | 40,397,363 | 81 | 1 in 498,732 |
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
All DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
ISRG (Let’s Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
In brief, it seems that DigiCert’s common prospects are simply as possible as GoDaddy or Sectigo prospects to reveal their non-public keys.
The takeaway from all that is pretty easy, and never overly shocking, I consider.
The much less people must do with certificates issuance, the much less possible they’re to compromise that certificates by exposing the non-public key.
Whereas it might not be shocking, it’s good to have some empirical proof to again up the frequent knowledge.
Totally-managed TLS suppliers, equivalent to CloudFlare, AWS Certificates Supervisor, and no matter Azure’s factor known as, is the platonic excellent of this precept: by no means give people any alternative to reveal a personal key.
I’m not saying you ought to use one in all these suppliers, however the safety strategy they’ve adopted seems to be the optimum one, and ought to be emulated universally.
The ACME protocol is the following greatest, in that there are a number of standardised instruments broadly out there that permit people to take themselves out of the loop, nevertheless it’s nonetheless potential for people to deal with (and mistakenly expose) key materials if they fight onerous sufficient.
Legacy issuance strategies, which both can’t be automated, or require customized, per-provider automation to be developed, seem like no less than 4 instances much less useful to the purpose of avoiding compromise of the non-public key related to a certificates.
People Are, Of Course, The Downside

This statement – that should you don’t let people close to keys, they don’t get leaked – is additional supported by contemplating the largest issuers by quantity who haven’t issued any certificates whose keys have been compromised: Google Belief Companies (fourth largest issuer general, with 57,084,529 unexpired precertificates), and Microsoft Company (sixth largest issuer general, with 22,852,468 unexpired precertificates).
It seems that someplace between “most” and “mainly all” of the certificates these organisations situation are to prospects of their public clouds, and my understanding is that the keys for these certificates are managed in identical method as CloudFlare and AWS – the keys are locked away the place people can’t get to them.
It ought to, in fact, go with out saying that if a human can by no means have entry to a personal key, it makes it quite tough for a human to reveal it.
Extra broadly, in case you are constructing one thing that handles delicate or secret information, the extra you are able to do to maintain people out of the loop, the higher all the pieces can be.
In the event you’d prefer to see extra evaluation of how key compromise occurs, and the teachings we are able to study from inspecting billions of certificates, please present your help by buying me a refreshing beverage.
Trawling CT logs is thirsty work.
Within the pursuits of readability, I really feel it’s vital to explain methods by which my analysis may be flawed.
Listed below are the issues I do know of which will have impacted the accuracy, that I couldn’t feasibly account for.
-
Time Intervals: As a result of time by no means stops, there may be more likely to be some slight “mismatches” within the numbers obtained from the assorted information sources, as a result of they weren’t collected at precisely the identical second.
-
Issuer-to-Organisation Mapping: It’s potential that the way in which I mapped issuers to organisations doesn’t match precisely with how crt.sh does it, that means that counts may be skewed.
I attempted to minimise that through the use of the identical information sources (the CCADB AllCertificates report) that I consider that crt.sh makes use of for its mapping, however I can’t be sure of an ideal match. -
Unwarranted Grouping: I’ve drawn some conclusions concerning the practices of the assorted organisations based mostly on their normal strategy to certificates issuance.
If a selected subordinate CA that I’ve grouped into the dad or mum organisation is managed in some uncommon method, that may trigger my conclusions to be misguided.
I used to be in a position to pretty simply separate out CloudFlare, AWS, and Azure, however there are virtually definitely others that I didn’t spot, as a result of hoo boy there are a whole lot of intermediate CAs on the market.