Bad Certificate Revocation: DigiCert Offers Temporary Pause
Citing 'Critical Infrastructure' Problems, Certificate Authority Offers 3-Day Delay
DigiCert, the world's largest certificate authority, said it will temporarily pause a planned mass revocation of digital certificates that failed to comply with validation rules.
See Also: Webinar | Identity Crisis: How to Combat Session Hijacking and Credential Theft with MDR
The company on Monday announced plans to revoke within 24 hours certificates that did not have proper Domain Control Verification. "We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately," it said.
Following a customer outcry over the snap revocations, the Utah company backtracked slightly on Tuesday. DigiCert certificates secure 28 billion web connections and underpin 40 billion DNS queries, according to its website.
"Some customers operating critical infrastructure are not in a position to have all their certificates reissued and deployed in time without critical service interruptions," it said.
To avoid disrupting these services, the company said it was working with "browser representatives" to buy time for customers and offered a deadline of 19:30 UTC Wednesday to request a revocation delay. DigiCert said that for any customer that hasn't filed such a request, "we will assume your certificates have been replaced and revoke them," possibly with immediate effect.
Any organization requesting a delay must specify what "exceptional circumstances" require such a delay and when it plans to have the reissued certificates in place.
DigiCert said a delay request will only extend matters until 19:30 UTC Saturday, at which point it plans to revoke any still-outstanding certificates
Not all DigiCert customers have responded by immediately reissuing their certificates or saying when they plan to do so. On Tuesday, benefit funding and payment solutions SaaS provider Alegeus Technologies filed a complaint in U.S. District Court in Utah, seeking a temporary restraining order prohibiting DigiCert from invalidating its certificates.
U.S. District Judge Howard C. Nielson granted the company's request. "DigiCert is prohibited from revoking the security certificates for the Alegeus Websites for a period of seven days, or until the court is able to schedule a hearing on the motion, whichever is earlier," he said.
Domain Validation Errors
The underlying problem stems from DigiCert failing to correctly validate ownership for domains owned or controlled by some customers.
Customers can prove ownership in multiple ways. One technique involves them adding a random, DigiCert-provided value to their domain's DNS CNAME records. DNS CNAME - for canonical name - maps a domain name to one or more aliases.
After doing a DNS lookup and confirming the random value in the DNS CNAME record, DigiCert can confirm that the customer does in fact own or operate the domain.
This type of validation includes this crucial requirement: The random value must be preceded by an underscore character. "The underscore prefix ensures that the random value cannot collide with an actual domain name that uses the same random value," DigiCert said. "While the odds of that happening are practically negligible, the validation is still deemed as non-compliant if it does not include the underscore prefix."
In a root cause analysis, DigiCert said it recently learned that things had gone wrong. The company said that beginning around August 2019, it transitioned from a legacy system that functioned correctly to a new service that only added the underscore character in some cases. This persisted until June 11, when the company inadvertently fixed the problem when its engineering team "completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service."
All certificates issued since then comply with the CABF's rules.
Several week ago, the company received an email to its problem-reporting channel "asking about random values used in validation," it said. After undertaking multiple reviews, "DigiCert discovered an issue regarding the underscore prefix for random values," it said, and "then initiated this incident management process."
The company said the error has affected "approximately 0.4% of the applicable domain validations we have in effect" and that CA/Browser Forum rules require that erroneously issued certificates - such as those that have a domain validation problem - "must be revoked within 24 hours, without exception." In an emailed response, the company said approximately 6,800 customers must reissue 83,267 certificates.
Failing to follow CA/Browser Forum rules can have serious consequences, including leading to a certificate authority and all certificates it issued becoming untrusted.
Multiple IT administrators took to social media to decry the short time frame in which they've been forced to deal with DigiCert's mistake.
"I just got an email from DigiCert stating that they are going to invalidate all of my certificates within 24 hours," one system administrator posted to Reddit on Monday, saying they renewed all of their organization's 100 certificates less than a month ago and had to cancel their planned time off to do so again.
"I'm tired of fixing other people's f***-ups," the admin said (see: CrowdStrike, Microsoft Outage Uncovers Big Resiliency Issues).
