Lets Encrypt Root CA expiry (server certificate verification failed): Make sure to remove DST Root CA X3!

Looking closer at the last certificate in this chain, CN = ISRG Root X1, reveals that it is issued by DST Root CA X3:

ck@mintp ~ $ openssl x509 -text -in /tmp/ISRG_Root_X1.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject Public Key Info:
[...]

The last step of certificate validation always happens on the local machine (the client) where the last issuer (DST Root CA X3 in this case) needs to be installed locally. Does this "DST Root CA X3" exist on the local machine?

ck@mintp /etc/ssl/certs $ ll|grep -i DST
lrwxrwxrwx 1 root root     18 Feb  4  2021 12d55845.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     18 Feb  4  2021 2e5ac55d.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     53 May 26  2018 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

Yes, it exists. If it was missing, this could have been the reason why the chain was not trusted. But let's take a closer look at this one, too:

ck@mintp /etc/ssl/certs $ openssl x509 -text -in DST_Root_CA_X3.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
[...]

This Root CA certificate has expired (Validity Not After: Sep 30 14:01:15 2021 GMT)!

DST Root CA X3 has expired

While researching information on this particular Root CA, a blog post from Let's Encrypt, last updated on September 30th 2021, turned up:

Let’s Encrypt has a “root certificate” called ISRG Root X1. Modern browsers and devices trust the Let’s Encrypt certificate installed on your website because they include ISRG Root X1 in their list of root certificates. To make sure the certificates we issue are trusted on older devices, we also have a “cross-signature” from an older root certificate: DST Root CA X3.
[...]
DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates.
[...]
If you provide an API or have to support IoT devices, you’ll need to make sure of two things: (1) all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and (2) if clients of your API are using OpenSSL, they must use version 1.1.0 or later.

To sum that up: The ISRG Root X1 CA certificate needs to be installed on the system. But it turns out, that this is already the case:

ck@mintp /etc/ssl/certs $ ll|grep -i ISRG
lrwxrwxrwx 1 root root     16 Feb  4  2021 4042bcee.0 -> ISRG_Root_X1.pem
lrwxrwxrwx 1 root root     16 Feb  4  2021 6187b673.0 -> ISRG_Root_X1.pem
lrwxrwxrwx 1 root root     51 May 26  2018 ISRG_Root_X1.pem -> /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

Taking a closer look at how the trust chain looks like (using SSLLabs), there are two certificate paths detected:

As the ISRG Root X1 still contains the information this it was issued by the now expired DST Root CA X3, this trust path is checked (until the end of the chain) and then fails. So the fault is really that the chain was not updated during the automatic renewal of the Let's Encrypt certificates. This is also the case for certificates still renewed in September and October. In our case, the Let's Encrypt certificate of gitlab.infiniroot.net was automatically renewed on September 25th 2021 - yet it still contains the chain leading up to DST Root CA X3, just a few days before this Root CA would expire.

Update the local certificates using system updates

Luckily the browsers automatically detect and handle the different chain paths, but how can this be solved now for CLI commands, such as git?  According to the ca-certificates package, deleting the "DST Root CA X3" certificate should be enough. This can be seen in the changelog of the latest ca-certificates package update:

After updating the package, the machine was restarted (a session logout should also do the job) and git pull now worked again, without any certificate warning:

ck@mintp ~/Git/infiniroot/ansible $ git pull
Username for 'https://gitlab.infiniroot.net': 

Manually updating the local certificates

Older Operating Systems or Linux distributions have not yet received an update of the ca-certificates - or never will. On Debian Stretch the current ca-certificates package hasn't been updated in a while:

root@stretch ~ # dpkg -l|grep ca-cert
ii  ca-certificates     20200601~deb9u2       all          Common CA certificates

On these systems, the Let's Encrypt problem can be reproduced by using curl on a known domain using Let's Encrypt:

root@stretch ~ # curl https://gitlab.infiniroot.net
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Looking at the ca-certificates, the expired DST Root CA X3 is still on the system:

root@stretch /etc/ssl/certs # ll|grep -i DST
lrwxrwxrwx 1 root root     18 Jun 10  2020 12d55845.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     18 Jun 10  2020 2e5ac55d.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     53 Mar 15  2016 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

This certificate is also inside the ca-certificates.crt file, which is actually the one the system (openssl) reads:

root@stretch ~ # awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep DST
subject=O = Digital Signature Trust Co., CN = DST Root CA X3

The configuration of ca-certificates is located at /etc/ca-certificates.conf and holds this information as well:

root@stretch ~ # grep DST /etc/ca-certificates.conf
!mozilla/DST_ACES_CA_X6.crt
mozilla/DST_Root_CA_X3.crt

To manually remove the certificate, append a "!" in front of mozilla/DST_Root_CA_X3.crt, followed by running the update-ca-certificates command:

root@stretch ~ # vi /etc/ca-certificates.conf

root@stretch ~ # grep DST /etc/ca-certificates.conf
!mozilla/DST_ACES_CA_X6.crt
!mozilla/DST_Root_CA_X3.crt

root@stretch ~ # update-ca-certificates
Updating certificates in /etc/ssl/certs...
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.bundle.crt.20200602
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.bundle.crt.20200602
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.crt
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.crt
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

The output says it: 1 certificate was removed. And the ca-certificates.crt is updated:

root@stretch ~ # ll /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 197216 Oct  7 09:15 /etc/ssl/certs/ca-certificates.crt

Let's use the awk command from above to verify that the DST Root CA X3 is really gone:

root@stretch ~ # awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep DST
root@stretch ~ #

Yep, it's gone. Does the curl command work now?

root@stretch ~ # curl https://gitlab.infiniroot.net
<html><body>You are being <a href="http://gitlab.infiniroot.net/users/sign_in">redirected</a>.</body></html>

And yes, curl on our gitlab.infiniroot.net server works again!

A global problem

There are two major problems with this Root CA expiration from Let's Encrypt:

1. In order to allow older Android devices to still work with Let's Encrypt certificates, LE still continues with the "alternative" trust chain, referring to the expired DST Root CA X3 as issuer. This alternative chain is seen (mainly) in command line tools and libraries and (may) leads to the server certificate verification failure.
   
   
2. Due to problem 1, every affected client (and there are a lot!) will have to manually update and remove the DST Root CA X3 certificate from its own list of certificates. Either by updating the system packages (ca-certificates in Linux) or by manually removing the DST Root CA X3 certificate (see above). As this is something to do/fix on the client side, this has a global effect and server administrators can nothing do to fix this - it needs to be fixed on the client side.