Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
Monitoring of a certificate's expiration date is something very common (or at least should be common) in an organisation. Although an expired certificate will not cut the secure transmission between client and server, it is nasty to see a security related warning in a browser.
A simple way how this can be achieved is by using the check_http monitoring plugin, which is part of the monitoring-plugins or nagios-plugins package:
$ /usr/lib/nagios/plugins/check_http -H www.infiniroot.com -S --sni -C 14
OK - Certificate 'www.infiniroot.com' will expire on Sat 01 Aug 2020 05:10:00 PM CEST.
Using the -C parameter enables the SSL/TLS certficate check inside check_http. The number following -C indicates the number of days a certificate should still be valid. Anything below that number (e.g. 13 days) will result in a warning.
So far so good, the server certificate is being monitored and a notification is sent in advance before the certificate expires. But what about the other certificates in the chain; the intermediate and Root CA certificates?
To understand why the chain is important, let's back up a little and explain how certificates are created.
A server certificate is issued by a so-called certificate issuer. This issuer can be a certificate reseller such as Gandi, Namecheap, Digicert, LetsEncrypt, etc. These issuers basically create your server certificate signed by their own certificate. You can even create your own certificate which will then sign your certificate (however it will not be shown as trusted unless your signing certificate is added to a trust store). Such a "signing certificate" is called the CA (Certificate Authority).
Think of it this way: In order to proof you're able to drive a car, you need a driver's license. Your driver's license is issued by the responsible government department, e.g. department of transportation, after a successful driving exam. Would you be able to drive a car without a driver's license? Of course. But is it secure, especially for passengers? Arguably not - unless you have your driver's license, which proves your ability to drive a car.
In the example above, the issuer creates the server certificate. This means there are two certificates: The CA certificate from the issuer and the server certificate. But in most common cases, the chain consists of three (sometimes even more) certificates.
Most certificate issuers use themselves a certificate issued by a top-level issuer, called Root CA. The issuers of the server certificates are therefore sitting "in the middle" between the Root CA and the server certificate. That's why the issuer certificates are called Intermediate CA or ICA certificates. A certificate chain consisting of three certificates therefore looks like this:
The Root CA's certificate plays a very special role in the whole trust chain. Not only is it at the top of the chain, having issued the intermediate certificate which in turn issued the server certificate, it also is "known" to the client accessing the web site or application. The most used Root CA certificates are already installed in browsers and operating systems. This can be manually verified. In Firefox, open Firefox's preferences, select Privacy&Security and scroll all the way down to "View Certificates...". The tab "Authorities" in the certificate manager shows the already installed Root CA certificates:
Other browsers have similar ways to show the installed certificates.
Note: Certificates can also be installed inside the operating system. Browsers (such as Internet Explorer / Edge) may use the certificates from the OS without having its own trust stores embedded in the browser (such as Firefox' Certificate Manager).
A browser deems a server certificate trustworthy as long as the chain is known (server certificate is issued by this ICA and this ICA is issued by this Root CA) and the Root CA certificate is installed/known to the browser. This is why at least the ICA certificate should be deployed alongside the server certificate. Usually it doesn't hurt to install the Root CA certificate, too. To be able to see the chain without using openssl commands, Qualys' SSL Labs have created an excellent SSL Testing tool which is highly recommended - not just to check the certificate chain but also for security related SSL/TLS settings.
But what if the Intermediate or even the Root CA certificate expires?
Due to the nature of being at the "top of the chain", Root CA certificates are valid for a long time (usually over 10 years). This is to ensure that users have enough time to install browser or OS updates and eventually install newer Root CA certificates through the updates. But it is also to ensure that a newly created server certificate will expire before the Intermediate and Root CA certificates and must be renewed before. If either one (ICA or Root CA) is getting close to being expired, the new server certificate will be issued with an already newer ICA certficate, and so on...
But humans make errors. Server administrators may install an outdated or wrong ICA or Root CA or certificate issuers themselves can also point to the wrong Root CA or send a wrong bundle. This happened very recently, when the widely used "AddTrust Root CA" certificate expired on May 30th 2020. A wildcard certificate, issued in 2019 and valid until 2021, contained this AddTrust Root CA in the certificate bundle, sent by the issuing company.
On May 30th 2020, right after the AddTrust Root CA certificate expired, communication between applications stopped working. A good example can be seen by a mail client (Roundcube), accessing the SMTP server using a secure (TLS) connection on port 587:
30-May-2020 14:33:05 UTC] ERROR: STARTTLS failed ()
[30-May-2020 14:33:05 UTC] ERROR: Invalid response code received from server (-1)
[30-May-2020 14:33:05 UTC] ERROR: Failed to write to socket: unknown error ()
[30-May-2020 14:33:05 +0000]:
The same also happened on the (deprecated) SSL connection on port 465. But in this setup, additional information about an expired certificate was logged:
[30-May-2020 14:22:12 UTC] PHP Warning: stream_socket_client(): SSL operation failed with code 1. OpenSSL Error messages:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed in /usr/share/psa-roundcube/vendor/pear/net_socket/Net/Socket.php on line 159
[30-May-2020 14:22:12 UTC] PHP Warning: stream_socket_client(): Failed to enable crypto in /usr/share/psa-roundcube/vendor/pear/net_socket/Net/Socket.php on line 159
[30-May-2020 14:22:12 UTC] PHP Warning: stream_socket_client(): unable to connect to ssl://localhost:465 (Unknown error) in /usr/share/psa-roundcube/vendor/pear/net_socket/Net/Socket.php on line 159
[30-May-2020 14:22:12 UTC] ERROR: stream_socket_client(): unable to connect to ssl://localhost:465 (Unknown error) (0)
[30-May-2020 14:22:12 UTC] ERROR: Failed to connect socket: stream_socket_client(): unable to connect to ssl://localhost:465 (Unknown error) ()
[30-May-2020 14:22:12 +0000]:
Yet in monitoring did not see any issues. The server certificate was still valid and SMTP seemed to work, too. The meanwhile expired Root CA certificate, which was part of the installed bundle, caused the application's code to throw an error and stop working (correctly though).
And this is why certificate monitoring of the whole chain is important.
Since version 1.115.0, the monitoring plugin check_ssl_cert is able to validate all the certificates sent by the server. Version 1.117.0 contains a bugfix, to correctly output the expiry date of the expired certificate(s) in the chain, so this version and newer should be used.
$ /usr/lib/nagios/plugins/check_ssl_cert -H www.infiniroot.com
SSL_CERT OK - x509 certificate 'www.infiniroot.com' from 'Let's Encrypt Authority X3' valid until Aug 1 16:10:26 2020 GMT (expires in 52 days)|days_chain_elem1=52;20;15;; days_chain_elem2=280;20;15;;
The plugin's output not only shows the name of the server certificate issuer (Let's Encrypt Authority X3) but also the expiry date of the server certificate. In the performance data part of the output (following the pipe character), the days until certificate expiry of two certificates (server and ICA) are shown: 52 days left for the server certificate, 280 days left for the ICA certificate.
Note: Append --sni parameter followed by the domain name for virtual hosts.
On a server, where the third certificate (installed Root CA) expired, the output looks like this:
$ ./check_ssl_cert -H www.example.com
SSL_CERT CRITICAL www.example.com: x509 certificate element 3 is expired (was valid until May 30 10:48:38 2020 GMT)|days_chain_elem1=281;20;15;; days_chain_elem2=3857;20;15;; days_chain_elem3=-10;20;15;; days_chain_elem4=-10;20;15;;
The plugin clearly shows that certificate element 3 (the Root CA in this case) has expired. And finally: This is now showing up in monitoring (here Icingaweb2 using Icinga2):
Although Intermediate or Root CA certificates have a longer lifetime than server certificates, they should be monitored nevertheless. It is also possible, that an ICA or Root CA certificate could become compromised in the future and would therefore be revoked (which check_ssl_cert is also able to monitor, too). By switching from check_http to check_ssl_cert for certificate monitoring, the whole chain is constantly monitored, giving more insights and better control (and more time to react in case of an alert).