Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
After upgrading the operating system of a monitoring server, the nagios-nrpe-plugin package (and therefore the check_nrpe plugin) was also updated. The current NRPE version on Ubuntu 20.04 (Focal) is 4.0.0:
root@focal:~# dpkg -l|grep nrpe
ii nagios-nrpe-plugin 4.0.0-2ubuntu1 amd64 Nagios Remote Plugin Executor Plugin
ii nagios-nrpe-server 4.0.0-2ubuntu1 amd64 Nagios Remote Plugin Executor Server
Shortly after the upgrade Windows hosts, which are monitored using NSClient++ (nscp), started to alert. It seems that the communication between check_nrpe and NSClient++ stopped working:
This can also be confirmed on the command line:
root@focal:~# /usr/lib/nagios/plugins/check_nrpe -H 10.10.22.23
CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 10.10.22.23: 1
Even forcing the older NRPE protocol (v2) results in the same error:
root@focal:~# /usr/lib/nagios/plugins/check_nrpe -H 10.10.22.23 -2
CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 10.10.22.23: 1
The communication however still works with the old check_nrpe version (2.15), which was restored as nrpe2:
root@focal:~# /usr/lib/nagios/plugins/check_nrpe2 --help | head
NRPE Plugin for Nagios
Copyright (c) 1999-2008 Ethan Galstad (nagios@nagios.org)
Version: 2.15
Last Modified: 09-06-2013
License: GPL v2 with exemptions (-l for more info)
SSL/TLS Available: Anonymous DH Mode, OpenSSL 0.9.6 or higher required
Usage: check_nrpe -H <host> [ -b <bindaddr> ] [-4] [-6] [-n] [-u] [-p <port>] [-t <timeout>] [-c <command>] [-a <arglist...>]
root@focal:~# /usr/lib/nagios/plugins/check_nrpe2 -H 10.10.22.23
I (0.4.4.19 2015-12-08) seem to be doing fine...
So what did change in between?
NRPE itself staid at version 2.15 for many many years. This introduced a lot of weaknesses and security concerns. In 2018 NRPE was finally picked up again by Nagios Enterprises and until this year, 2021, new NRPE versions were released.
The major changes between the NRPE 2.x and the newer NRPE 3.x and 4.x concern the TLS/SSL settings. This also added a newer DH (Diffie-Hellman) key with a larger key size. From the README:
Running ./configure will now create a 2048-bit DH key instead of the old 512-bit key.
But how does that affect NSClient++?
Thanks to this excellent blog post by Milan Kozak, one can find out that NSClient++ actually uses a smaller DH key. This can be seen inside the NSClient++ installation path, in the "security" folder:
NSClient++ itself uses an internal nrpe_dh_512.pem DH key, obviously still referring to the old NRPE version.
To be able to communicate with the newer check_nrpe version, a new DH key with a size of 2048 bytes (same as the check_nrpe plugin itself) needs to be created.
The new DH key can easily be created on the monitoring server using the openssl command:
root@focal:~# openssl dhparam -C 2048 2> /dev/null|sed -n '/BEGIN/,/END/p'
-----BEGIN DH PARAMETERS-----
MIIBXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXwIBAg==
-----END DH PARAMETERS-----
Note: This command might take a few minutes. On a busy machine this will be generated faster.
The final output can be copied (including the BEGIN and END lines) and saved into a new file on the Windows server: C:\Program Files\NSClient++\security\nrpe_dh_2048.pem.
Now the configuration of NSClient++ (usually C:\Program Files\NSClient++\nsclient.ini) needs to be adjusted. Inside the NRPE server settings, add the parameter for the dh key:
; Section for NRPE (NRPEServer.dll) (check_nrpe) protocol options.
[/settings/NRPE/server]
; ENABLE SSL ENCRYPTION - This option controls if SSL should be enabled
use ssl = 1
; COMMAND ALLOW NASTY META CHARS - This option determines whether or not that we will allow clients to specify nasty (as in |`&><'"\[]{}) characters in arguments.
allow nasty characters = 1
; COMMAND ARGUMENT PROCESSING - This option determines whether or not that we will allow clients to specify arguments to commands that are executed.
allow arguments = 1
; PORT NUMBER - Port to use for NRPE.
port = 5666
; EXTENDED RESPONSE - Send more than 1 return packet to allow response to go beyond payload (requires modified client, if legacy is true this defaults to false).
extended response = 1
; # DH KEY
dh = ${certificate-path}/nrpe_dh_2048.pem
The ${certificate-path} variable is an internal NSClient++ variable which points to the security folder.
Now restart the NSClient++ service in services.msc:
Again Kudos to Milan Kozak for finding and sharing this solution!
As soon as NSClient++ was re-configured to use the larger DH key and the service restarted, the check_nrpe plugin was able to communicate with this Windows host again.
However another warning started to show up:
root@focal:~# /usr/lib/nagios/plugins/check_nrpe -H 10.10.22.23
CHECK_NRPE: Invalid packet version received from server.
I (0.5.2.35 2018-01-28) seem to be doing fine...
The newer check_nrpe plugin uses the newer NRPE v3 protocol - but it seems that the NSClient++ agent answers with the older NRPE v2 protocol. To handle this, simply tell the check_nrpe plugin to use the older v2 protocol using the optional -2 parameter:
root@focal:~# /usr/lib/nagios/plugins/check_nrpe -H 10.10.22.23 -2
I (0.5.2.35 2018-01-28) seem to be doing fine...