Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
This week a critical security vulnerability, dubbed "IngressNightmare", was published. This includes multiple CVE-assigned vulnerabilities which affect the Kubernetes Ingress Nginx controller. The worst one, CVE-2025-1974, has a very high CVE score of 9.8 (10.0 is the maximum).
Nginx is likely the most widely used Ingress controller in Kubernetes clusters. Hence the vulnerability has a large impact on Kubernetes clusters worldwide.
While the main function of the Nginx Ingress (handling incoming HTTP traffic, routing to the relevant application/service) is not problematic, the security vulnerability affects the admission controller part, used for validating Ingress configuration changes.
From the security researches at WIZ:
When the Ingress-NGINX admission controller processes an incoming ingress object, it constructs an NGINX configuration from it and then validates it using the NGINX binary. Our team found a vulnerability in this phase that allows injecting an arbitrary NGINX configuration remotely, by sending a malicious ingress object directly to the admission controller through the network.
During the configuration validation phase, the injected NGINX configuration causes the NGINX validator to execute code, allowing remote code execution (RCE) on the Ingress NGINX Controller’s pod.
In Rancher managed Kubernetes clusters, created with RKE1 or RKE2, the "ingress-nginx-controller" deployment in the "ingress-nginx" namespace is by default deployed as Ingress for the whole cluster.
While a lot of security blogs and articles quickly covered this security issue, an official response from SUSE for Rancher Kubernetes (RKE) or in general Rancher as product took time. As we are using Rancher ourselves, we requested an official statement from SUSE whether or not Rancher users are affected - and if yes, what mitigations should be done. Unfortunately no official response was added in the GitHub issue.
Eventually a Knowledge Base article was created on the SUSE website, which basically describes the topic and possible mitigation workarounds discussed in the GitHub issue. But at least this gave some kind of clarity: Yes, Rancher managed Kubernetes is affected.
However the criticality of the vulnerability can be slightly reduced if you are using Rancher. The webhook, used for the configuration validation, runs on a dedicated port. This can be verified by looking at the "ingress-nginx-controller" deployment in the "ingress-nginx" namespace:
Under normal circumstances, this dedicated port (TCP/8443) is not exposed to the Internet, as it is not used for handling and serving HTTP requests. This means that the validating webhook can only be targeted from inside the Kubernetes cluster. An internal POD could be abused to target and attack the Ingress controller, but an attacker first needs to gain access to such a pod or find yet another RCE vulnerability in a pod.
This finding is also shared by Sysdig:
By default, the NGINX Ingress Controller is not publicly exposed. However, an attacker who gains a foothold on a pod may be able to access the controller. If the controller has been configured to be publicly exposed, then remote attackers may be able to exploit the vulnerability.
This gives Rancher users a little breather, but they still need to act.
The mitigation, as mentioned in the WIZ findings is to temporarily disable the admission controller component of the Ingress controller, by removing the --validating-webhook argument.
As shown in the screenshot above, this can even be done in the Rancher UI, by editing the deployment and modifying the command.
On the command line using kubectl, the rke2-ingress-nginx-controller daemonset can be adjusted, too:
ck@infiniroot ~ $ kubectl edit ds rke2-ingress-nginx-controller -n kube-system
By removing the --validating-webhook=:8443 parameter the mitigation can be applied.
The mitigation helps to protect your Ingress against the discovered vulnerabilities. However this is considered a temporary workaround. The real fix comes with a Kubernetes update, containing the fixed version of the Nginx Ingress Controller.
For Rancher Kubernetes clusters deployed with RKE1 or RKE2 this means waiting for the next RKE release. RKE2 already prepared these releases today (March 27th 2025). The fixes for the IngressNightmare vulnerability will be in:
It will take a bit of time, until these new version are select-able in your Rancher UI to update the Kubernetes version on a selected downstream cluster.
Once they appear, Rancher administrators can update the Kubernetes version of a downstream cluster in the Rancher UI under Cluster Management -> Edit Cluster Config -> Under Basics select the Kubernetes Version containing the fix. We expect the fixed RKE2 versions to appear in the drop down menu within the next 24 hours.
We at Infiniroot love to share our troubleshooting knowledge when we need to tackle certain issues - but we also know this is not for everyone ("it just needs to work"). So if you are looking for a managed Rancher Kubernetes cluster in Switzerland or even in your own on-premise data center, check out our Private Kubernetes Container Cloud Infrastructure service at Infiniroot.