Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
When launching a new LXC container using lxc-create, the following error might show up and container creation fails:
root@host ~ # lxc-create -n ubuntu1 -B lvm --vgname=vglxc --fstype=ext4 --fssize=15G -t download-infiniroot -- -d ubuntu -r focal -a amd64
The cached copy has expired, re-downloading...
Setting up the GPG keyring
ERROR: Unable to fetch GPG key from keyserver
The reason is that since LXC 3.x, the (suggested) default template is the lxc-download template (/usr/share/lxc/templates/lxc-download). This connects to GPG key servers to verify the downloaded image and prevent a man in the middle (MITM) attack. The connection to the keyserver does not happen via http or https but rather via hkp(s), a different protocol using a different port (tcp/11371). See my notes in a previous articles How to import apt repository key behind outgoing proxy and Setting up a LXC host on Ubuntu 18.04 for additional information related to HKP(S).
But even if the LXC host is able to communicate with the Internet on port tcp/11371, this error can still happen.
In the background LXC uses the keyservers behind the address pool.sks-keyservers.net. What happened in this particular situation was that no DNS resolving happened for this DNS address:
root@host ~ # dig -t A pool.sks-keyservers.net
; <<>> DiG 9.11.5-P4-5.1+deb10u3-Debian <<>> -t A pool.sks-keyservers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 50405
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;pool.sks-keyservers.net. IN A
;; AUTHORITY SECTION:
sks-keyservers.net. 288 IN SOA ns2.kfwebs.net. kf.kfwebs.net. 3210622135 600 14400 172800 600
;; Query time: 3 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 21 20:31:55 CEST 2021
;; MSG SIZE rcvd: 102
Whatever happened here, DNS caching problem (at Google??!) or DNS misconfiguration of the target domain doesn't really matter. Fact is that at this point in time the key server could not be reached and therefore no GPG key could be downloaded. This leads to no verification of the downloaded image/rootfs and container creation fails.
The lxc-download template has a hardcoded key server configured:
if [ -z "${DOWNLOAD_KEYSERVER:-}" ]; then
DOWNLOAD_KEYSERVER="hkp://pool.sks-keyservers.net"
# Deal with GPG over http proxy
if [ -n "${http_proxy:-}" ]; then
DOWNLOAD_KEYSERVER="hkp://p80.pool.sks-keyservers.net:80"
DOWNLOAD_GPG_PROXY="--keyserver-options http-proxy=\"${http_proxy}\""
fi
fi
From this code snippet we can also see that a variable ${DOWNLOAD_KEYSERVER} is checked. If it does not exist, the default key server is defined.
This means that an alternative key server can be specified using this variable!
According to a discussion in the Linuxcontainers community forums,
this isn't the first time that the default key servers were not
reachable. User simos suggest to switch to the Ubuntu key servers.
With this information, an alternative key server can be defined - right in front of the lxc-create command:
root@host ~ # DOWNLOAD_KEYSERVER="keyserver.ubuntu.com" lxc-create -n ubuntu1 -B lvm --vgname=vglxc --fstype=ext4 --fssize=15G -t download -- -d ubuntu -r focal -a amd64
The cached copy has expired, re-downloading...
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
This time it worked - by using a different GPG key server!
Another (insecure!) alternative is to completely disable the image download verification. There are two ways to do this. Either by using the -no-validate (template) parameter or by modifying the LXC template.
root@host:~# lxc-create -n container1 -B lvm --vgname vglxc --fstype=ext4 --fssize=50G -t download -- --no-validate -d debian -r buster -a amd64
Instead of using the command line parameter, the template (/usr/share/lxc/templates/lxc-download) can also be manually modified by setting the DOWNLOAD_VALIDATE variable to "false". By default this value is set to "true":
root@host:~# grep "^DOWNLOAD_VALIDATE" /usr/share/lxc/templates/lxc-download
DOWNLOAD_VALIDATE="true"
Setting it to false:
root@host:~# sed -i "/^DOWNLOAD_VALIDATE/s/true/false/" /usr/share/lxc/templates/lxc-download
The next lxc-create command adds a warning in the output:
root@host:~# lxc-create -n container1 -B lvm --vgname vglxc --fstype=ext4 --fssize=50G -t download -- -d debian -r buster -a amd64
Downloading the image index
WARNING: Running without gpg validation!
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
---
You just created a Debian buster amd64 (20210630_05:24) container.
To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.