Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
HAProxy usually works like a charm. In fact HAProxy is (probably) one of the most stable open source software capable of handling thousands of simultaneous connections. Using HAProxy for more than 7 years now, you can imagine my surprise when I was not able to run HAProxy in a Docker/application container.
Very simple stuff actually. Ubuntu 18.04 was used as base image. In Dockerfile a simple installation of the haproxy package, followed by the RUN of the haproxy process, was added:
FROM ubuntu:18.04
MAINTAINER Claudio Kuenzler
# install packages
RUN apt-get update \
&& apt-get install -y -qq haproxy
CMD ["/usr/sbin/haproxy","-W","-db","-f","/etc/haproxy/haproxy.cfg"]
Once the image was successfully built on Docker Hub, I wanted to deploy the container:
root@host:~# docker run napsty/haproxy:latest
Unable to find image 'napsty/haproxy:latest' locally
latest: Pulling from napsty/haproxy
7ddbc47eeb70: Pull complete
c1bbdc448b72: Pull complete
8c3b70e39044: Pull complete
45d437916d57: Pull complete
8e8788d95679: Pull complete
Digest: sha256:37d36ca920099c8d8288b1207fa7705bc5c169cc8747cf5afd599ab138c316e4
Status: Downloaded newer image for napsty/haproxy:latest
[ALERT] 337/164645 (1) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]
The error stating "cannot bind UNIX socket" is unclear without additional information.
Let's get into this container to see what is actually happening:
root@harbor:~# docker run -it napsty/haproxy:devel bash
root@d8e6cf9fa2d5:/# stat /usr/sbin/haproxy
File: /usr/sbin/haproxy
Size: 1634568 Blocks: 3200 IO Block: 4096 regular file
Device: fd0eh/64782d Inode: 682455 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-12-04 16:22:23.299177985 +0000
Modify: 2019-12-02 15:38:31.000000000 +0000
Change: 2019-12-04 16:23:27.219793835 +0000
Birth: -
root@d8e6cf9fa2d5:/# stat /etc/haproxy/haproxy.cfg
File: /etc/haproxy/haproxy.cfg
Size: 1276 Blocks: 8 IO Block: 4096 regular file
Device: fd0eh/64782d Inode: 401588 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-12-04 16:22:23.099176057 +0000
Modify: 2019-10-28 12:01:03.000000000 +0000
Change: 2019-12-04 16:23:27.019791909 +0000
Birth: -
Both files /usr/sbin/haproxy and /etc/haproxy/haproxy.cfg do exist. But why would HAProxy not start? Launching the RUN command manually should show more:
root@d8e6cf9fa2d5:/# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
[ALERT] 337/162548 (13) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]
And indeed - HAProxy fails to start because it cannot bind to the Unix socket on /run/haproxy/admin.sock. Interestingly, this directory does not even exist:
root@d8e6cf9fa2d5:/# stat /run/haproxy
stat: cannot stat '/run/haproxy': No such file or directory
What if the directory is added manually?
root@d8e6cf9fa2d5:/# mkdir /run/haproxy
root@d8e6cf9fa2d5:/# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
root@d8e6cf9fa2d5:/# ps auxf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 18508 3516 pts/0 Ss 16:23 0:00 bash
haproxy 17 0.0 0.0 54284 1256 ? Ss 16:27 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
root 18 0.0 0.0 34400 2884 pts/0 R+ 16:27 0:00 ps auxf
Ha! HAProxy is running now!
An obvious workaround is to add the creation of the directory already in Dockerfile - at the creation of the image:
FROM ubuntu:18.04
MAINTAINER Claudio Kuenzler
# install packages
RUN apt-get update \
&& apt-get install -y -qq haproxy
# add missing socket path
RUN mkdir /run/haproxy
CMD ["/usr/sbin/haproxy","-W","-db","-f","/etc/haproxy/haproxy.cfg"]
Once the image is built on Docker Hub, a docker pull will download the changed image and the container hopefully starts up this time:
root@host:~# docker pull napsty/haproxy:latest
devel: Pulling from napsty/haproxy
7ddbc47eeb70: Already exists
c1bbdc448b72: Already exists
8c3b70e39044: Already exists
45d437916d57: Already exists
af6d65a824f2: Already exists
cbf99dcc6c4f: Pull complete
b6229eac0350: Pull complete
a5d72fba4c6a: Pull complete
Digest: sha256:c03d7f770fa82ab77bcb2fa021b49300526367b47dc16d2ec9fe89c6be24febd
Status: Downloaded newer image for napsty/haproxy:latest
root@host:~# docker run napsty/haproxy:latest
No errors anymore!
That's the bigger challenge. "Many minds == many opinions"; some may think I'm completely wrong but in my opinion the haproxy package coming from the Ubuntu repositories should be responsible to create this path during the installation (postinst). I created Ubuntu bug #1855140, we'll see how this will be (eventually) solved.
One big question remains though: Why do installations of the haproxy package on "normal" Ubuntu 18.04 work but seem to fail every time when deployed in a Docker container? This can be double-checked in the deb source package.
Note: This package analysis was done after the discussion in the comments (see comments below the article).
After a deb-src repository is added to /etc/apt/sources.list, the haproxy source package can be installed:
root@linux:~# grep src /etc/apt/sources.list
deb-src http://us.archive.ubuntu.com/ubuntu/ bionic main restricted
root@linux:~# apt-get update
root@linux:~# apt-get source haproxy
Reading package lists... Done
NOTICE: 'haproxy' packaging is maintained in the 'Git' version control system at:
https://salsa.debian.org/haproxy-team/haproxy.git
Please use:
git clone https://salsa.debian.org/haproxy-team/haproxy.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 2,122 kB of source archives.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (dsc) [2,280 B]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (tar) [2,055 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (diff) [65.2 kB]
Fetched 2,122 kB in 1s (2,280 kB/s)
dpkg-source: info: extracting haproxy in haproxy-1.8.8
dpkg-source: info: unpacking haproxy_1.8.8.orig.tar.gz
dpkg-source: info: unpacking haproxy_1.8.8-1.debian.tar.xz
dpkg-source: info: applying 0002-Use-dpkg-buildflags-to-build-halog.patch
dpkg-source: info: applying haproxy.service-start-after-syslog.patch
dpkg-source: info: applying haproxy.service-add-documentation.patch
dpkg-source: info: applying haproxy.service-use-environment-variables.patch
W: Download is performed unsandboxed as root as file 'haproxy_1.8.8-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
Inside the haproxy-1.8.8 folder the structure of the source package can be seen:
root@linux:~# cd haproxy-1.8.8/
root@linux:~/haproxy-1.8.8# ll
total 652
drwxr-xr-x 12 root root 4096 Dec 5 14:15 ./
drwx------ 6 root root 4096 Dec 5 14:15 ../
-rw-r--r-- 1 root root 490605 Apr 19 2018 CHANGELOG
drwxr-xr-x 18 root root 4096 Apr 19 2018 contrib/
-rw-r--r-- 1 root root 41508 Apr 19 2018 CONTRIBUTING
drwxr-xr-x 6 root root 4096 Apr 19 2018 debian/
drwxr-xr-x 5 root root 4096 Apr 19 2018 doc/
drwxr-xr-x 2 root root 4096 Apr 19 2018 ebtree/
drwxr-xr-x 3 root root 4096 Apr 19 2018 examples/
-rw-r--r-- 1 root root 788 Apr 19 2018 .gitignore
drwxr-xr-x 6 root root 4096 Apr 19 2018 include/
-rw-r--r-- 1 root root 2029 Apr 19 2018 LICENSE
-rw-r--r-- 1 root root 3089 Apr 19 2018 MAINTAINERS
-rw-r--r-- 1 root root 36682 Apr 19 2018 Makefile
drwxr-xr-x 6 root root 4096 Dec 5 14:15 .pc/
-rw-r--r-- 1 root root 15356 Apr 19 2018 README
-rw-r--r-- 1 root root 2713 Apr 19 2018 ROADMAP
drwxr-xr-x 2 root root 4096 Apr 19 2018 scripts/
drwxr-xr-x 2 root root 4096 Apr 19 2018 src/
-rw-r--r-- 1 root root 14 Apr 19 2018 SUBVERS
drwxr-xr-x 2 root root 4096 Apr 19 2018 tests/
-rw-r--r-- 1 root root 24 Apr 19 2018 VERDATE
-rw-r--r-- 1 root root 6 Apr 19 2018 VERSION
Building deb packages is not trivial, but if one has gotten into it (for whatever reason) there are a couple of important files to look at. The first file worth to check out is debian/haproxy.dirs:
root@linux:~/haproxy-1.8.8# cat debian/haproxy.dirs
etc/haproxy
etc/haproxy/errors
var/lib/haproxy
var/lib/haproxy/dev
A couple of paths are mentioned here, but the (now famous) /run/haproxy does not appear in that list. However it appears in haproxy.tmpfile:
root@linux:~/haproxy-1.8.8# cat debian/haproxy.tmpfile
d /run/haproxy 2775 haproxy haproxy -
What exactly is this tmpfile and what is its purpose? The last time I personally was involved in more deb package building was during the Debian Wheezy (7) release. This is a pre-systemd Debian release. Might tmpfile be related to systemd? Interestingly this exact information can be found in the package's changelog:
root@linux:~/haproxy-1.8.8# grep -rni tmpfile *
debian/changelog:980: tmpfiles.d config for systemd.
By taking a closer look at the documentation of dh_systemd_enable it's clearly written:
debian/package.tmpfile
If this exists, it is installed into usr/lib/tmpfiles.d/package.conf in the package build directory. (The tmpfiles.d mechanism is currently only used by systemd.)
There's no mistake: "currently only used by systemd". And as it is common knowledge: Systemd does not run in a Docker container.
Further information can also be found on the Ubuntu manpages for tmpfiles.d:
systemd-tmpfiles uses the configuration files from the above directories to describe the creation, cleaning and removal of volatile and temporary files and directories which usually reside in directories such as /run or /tmp.
This sums it up pretty clearly: The /run/haproxy directory is only created by using systemd-tmpfiles, which does not exist in a Docker container. As the haproxy is installed during the Docker image building (which happens inside the Docker container), there is no Systemd available. Hence the directory /run/haproxy is never created.