HAProxy for Re-Encrypting Load Balancing

I've been working with HAProxy for a while now. I should have written a blog post about installation and basic configuration, but for that I'm going to direct you to this rather good tutorial. My only disagreement with that page is making the haproxy stats page available without encryption and authentication, but it's a minor quibble because it's good to know the stats page exists and adding encryption and authentication is significantly more advanced.

What I want to show is how to do TLS termination AND re-encrypting to your back end, while making sure all connections are encrypted (ie. any connections coming in on port 80 will be redirected to port 443). Why would you want to do this? If you have a trusted network for your back end servers, more power to you - skip the re-encryption. My back ends are VMs in a huge data centre I don't own. Why unencrypt in the first place? HAProxy has two modes, "http" (which I think is the default), and "tcp". You can do more with the data if you can see it (ie. it's unencrypted) and HAProxy is more flexible in "http" mode. But much more important, this arrangement means that I can take backends out of circulation or add new ones at any time, and it's transparent and without downtime for the user - I just change the config.

As mentioned in a previous blog post, HAProxy's documentation is extensive, incredibly detailed, and utterly useless until you've learned the basics from another source. I recommend (in that blog post) Load Balancing with HAProxy: Open-source technology for better scalability, redundancy and availability in your IT infrastructure by Nick Ramirez.

A basic configuration looks like this:

global # this section contains settings that apply to the HAProxy process itself
    log     /dev/log    local0
    chroot  /var/lib/haproxy
    stats   socket /run/haproxy/admin.sock mode 660 level admin
    stats   timeout 30s
    user    haproxy
    group   haproxy
    daemon

defaults # this section configures settings that are reused across all of the proxies that we'll define
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s

frontend all # frontend defines a reverse proxy that will listen for incoming requests on a certain IP and port
    bind            *:80
    default_backend app

backend app # this section defines a pool of servers that the frontend will forward requests to
    balance leastconn
    server  app1 localhost:8080 check
    server  app2 localhost:8081 check

A useful tool that HAProxy itself provides is the ability to test any new configuration you put into place with:

$ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c

This will throw human-readable errors. Sometimes it's useful to quash those errors and just get a yes/no exit value from the test (for scripting and the like):

$ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q

Here's the re-encrypting version. Some explanation is shown in the comments, but the certificates require further explanation which follows this file:

global
    # these are default-ish settings, nothing to change here and/or use your own
    log    /dev/log    local0
    chroot /var/lib/haproxy
    user   haproxy
    group  haproxy
    daemon

    # HAProxy defaults to 1024 bit DH params, and that caps your Qualys
    # ( https://www.ssllabs.com/ssltest/ ) score at 'B'.  This took me
    # up to 'A+'.
    tune.ssl.default-dh-param 2048

    # Default ciphers to use.  This list is from:
    # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    # I'd suggest some serious research, not blindly copying this!
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3
    # consider adding 'no-tlsv10'

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s

frontend all
    bind *:80
    # list multiple certificates on the same line separated by a space
    # (if you do that you have to address the handling of other domains with acls)
    bind *:443 ssl crt /etc/ssl/private/fullchainandkey.pem
    # 'crt' parameter must include both the full chain AND key, catted together, order doesn't matter

    # if they've arrived encrypted, send them to the back end
    acl tls_test ssl_fc_sni test.example.com
    use_backend test if tls_test

    acl host_test hdr(host) -i test.example.com
    # if this is test AND NOT https, make it https
    redirect scheme https if !{ ssl_fc } host_test
    use_backend test if host_test

# and we re-encrypt to our back-end:
backend test
    balance leastconn
    server  test static.example.com:443 ssl check ca-file /etc/ssl/certs/ca-certificates.crt

I use Debian's certbot package to get a certificate from Let's Encrypt. It provides you with several things, including a key and a fullchain.pem file which consists of the certificate and the appropriate parent certs. But there's a significant catch: the HAProxy crt must consist of both the fullchain.pem file and the key. I used cat to output the contents of both files to a single file called fullchainandkey.pem. Make sure to store it in a secure location: that's sensitive information.

Next is the ca-file parameter. This is simply a pointer to all the world's certificate authorities, or at least what your distro thinks is an appropriate list. The location given is correct for Debian, but may not match other distros.

To encrypt your backend - at least if you're working with Nginx, which I recommend - see this blog post.