While this is fascinating, and Willy is brilliant as always, I always wondered why HAProxy couldn't just, you know, reload the config.
Surely you don't need to fork: Just parse the new config, create the necessary internal data structures, and let traffic flow into the new ruleset while keeping all the sockets (except for those that are superfluous, and of course let in-flight requests finish). Is it because HAProxy's internals weren't designed to do that and that it would too big of a rewrite?
I always found Varnish's design very cool: It compiles the configuration (which is a DSL called VCL) to C and loads it as a dynamically loaded library. I don't know how it does hot reloads, but I believe it does do them seamlessly.
The post kinda touches on this, and makes it clear that config changes isn't the only situation that this would come in handy -- software updates are a big reason too.
"Service upgrades are even more important because, while some products are designed from the ground to support config updates without having to be reloaded, they cannot be upgraded at all without a stop/start sequence. Consequently, admins leave bogus versions of those software components running in production because it is never the right time to do it."
And if you have restarts without downtime, there is no need for configuration reloads anymore. Why solve the same problem in 2 ways? It would just increase a chance for bugs.
All in all, HAproxy is a brilliant piece of software and Willy is running it in an exemplary way. Kudos!
I mean, "apachectl graceful" has existed for 20+ years. Sending a graceful (USR1) to apache will cause it to re-read its config, close and re-open log files, and send a signal to all children that they should exit after their current work is finished. If they have no work, they die immediately. New children are created under a master process that has the new configuration.
I know that HAProxy is not an apples to apples comparison with something like apache, but I don't see how something similar would be a huge burden to add.
A multi-protocol (way more than http) load balancer with health checking just has more state than a web server. Check bug history for apache, and you'll see that modules that deal with state, like mod_security, have had issues with graceful reload in the past.
It's not that surprising to me, especially given that haproxy is 17 years old. Expectations of a load balancer where pretty light when it was invented, so the internals weren't built for hot reload.
If you can restart without downtime you don't need a hot config reload. Suddenly you can treat config as immutable and discard an entire category of bugs as well.
> Is it because HAProxy's internals weren't designed to do that and that it would too big of a rewrite?
Bingo. The internals will eventually be rewritten to support things like this (and I think that's what Willy was hinting at towards the end of the article) in time. But it's a big project, and it's a complicated piece of software, and there are a lot of conflicting demands on dev time.
That said: it's a welcoming community, and if you want to help... do it!
>>> I always wondered why HAProxy couldn't just, you know, reload the config.
It reloads just fine, same as all software.
Or if you prefer the pessimistic version: HAProxy, nginx, Apache and Varnish all suck at reloading configurations.
The difference is that HAProxy 1) tests it and 2) have a TCP mode.
To quote the issue: They manage to achieve 1 error per 40 000 connections... only if pinning to specifics CPU (typical in high performance environment to achieve 100% usage of all cores) while doing 10 reloads per second and creating 80k new connections per second.
Do you think any of apache/nginx/varnish, would do better than that in these circumstances? If you do, you are not very realistic ;)
Surely you don't need to fork: Just parse the new config, create the necessary internal data structures, and let traffic flow into the new ruleset while keeping all the sockets (except for those that are superfluous, and of course let in-flight requests finish). Is it because HAProxy's internals weren't designed to do that and that it would too big of a rewrite?
I always found Varnish's design very cool: It compiles the configuration (which is a DSL called VCL) to C and loads it as a dynamically loaded library. I don't know how it does hot reloads, but I believe it does do them seamlessly.