based.quest – Moving from nginx to Caddy

Posted on 03/09/2023 13:00:00

Backstory

I’ve heard of Caddy for a long while, but have always been dismissive about it - ignoring it as “yet another webserver software” when I had my perfectly good trusty nginx to rely on for many years already. Over the years, my nginx config has become somewhat bloated and even take a while to do restarts on due to the complexity of said config. I did attempt to resolve it by reducing amount of expensive routines - which did help to an extent - I was complacent with my setup. Until I took a deeper dive into Caddy.

One Saturday night (…yesterday as of writing this post), BieHDC brought to my attention that Caddy has a nginx configuration adapter. I thought to myself why not try it out if it’s really plug-and-play and try to see what all the fuss is about. What followed this was all worth it.

The nginx-adapter

I started off with building the nginx adapter for Caddy which took a while. Very quickly after building it, I ran into issues - no matter what I did, I kept getting null routes on the generated output with no idea where it’s coming from. You would think I would have stopped here, but I foolishly tried to make it work for something that was never meant to be, you see, I forgot to do one key thing before even starting this whole thing - RTFM (Read The Fucking Manual). The README had a list of directives it supported and I had a ton of which were not supported, but very essential to the work of my services.

I shelved the nginx-adpater for good at this point, after wasting an hour with it.

The story doesn’t end

While I may have been burnt by the mischiefs of nginx-adapter, I wasn’t ready to give up - by this point I had already familiarised myself with Caddy’s documentation and reference syntax. I decided to give it a try for based.quest at very least to have a fair and honest impression of Caddy as a software so I wouldn’t start to bash on it going forward from a bad experience on a beta-phase config adapter.

In literally a few minutes, I had made it, the Caddyfile for based.quest - I present it to you in all its glory in this blogpost:

http://based.quest, based.quest {
	root * /var/www/based.quest/html
	file_server
}

breezewiki.based.quest {
	reverse_proxy localhost:10416
}

rimgo.based.quest {
	reverse_proxy localhost:3000
}

proxitok.based.quest {
	reverse_proxy localhost:8999
}

quetre.based.quest {
	reverse_proxy localhost:3333
}

Yes, you are seeing this right - this little configuration only needed. I was shocked. All this time I had been writing more for no reason, because I turned a blind eye to the alternative that was right in front of me for many years - I’ve seen the name, I’ve seen it used in several applications that I host under Docker, but never gave it a shot until it was pointed out to me that an adapter exist to load nginx configs (which obviously didn’t work out, but that’s beside the point).

The main server

I wasn’t going to stop with only based.quest server - after seeing the potential, I took on the seemingly herculean task of porting my main server’s 2000+ lines of nginx config to a Caddyfile. I delved into documentation for over an hour, searching for answers for any immediate question I had.

You may find what I wrote useful for your own configs, so here are (not fully mine) templates you can use for building your own Caddy virtualhosts:

(errors) {
	handle_errors {
		@custom_err file /err-{err.status_code}.html /err.html
		handle @custom_err {
			rewrite * {file_match.relative}
			file_server
		}
		respond "{err.status_code} {err.status_text}"
	}
}
(php-fpm) {
	encode gzip
	php_fastcgi unix//run/php/php7.4-fpm.sock {
		try_files {path} {path}/index.php =404
	}
	file_server
}
(cache-static) {
	@static {
		file
		path *.ico *.css *.js *.gif *.webp *.avif *.jpg *.jpeg *.png *.svg *.woff *.woff2
	}
	header @static Cache-Control max-age=1209600
}

You can find most of this on the official documentation, but thought it may be useful to consolidate it here for your convenience.

Migrating Matrix is super easy, but be attentive to details, they provide the Caddyfile samples on their own repository already - here is my final variant of it:

http://cernodile.com, cernodile.com, cernodile.com:8448 {
	header /.well-known/matrix/* Content-Type application/json
	header /.well-known/matrix/* Access-Control-Allow-Origin *
	respond /.well-known/matrix/server `{"m.server": "cernodile.com"}`
	respond /.well-known/matrix/client `{"m.homeserver":{"base_url":"https://cernodile.com"}}`
	reverse_proxy /_matrix/* localhost:8008
	reverse_proxy /_synapse/* localhost:8008
	# [...]
}

The reason I ask you to be attentive to details is because I broke my federation with it. Notice the /matrix/server field. I had copied the sample from Synapse repository and left it as "cernodile.com:443" whereas my original well-known record was just "cernodile.com".

There were 2 blockers when migrating my services to Caddy. PeerTube and Gitweb. Unfortunately for PeerTube, since PeerTube does not support anything else than nginx, I had to keep one nginx virtualhost up and use reverse_proxy directive against it. As for Gitweb which doesn’t supply any other configuration than for Apache, I did manage to port it over to Caddy by adapting generic fastcgi adapter in a way that works for Gitweb, feel free to use it as well:

git.based.quest {
	root * /usr/share/gitweb
	try_files {uri} index.cgi
	handle /static/* {
		file_server
	}
	reverse_proxy unix//var/run/fcgiwrap.socket {
		transport fastcgi {
			env GITWEB_CONFIG /etc/gitweb.conf
			split .cgi
		}
	}
}

Closing thoughts

After all this effort, I am down to 108 lines on my Caddyfile from 2000+ lines on nginx. The performance is as great as ever with no hickups noticed. I want to thank BieHDC for bringing this to my attention, I wouldn’t have probably gone down this rabbit hole if it weren’t for the nginx adapter. This experience was all the worth it and I suggest you give Caddy a try if you are currently running an nginx site - you won’t regret having to write 20x less configuration for your webserver.

Thank you for reading

Cernodile