Victor Franzi's Home Page

5 February 2023

Find all the sources here

Note that this article has been written over several months/years. Migrating to Ansible was done during the end of 2020, while I left NextCloud early 2023.

Automating Stuff

Yay, it’s been a while.

During the past year, I have been improving my self-hosted instance. The first version had a few flaws that made it uncomfortable to use. For example, the deployment was a 2-steps process with an initialization followed by the actual deployment. Adding services was painful and kinda spaghetti-esque.

The major improvements are the followings:

Modules and Ansible

As the whole project kept growing, several services were added. Instead of the previous deployment script that used Docker-Compose, I now use Ansible.

Some of the services I run still require a manual intervention after the deployment. One of them was the reverse proxy, NginxReverseProxy, I replaced it with Traefik. It handles the subdomains and routing, and is fully automated using environment variables and Docker labels.

The infrastructure has been split in those modules:

The benefit of this new method is that I can deploy the modules over different servers, and have one dedicated for storage, one for routing, etc. Adding new modules is simple and making them available over the internet as well using Traefik (actually 2 Docker labels per routed service).

In the future, I plan to add a Wireguard profile into each server and inter-connect them all. I could then add an Adblocker and set it as the main DNS and eventually block all those ads.

Testing on localhost and AWS

It is actually very recent that I properly test the infra before deploying on the production servers. I now validate the different services in 2 environments.

Local host

The first one is the local host. I start a virtual machine on QEMU using Vagrant. Those stuffs are relatively new to me and it is really funny to set them up and watch the creation / provision of the VM.

I can’t really test it as the URL is localhost or a local IP address. What I do is checking the different services by using curl -H Host:franzi.fr <ip>, and visiting the Traefik dashboard at <ip>:8080.

If everything goes well so far, I destroy the VM and deploy on the cloud.

AWS

At first, the environment creation. I have been recently introduced to Terraform, deployment made easy. I use a simple 50-lines-ish HCL file to describe an EC2 instance and Gandi CNAMEs. Then all the playbooks are ran against it and I can really check the infra.

The certificates are managed by Traefik again, I just check that the services are available and basically setup dumb users on each service. Once everything works, I use Terraform again to destroy the instance and remove the DNS records.

./utils/manage

Okay that is the tenth time I rewrite another script that is always simpler and more beautiful and so on. This one will not derogate from the rule, it is very minimal and handles most of the situations.

manage is fairly explicit in its usage as it follows this format: ./manage env action. To setup an environment (one of local, aws or prod), just deploy it, once done, destroy it.

Using vagrant to deploy on local host simplified most of the work as it consists in one only command for each action. On AWS, I had to customize the Ansible playbook with a custom hostname and the EC2 variable IP. A custom subdomain is dedicated to testing, dev.franzi.fr, so it requires to be updated in the DNS records and eventually deleted (thanks Terraform again).

The production deployment is made easy by being defined in the config itself. It is formed by a list made of (url, modules, ip) tuples. The modules are then deployed on each server.

In the future, this script could also handle VPN configurations.

Backups with Borg

The backups have been a not-so-important task for me for way too long. At first, I used to use a basic rsync script that would keep /mnt synchronized on both my laptop and my server.

But I wanted something more reliable, and something I know would work easily if I had to restore the actual backups. And… this is when Borg comes in the place. Since the first time I have heard about it, I want it, so here it is.

Here is a bit of spec. I want all my machines to be backed-up on a dedicated external server. I want a daily backup, and I just want to backup the /mnt folder.

So I added 2 sub-tasks in my default playbook, one for the server, and one for the clients. The server is defined in the config file under the backup_server variable.

Server deployment

On the backup server, a dedicated user, called backup, is created. For each client, its SSH key (we will cover this subject later) is added to the backup user authorized_keys. Then, a Borg repository is created for this client, under /home/backup/client-hostname.

Client deployment

On the client side, Borgmatic is enabled to perform daily backups. The backup server name has to be setup by hand (or by a script, obviously), so it’s not hard-coded in the config file.

Then the client private key is copied under the root home, and the Borg repository is created. Easy.

SSH keys management

For each client, a SSH keypairs is created for this specific purpose. The keys have to be matching the host name (i.e. franzi.fr.{priv,pub}key).

The public key is added on the server, and the private key to the client’s root user.

An improvement for later would be to automate the keys creation based on the hosts defined in the config file. So far, it’s working perfectly, and I can see my daily backups. Perfect.

Goodbye NextCloud

What a pain that is NextCloud. It’s so slow, the Android app is buggy. I don’t wanna see it, ever, so I replaced it.

To replace it, I split it in different applications based on my need. I wanted synchronization between my devices for:

And a basic web-based file browser. This one is handled by Filebrowser. It’s really easy to use, and it even provides files sharing!

For the calendar and contacts, {Cal,Card}DAV is my best friend. I deployed the well-known Baikal which seems to be good enough for me (single user, neither a lot of contacts nor a busy calendar).

For the bookmarks, I use LinkDing. This thing is awesome. It’s very easy to use, integrates well with all of my devices (even my phone!), and handles bookmarks’ tagging.

Finally, to keep my notes, files, and photos synchronized, I use the famous Syncthing. It was a bit of a pain to set it up for the photos, and for now I can only save and sync them, not see them. The maybe there will be another blog post in a couple year to talk about my media management.

Happy ending

I noticed over the time that I tend to not take care of the whole infrastructure. I mean, it just works and asks for no maintenance.

Obviously a lot of things were not optimal, or I forgot about them, or even worse, they became laggy and slow and painful to use (Hey NextCloud). But with the modifications I made over the past years, it became more and more victor-friendly, and just like my dotfiles, I start loving it.

I got everything I want, everything I need.

Notes for the me from the future: you should buy a dedicated server for all of this, and set it up at home. So it could handle a Minecraft server, home automation, medias’ preview, and so much more things!