How I Made Servers Invincible To Hangs

Nassella has an unusual design constraint: no SSH, or server login of any kind.

Nassella deploys immutable instances for self-hosting. If something goes wrong, you can't login to diagnose and fix it. The only option is to replace the instance with a new one. So how do you recover when even the replacement server hangs?

At first, I thought the architecture of Nassella actually made this trivially easy: just re-deploy and within a few minutes you'll have a new server with all of your configuration and application data restored. If there is an issue with the configuration, that is easy to fix by just updating the configuration on the Nassella interface and re-deploying. But what if there is an issue with the application data itself that is somehow causing even the newly created server to hang? This is all getting more complex.

The solution I came up with then is to "bake into" an instance's Flatcar Linux configuration, a hash of a backup snapshot to restore on first boot.

At the point in time I was working on solving this problem, I had not yet nailed down how the restoration of a backup snapshot would even work, so this was also forcing me to think that through even more. What I really wanted was a solution that would ensure that there was a never a scenario in which I was stuck with a server that was unresponsive or borked in some way, with no way for me to recover it, short of starting over and losing the configuration or application data. And I definitely did not want any solution that required logging into a server or requiring the server to be functional in someway. This really meant, I needed to make sure that the control plane always had some way of controlling both the configuration running on a server as well as having the ability to restore the application data, based on backups, without the instance being functional. If the control plane can always restore an instance to a "known good" state and also always had the ability to re-create an instance, then I would be assured I could always bring up a functional instance, no matter what.

Restoring An Instance From Backup Even If The Instance Is Offline

My idea then, was to have an instance restore a backup/snapshot, on first boot.

At first, you might think that is not necessary, as you could design it so that you can send a command to an instance to perform a restore, but that wouldn't work, if the instance is unresponsive.

Flatcar Linux has a mechanism that performs the initial configuration only on first boot, after which point the configuration is read-only, even across future boots. So if you want to restore the application data for an instance, the Nassella control-plane just needs to know which snapshot you want to restore and then it can deploy a new instance with the ID/hash of that snapshot baked into the initial setup configuration.

I created a "oneshot" systemd service, "restic-restore", that runs before the systemd service that brings up all of the instance's apps. This "restic-restore" service runs a simple shell script. The shell script checks for the existence of a "marker" file and also a restic snapshot ID, stored in the configuration. If the marker file does not exist and a restic snapshot ID is in the configuration, then it runs the restic restore process with the given snapshot ID. After it is done, the normal startup process continues and the instance will have been "restored" to the state of the backup. It then creates the marker file. As the configuration this is stored in becomes read-only after this first boot, the system will not attempt to perform the restore on any subsequent boots.

This architecture should solve most of the common server "hang" issues and still allow me to deploy immutable instances that don't need an SSH login, meeting my other security and simplicity goals. I obviously can't guarantee this covers all possible failure modes but it does cover all of the most common issues I've run into over the years of self-hosting including: common infrastructure failures, configuration mistakes, and corrupted application state.

Overall, this is one of many decisions I've made aimed at making self-hosting accessible to normal users, without them needing to become Linux or DevOps experts. If users can't login, don't need to login, we've really eliminated another hurdle to self-hosting.

Why no SSH?

So why does Nassella deploy instances without SSH? The initial thought was that not allowing logins of any kind increases security and simplifies management. But, also, just having the ability to login and change things on a server encourages modification of that server which leads down the path of having servers that are difficult to replace and maintain. If you just can't make changes to the configuration of a server, then you eliminate any possibility for configuration drift.

Already with the Flatcar Linux setup, configuration drift is not really a thing because the configuration for the entire system and all application configuration is read-only, so even if you could login, you couldn't change it. But you could still login and change application data, if logins were allowed. This would not be quite as bad as configuration drift, but if there is a way to maintain systems without that, I want to take it.

The more I work on Nassella, the more I find you can push the boundaries of what is possible and how close we really are to truly making "self-hosting for everyone".