Gateway Co-op

yunohost maintenance ritual

currently our yunohost server in lima has a daily automated backup to borgbase. borgbase will give us an alert if more than 10 days pass without new data being uploaded, which is a good safeguard that backups are continuing to run, but it does not verify the integrity of the backups (that the running applications can actually be restored from the backups).

this is a ritual intended to do two things:

• verify the integrity of the backups
• share knowledge of how our yunohost server is maintained, so that the maintenance of the server is collectivized

this ritual requires the existence of two (or more) servers running yunohost, as well as access to the borgbase backup.

if you don't have the credentials for all of these, then message someone on signal who does, and they will send you a burn-after-reading pastebin with the needed credentials.

steps

in this procedure, we are migrating a set of applications from one yunohost server, to another, via the backups that are taken to borgbase. we will also change all domains associated with these applications to point to the tailscale IP address of the new server instead of the old server in the process. in the following steps the server we are migrating from will be called the "old server" and the server we are migrating to will be called the "new server", even though "the old server" will be "live" until we finish activating the "new server".

1. before starting, update all applications and the system of the old server and new server to the latest version. this can be done through the "System Update" section of the yunohost admin web interface on both servers. before migrating, its helpful to ensure that both servers are running the same version of yunohost (via this update), and updating to the latest is a good maintenance practice anyway, so it also makes sense to fold-in this task into the beginning of this recipe.
2. for each of the applications that we are going to migrate from the old server, to the new server, go into the gateway web admin interface, and change the IP address of their associated domain from the tailscale IP of the old server, to the tailscale IP of the new server (after updating this in the gateway, at this point, all of these applications will be temporarily down, until we finish the migration)
3. initiate a backup of all apps on the old server
   • ssh into the old server (e.g. ssh insun@lima-tailscale-ip), and then run systemctl start borg which initiates a new backup. you can confirm the backup finished via logging in https://borgbase.com
4. on the old server, in the yunohost web admin interface, in the borg application, you can find instructions on how to download the backups from borgbase (but don't run them yet). it will look something like the below:

app=borg
PATH="/var/www/$app/venv/bin/:$PATH"
export BORG_PASSPHRASE="$(sudo yunohost app setting $app passphrase)" 
export BORG_RSH="ssh -i /root/.ssh/id_${app}_ed25519 -oStrictHostKeyChecking=yes"
repository="$(sudo yunohost app setting $app repository)"

Then run for example:

    List archives: borg list "$repository" | less
    List files from a specific archive: borg list "$repository::ARCHIVE_NAME" | less
    View archive info: borg info "$repository::ARCHIVE_NAME"
    Verify data integrity: borg check "$repository::ARCHIVE_NAME" --verify-data

5. for each of the associated domains for the applications we are migrating, on the new server, in the yunohost web admin interface, make sure the domain is created on the new server, that it has an https certificate, and there is no application installed there
6. ssh into the new server and use the borg commands we found in step 2, to download the archives we need for all the applications we are migrating onto the new server
7. after downloading the archives onto the new server, use mv to put them in the right place where yunohost looks for backups (/home/yunohost.backup/archives)
8. in the yunohost web admin interface of the new server, go the backups section, and find the backups that you downloaded. click "restore" to restore each of the backups (or all of them at once if you downloaded them together).
9. for each of the apps that was migrated, you can now go to their new domain, and see that they are running correctly on the new server. the migration is now complete.
10. if anything doesn't work on the new server, you can revert it the old version, by going back into the gateway admin interface and changing its domain to point again to the tailscale IP of the old server.
11. lastly record in the maintenance log whatever maintenance actions you performed so that others can follow along

note: perhaps in the future, some or all of this recipe could be automated via bash scripts, but breaking it down into manual parts could also be helpful for understanding and collectivizing knowledge

*another idea: could be interesting to always have both servers have some live apps, so no server is ever "the backup". rather, via this ritual, which apps are in which server are swapped.