WEDNESDAY, NOVEMBER 11, 2026

How I Back Up and Recover My Debian Server

My Debian server has gradually accumulated quite a few responsibilities. It hosts Samba file shares, Jellyfin, WireGuard, local DNS, a Caddy reverse proxy, Docker containers, PostgreSQL, and a few custom services and timers.

Once enough things depended on it, knowing that at least my media files lived on separate disks was no longer enough. If the SSD containing Debian failed, I could reinstall the operating system, but reconstructing every service, configuration file, database, key, certificate, and timer by memory would be much less feasible.

So I built a fairly simple backup system around tools that already exist on Linux: rsync, hard links, pg_dump, Bash, and systemd. I think of it as a poor-man's snapshot system.

Background

A separate 1 TB drive is used for backups. The Debian server gets its own directory on that disk:

Filesystem layout
/srv/backup/debian-server/
├── latest -> snapshots/20261111-040000
└── snapshots/
    ├── 20261109-040000/
    ├── 20261110-040000/
    └── 20261111-040000/

Every dated directory looks like a complete point-in-time backup of the server.

The backup drive itself is mounted independently so it can act as a general backup for more than just the Debian server. That also means that the 1TB drive is a very large single point of failure and ought to be backed-up "off-site" as well.

The useful part of this layout is that recovering the server does not require combining a full backup with a chain of incremental backups. I can simply open the newest successful snapshot and treat it as complete.

Creating Reasonable Snapshots

While copying most of the server every night would work, it would waste a lot of space. Instead, after one original backup, subsequent ones use rsync with --link-dest pointing at the previous snapshot.

Bash
rsync --link-dest=<previous-snapshot> ...

When a file has changed, rsync stores a new copy. Otherwise it creates a hard link to the existing data in the previous snapshot.

This is different from a symbolic link which points to another path. If the target path disappears, a symbolic link becomes broken. A hard link, on the other hand, refers to the filesystem "inode" or physical data on the disk as opposed to a path. The data is only removed after the final hard link referring to it is deleted.

For example, if /etc/fstab did not change between two backups, both filenames would refer to exactly the same physical data on the disk.

That can be verified with stat:

Bash
sudo stat /srv/backup/debian-server/snapshots/*/system/etc/fstab

If two copies show the same inode number, they are hard-linked.

That gives me directories that behave like complete backups while avoiding another physical copy of every unchanged file.

The system currently retains up to fourteen snapshots with the backup script running nightly at 04:00AM. If more than fourteen snapshots exist, the script prunes the oldest.

Keeping Track of the Latest Backup

Alongside the dated snapshots is a symbolic link called latest which points to the newest successful backup.

Filesystem
latest -> snapshots/20261111-040012

I can inspect it with:

Bash
ls -l /srv/backup/debian-server/

or resolve it directly:

Bash
readlink -f /srv/backup/debian-server/latest

The important detail is that latest is only updated after a backup completes successfully. This is because the backup script finishes by updating the symbolic link:

Bash
ln -sfn "snapshots/$STAMP" "$BASE/latest"

If the backup fails halfway through, the script never reaches that step. latest therefore continues pointing at the previous known-good backup rather than an incomplete one.

What is Backed Up?

I made an important distinction between things that are easy to recreate and things whose state I really do not want to lose.

The snapshots preserve system configuration including things such as:

  • Samba configuration
  • WireGuard and PiVPN configuration
  • dnsmasq configuration
  • the Caddy configuration
  • SSH configuration
  • filesystem mount configuration
  • custom systemd services and timers

They also preserve important application states.

Jellyfin's data directory contains its database, users, library identity, metadata, and artwork.

Caddy's data directory is especially important because it contains the local certificate authority used for internal HTTPS. Restoring it means devices that already trust that CA can continue trusting the reconstructed server.

The backup also includes my application deployment under /srv/apps, including the Docker Compose configuration and the source for my NAS media indexing and search service.

My home directory is backed up as well, including things such as SSH configuration, Git data, shell configuration, and administrative files.

PostgreSQL gets special treatment. I deliberately do not copy its live Docker volume. The backup script uses pg_dump to create logical database backups which are much more portable and can be restored into a clean PostgreSQL instance.

The backup script also records an inventory of the machine including mounted filesystems, installed packages, and other useful system states gathered by running system commands and saving the output to a file with the tee command.

Some things are intentionally excluded. Docker images for instance can be downloaded or rebuilt. The large external media drives are not copied into the system snapshots either as they are separate storage devices that require their own backup strategy.

The purpose of this particular backup is primarily that if the Debian server system drive died today, I have everything prepared to bring it back on a new drive with minimal downtime.

The Recovery Procedure

Assume the SSD on which the Debian server is installed dies. The recovery procedure would therefore be as follows:

Install Debian on a Replacement SSD

Create the administrator acount, install SSH/system utilities as usual.

Reconnect the Storage

The backup records output from tools such as:

Bash
lsblk
findmnt

and includes the old /etc/fstab.

Those records make it much easier to reconstruct the expected filesystem layout and verify disk UUIDs. Once they are mounted correctly, Samba and the applications can once again see the paths they expect.

Working from the lowest infrastructure layer upward is intentional. There is little value debugging Jellyfin, for example, if its media filesystem has not mounted.

Restoring the Application Stack

The /srv/apps backup contains the Docker Compose configuration and the source for my NAS media indexing application.

Once that is restored, I can start PostgreSQL as an empty service and restore the logical database dump.

A restore would look approximately like:

Bash
docker exec -i nas-postgres pg_restore -U nasuser -d nasdb --clean --if-exists < nasdb.dump

But the exact command depends on whether the fresh Compose initialization has already created the database schema, so this is one of the steps that would require verification during the recovery procedure.

Once PostgreSQL has been restored, the FastAPI service can reconnect to the existing media index.

Restoring Jellyfin

Jellyfin should be stopped while its state is restored.

Its backed-up data and cache directories can then be placed back under /var/lib/jellyfin and /var/cache/jellyfin, with ownership corrected afterward.

That will recover the existing Jellyfin identity, users, metadata, artwork, and library information rather than creating an entirely new server.

Restoring Caddy and local HTTPS

My local network uses .home.arpa service names and Caddy provides HTTPS for those services using an internal certificate authority. The Caddy data directory contains that CA and its private key.

Restoring /var/lib/caddy therefore means machines that already trust the original local CA do not all need to be configured again after the server is rebuilt.

That also means this portion of the backup is sensitive and needs to be protected accordingly.

Restoring WireGuard

WireGuard is similar. The backup preserves the original server keys and peer information under the WireGuard/PiVPN configuration. Restoring those file means my existing device profiles can continue recognizing the same VPN server instead of requiring every peer to be recreated.

Again, this is one of the reasons the backup drive itself needs to be treated as sensitive data rather than as an ordinary spare disk.

Bringing automation back

I also preserve my custom systemd units.

At the moment these include things such as the media scanner and the server backup itself.

After restoring the relevant .service and .timer files:

Bash
sudo systemctl daemon-reload

allows systemd to discover them again, after which I can enable the required timers and services.

Acceptance testing

Recovery would finish with testing rather than assuming that a successful boot means the server is recovered. The first command to check is:

Bash
systemctl --failed

Then working through the system by dependency, similar to the order in which the server was recovered:

Recovery order
Storage
   |
   v
Operating system services
   |
   v
Docker / PostgreSQL
   |
   v
Applications
   |
   v
DNS / reverse proxy
   |
   v
Client access

That means checking the media mounts, Samba shares, Jellyfin, dnsmasq, Caddy and HTTPS, WireGuard, Docker, PostgreSQL, the NAS Movie Search application, and finally the backup system itself. The end goal is for the reconstructed server to perform the same jobs as the original one.

Knowing How to Restore

A backup is as useful as knowing how to restore it. This setup is deliberately uncomplicated. The snapshots are ordinary directories; the database backups are normal PostgreSQL dumps; the files are copied with rsync; hard links provide space efficiency; latest is a placeholder; systemd runs the job automatically. There is no proprietary backup format that requires special software before recovery can even start.

The design, however, is not perfect. The 1TB backup drive is still a single point of failure, and adding an "off-site" copy is the obvious next step.

But now, in response to the infamous dread of "what if the drive fails?" for this particular server, I have an answer.

Labels: technology

GitHub YouTube in