My Debian server has gradually accumulated quite a few responsibilities. It hosts Samba file shares, Jellyfin, WireGuard, local DNS, a Caddy reverse proxy, Docker containers, PostgreSQL, and a few custom services and timers.
Once enough things depended on it, knowing that at least my media files lived on separate disks was no longer enough. If the SSD containing Debian failed, I could reinstall the operating system, but reconstructing every service, configuration file, database, key, certificate, and timer by memory would be much less feasible.
So I built a fairly simple backup system around tools that already exist
on Linux: rsync, hard links, pg_dump, Bash,
and systemd. I think of it as a poor-man's snapshot system.
Background
A separate 1 TB drive is used for backups. The Debian server gets its own directory on that disk:
/srv/backup/debian-server/
├── latest -> snapshots/20261111-040000
└── snapshots/
├── 20261109-040000/
├── 20261110-040000/
└── 20261111-040000/
Every dated directory looks like a complete point-in-time backup of the server.
The backup drive itself is mounted independently so it can act as a general backup for more than just the Debian server. That also means that the 1TB drive is a very large single point of failure and ought to be backed-up "off-site" as well.
The useful part of this layout is that recovering the server does not require combining a full backup with a chain of incremental backups. I can simply open the newest successful snapshot and treat it as complete.
Creating Reasonable Snapshots
While copying most of the server every night would work, it would waste a lot of
space. Instead, after one original backup, subsequent ones use rsync with
--link-dest pointing at the previous snapshot.
rsync --link-dest=<previous-snapshot> ...
When a file has changed, rsync stores a new copy. Otherwise
it creates a hard link to the existing data in the previous snapshot.
This is different from a symbolic link which points to another path. If the target path disappears, a symbolic link becomes broken. A hard link, on the other hand, refers to the filesystem "inode" or physical data on the disk as opposed to a path. The data is only removed after the final hard link referring to it is deleted.
For example, if /etc/fstab did not change between two
backups, both filenames would refer to exactly the same physical data on
the disk.
That can be verified with stat:
sudo stat /srv/backup/debian-server/snapshots/*/system/etc/fstab
If two copies show the same inode number, they are hard-linked.
That gives me directories that behave like complete backups while avoiding another physical copy of every unchanged file.
The system currently retains up to fourteen snapshots with the backup script running nightly at 04:00AM. If more than fourteen snapshots exist, the script prunes the oldest.
Keeping Track of the Latest Backup
Alongside the dated snapshots is a symbolic link called
latest which points to the newest successful backup.
latest -> snapshots/20261111-040012
I can inspect it with:
ls -l /srv/backup/debian-server/
or resolve it directly:
readlink -f /srv/backup/debian-server/latest
The important detail is that latest is only updated
after a backup completes successfully. This is because
the backup script finishes by updating the symbolic link:
ln -sfn "snapshots/$STAMP" "$BASE/latest"
If the backup fails halfway through, the script never reaches that step.
latest therefore continues pointing at the previous
known-good backup rather than an incomplete one.
What is Backed Up?
I made an important distinction between things that are easy to recreate and things whose state I really do not want to lose.
The snapshots preserve system configuration including things such as:
- Samba configuration
- WireGuard and PiVPN configuration
- dnsmasq configuration
- the Caddy configuration
- SSH configuration
- filesystem mount configuration
- custom systemd services and timers
They also preserve important application states.
Jellyfin's data directory contains its database, users, library identity, metadata, and artwork.
Caddy's data directory is especially important because it contains the local certificate authority used for internal HTTPS. Restoring it means devices that already trust that CA can continue trusting the reconstructed server.
The backup also includes my application deployment under
/srv/apps, including the Docker Compose configuration and
the source for my NAS media indexing and search service.
My home directory is backed up as well, including things such as SSH configuration, Git data, shell configuration, and administrative files.
PostgreSQL gets special treatment. I deliberately do
not copy its live Docker volume. The backup script uses pg_dump
to create logical database backups which are much more portable and can
be restored into a clean PostgreSQL instance.
The backup script also records an inventory of the machine including mounted
filesystems, installed packages, and other useful system states gathered
by running system commands and saving the output to a file with the
tee command.
Some things are intentionally excluded. Docker images for instance can be downloaded or rebuilt. The large external media drives are not copied into the system snapshots either as they are separate storage devices that require their own backup strategy.
The purpose of this particular backup is primarily that if the Debian server system drive died today, I have everything prepared to bring it back on a new drive with minimal downtime.
The Recovery Procedure
Assume the SSD on which the Debian server is installed dies. The recovery procedure would therefore be as follows:
Install Debian on a Replacement SSD
Create the administrator acount, install SSH/system utilities as usual.
Reconnect the Storage
The backup records output from tools such as:
lsblk
findmnt
and includes the old /etc/fstab.
Those records make it much easier to reconstruct the expected filesystem layout and verify disk UUIDs. Once they are mounted correctly, Samba and the applications can once again see the paths they expect.
Working from the lowest infrastructure layer upward is intentional. There is little value debugging Jellyfin, for example, if its media filesystem has not mounted.
Restoring the Application Stack
The /srv/apps backup contains the Docker Compose
configuration and the source for my NAS media indexing application.
Once that is restored, I can start PostgreSQL as an empty service and restore the logical database dump.
A restore would look approximately like:
docker exec -i nas-postgres pg_restore -U nasuser -d nasdb --clean --if-exists < nasdb.dump
But the exact command depends on whether the fresh Compose initialization has already created the database schema, so this is one of the steps that would require verification during the recovery procedure.
Once PostgreSQL has been restored, the FastAPI service can reconnect to the existing media index.
Restoring Jellyfin
Jellyfin should be stopped while its state is restored.
Its backed-up data and cache directories can then be placed back under
/var/lib/jellyfin and /var/cache/jellyfin,
with ownership corrected afterward.
That will recover the existing Jellyfin identity, users, metadata, artwork, and library information rather than creating an entirely new server.
Restoring Caddy and local HTTPS
My local network uses .home.arpa service names and Caddy
provides HTTPS for those services using an internal certificate
authority. The Caddy data directory contains that CA and its private key.
Restoring /var/lib/caddy therefore means machines that
already trust the original local CA do not all need to be configured
again after the server is rebuilt.
That also means this portion of the backup is sensitive and needs to be protected accordingly.
Restoring WireGuard
WireGuard is similar. The backup preserves the original server keys and peer information under the WireGuard/PiVPN configuration. Restoring those file means my existing device profiles can continue recognizing the same VPN server instead of requiring every peer to be recreated.
Again, this is one of the reasons the backup drive itself needs to be treated as sensitive data rather than as an ordinary spare disk.
Bringing automation back
I also preserve my custom systemd units.
At the moment these include things such as the media scanner and the server backup itself.
After restoring the relevant .service and
.timer files:
sudo systemctl daemon-reload
allows systemd to discover them again, after which I can enable the required timers and services.
Acceptance testing
Recovery would finish with testing rather than assuming that a successful boot means the server is recovered. The first command to check is:
systemctl --failed
Then working through the system by dependency, similar to the order in which the server was recovered:
Storage
|
v
Operating system services
|
v
Docker / PostgreSQL
|
v
Applications
|
v
DNS / reverse proxy
|
v
Client access
That means checking the media mounts, Samba shares, Jellyfin, dnsmasq, Caddy and HTTPS, WireGuard, Docker, PostgreSQL, the NAS Movie Search application, and finally the backup system itself. The end goal is for the reconstructed server to perform the same jobs as the original one.
Knowing How to Restore
A backup is as useful as knowing how to restore it. This setup is
deliberately uncomplicated. The snapshots are ordinary
directories; the database backups are normal PostgreSQL dumps; the files
are copied with rsync; hard links provide space efficiency;
latest is a placeholder; systemd runs the
job automatically. There is no proprietary backup format that requires
special software before recovery can even start.
The design, however, is not perfect. The 1TB backup drive is still a single point of failure, and adding an "off-site" copy is the obvious next step.
But now, in response to the infamous dread of "what if the drive fails?" for this particular server, I have an answer.
in