Systems Administration

This section documents the infrastructure behind this website and related work (summarized in the 'Projects' section).

Linux NAS & Service Platform

NAS Movie Search web interface showing search results for a query
NAS Media Index search interface backed by FastAPI and PostgreSQL.

This page documents my Linux-based NAS and service platform. The project began as a simple network storage system hosted on a Raspberry-Pi and gradually developed into a small self-hosted Linux server environment for storage, media services, remote access, containerized applications, local DNS, reverse proxying, monitoring, backups, and service troubleshooting.

The current system runs Debian Linux directly, with Samba providing network file sharing and Docker Compose hosting application services.

  • Server: Old Lenovo desktop running Debian Linux
  • Storage: External ext4 drives mounted as Linux filesystems
  • File Services: Samba / SMB network shares
  • Media Services: Jellyfin
  • Remote Access: WireGuard VPN
  • Application Services: Docker Compose, FastAPI, PostgreSQL
  • Local DNS: dnsmasq with .home.arpa service names
  • Reverse Proxy: Caddy providing local HTTPS and service routing
  • Monitoring: SMART disk monitoring, systemd service status, application and system logs
  • Automation: Scheduled media indexing and nightly server backups

The NAS is used as both a practical home infrastructure and a learning environment for Linux administration, networking, containerization, service deployment, troubleshooting, and recovery.

                Client Devices
                        |
              Local Network / VPN
                        |
                        v
                   Local DNS
                   dnsmasq
                        |
            *.home.arpa service names
                        |
                        v
                    Caddy Proxy
                  HTTPS & routing
                  /         \
                 /           \
                v             v
            Jellyfin       Movie Search
            native          FastAPI
            service         Docker
                                |
                                v
                            PostgreSQL
                            Docker

                        |
                        v
                Debian Server
                        |
        +---------------+----------------+
        |               |                |
        v               v                v
    Samba           WireGuard        systemd
File Shares          VPN           Automation
        |
        v
Mounted ext4
    Storage
        

The design separates storage, OS services, and application services. Media files remain on the mounted storage drives, while applications such as the media index use the filesystem as their source of truth.

Local DNS & Reverse Proxy

One of the more recent improvements was replacing direct IP-address + port access with local DNS names. Insted of accessing services using addresses like 10.10.10.101:8096, local devices can use jellyfin.home.arpa.

dnsmasq provides local DNS resolution for the .home.arpa namespace. Caddy then acts as a reverse proxy for web applications. It accepts requests using the local service name and forwards them to the correct application running on the the server.

NAS Media Index

One of the services hosted on the NAS is a PostgreSQL-backed media indexing and search application.

A Python scanner walks the mounted media filesystem and records file metadata in PostgreSQL. A FastAPI application exposes search and streaming endpoints and provides a browser-based search interface. The production application runs using Docker Compose.

Mounted Media
     |
     | bind mount
     v
+----------------+
|    nas-api     |
|----------------|
| FastAPI        |
| scan.py        |
| Search UI      |
| HTTP streaming |
+-------+--------+
        |
        v
+----------------+
| nas-postgres   |
| PostgreSQL 16  |
+----------------+
            

The scanner runs automatically using a systemd timer and can also be executed manually for troubleshooting or after major system changes.

The filesystem remains the authoritative sourch fo media. PostgreSQL stores an index of that state rather than becoming the primary storage location.

Runbooks

These runbooks document common administrative and troubleshooting procedures for the Linux NAS and hosted services. They are intended as repeatable operational notes for checking storage, Samba shares, Docker containers, PostgreSQL, Jellyfin, DNS, the Caddy reverse proxy, WireGuard, automated backups, and application health.

Checking NAS storage and mounted drives

Start troubleshooting at the storage layer. Confirm that the expected filesystems are mounted and have adequate free space.

lsblk -f
findmnt
df -h

To inspect the filesystems mounted under the NAS storage path:

findmnt | grep "/srv/"

Confirm that expected media directories are accessible:

ls -lah /srv/

If a service reports missing media, confirm the filesystem is actually mounted before restarting applications or running the media scanner.

Checking disk health with SMART

SMART information can be used to inspect disk health and detect potential hardware problems.

List available disks:

lsblk -d -o NAME,MODEL,SIZE

Check SMART information for a disk:

sudo smartctl -a /dev/sdX

Important fields include reallocated sectors, pending sectors, reported errors, temperature, and the overall SMART health result.

Checking Samba file shares

Check that the Samba services are running:

systemctl status smbd --no-pager
systemctl status nmbd --no-pager

Validate the Samba configuration:

testparm

List configured shares:

smbclient -L localhost -U username

If a share is visible but files cannot be accessed, check both the Samba share configuration and the underlying Linux filesystem permissions.

namei -l /path/to/shared/directory

Samba permissions do not override Linux filesystem permissions; both layers must allow the required access.

Checking Docker and Docker Compose services

Check that Docker itself is running:

systemctl status docker --no-pager

From the application Compose directory:

cd ~/nas-homelab/compose
docker compose ps

The main application containers should include:

nas-api
nas-postgres

View all running containers:

docker ps

View recent API logs:

docker logs nas-api --tail 100

View recent PostgreSQL logs:

docker logs nas-postgres --tail 100
Checking the PostgreSQL database

Confirm that PostgreSQL is accepting connections:

docker exec nas-postgres pg_isready

Connect to the application database:

docker exec -it nas-postgres psql -U nasuser -d nasdb

Useful PostgreSQL commands include:

\dt
\d files
\d scan_runs

Count indexed files:

SELECT COUNT(*) FROM files;

Review recent scan runs:

SELECT *
FROM scan_runs
ORDER BY id DESC
LIMIT 10;

Exit PostgreSQL with:

\q
Checking application environment variables

Confirm that required environment variables are available inside the API container:

docker exec nas-api env | grep -E "DATABASE|SCAN_ROOT|ROOT_NAME"

Confirm that the configured scan root exists:

docker exec nas-api ls -lah /media
Running the media scanner manually

Before running the scanner, confirm that the real media filesystem is visible inside the container:

docker exec nas-api find /media -maxdepth 2 -type f | head

Once the mount has been verified, run the scanner:

docker exec -it nas-api python /app/scan.py

A successful scan reports the number of files seen, changed, and removed.

An unexpectedly large number of removed files should be investigated before relying on the updated database.

Viewing scanner logs

View recent scanner output:

journalctl -u nas-media-scan.service -n 100 --no-pager

Follow scanner output live:

journalctl -u nas-media-scan.service -f

Check the scanner timer:

systemctl status nas-media-scan.timer --no-pager

View the next scheduled execution:

systemctl list-timers --all | grep nas-media
Checking the FastAPI application

Test the application from the server itself:

curl http://127.0.0.1:8000/health

A healthy response confirms that the API is running and can reach PostgreSQL.

Check the API container:

docker compose ps

Inspect recent application logs:

docker logs nas-api --tail 100
Testing media search and streaming endpoints

Test the search endpoint:

curl "http://127.0.0.1:8000/files?limit=5"

The web interface can also be tested through the local service hostname:

https://movies.home.arpa

If search works but media playback fails, check the media bind mount and confirm that the API container can read the requested filesystem path.

Checking Jellyfin

Check the Jellyfin service:

systemctl status jellyfin --no-pager

Confirm that Jellyfin is listening locally:

sudo ss -ltnp | grep 8096

Test it directly from the server:

curl http://127.0.0.1:8096/

Inspect recent logs:

journalctl -u jellyfin -n 100 --no-pager

Follow Jellyfin logs live:

journalctl -u jellyfin -f

Verify filesystem access using the Jellyfin service account:

sudo -u jellyfin find /path/to/media -type f | head
Checking local DNS and .home.arpa names

Check the dnsmasq service:

systemctl status dnsmasq --no-pager

Test a local DNS record from a client or from the server:

nslookup jellyfin.home.arpa
nslookup movies.home.arpa
dig jellyfin.home.arpa
Checking the Caddy reverse proxy

Check the Caddy service:

systemctl status caddy --no-pager

Validate the Caddy configuration:

sudo caddy validate --config /etc/caddy/Caddyfile

View recent Caddy logs:

journalctl -u caddy -n 100 --no-pager

Test a proxied service:

curl -I https://jellyfin.home.arpa

When troubleshooting, isolate each layer: DNS, Caddy, and the backend application.

Checking WireGuard VPN access

Check the WireGuard interface:

sudo wg show

Check the service:

systemctl status wg-quick@wg0 --no-pager

Check IP forwarding:

sysctl net.ipv4.ip_forward

When troubleshooting a remote client, verify that the peer has a recent handshake and that the expected routes and DNS settings are present.

A simple external test can be performed by disconnecting a mobile phone from Wi-Fi, connecting through WireGuard over cellular data, and trying to reach an internal .home.arpa service.

Restarting the application stack safely

Confirm that storage is mounted before restarting application services.

findmnt
df -h

Then restart the Docker application stack:

cd ~/nas-homelab/compose
docker compose down
docker compose up -d

Confirm container state:

docker compose ps

Test the API:

curl http://127.0.0.1:8000/health

The dependency order is conceptually: Storage -> Docker -> PostgreSQL -> FastAPI -> Scanner

Reviewing system and service logs

View recent high-priority system messages:

journalctl -p warning -b

View logs from the current boot:

journalctl -b

Inspect a particular service:

journalctl -u SERVICE-NAME -n 100 --no-pager

Examples:

journalctl -u jellyfin -n 100 --no-pager
journalctl -u smbd -n 100 --no-pager
journalctl -u caddy -n 100 --no-pager
journalctl -u dnsmasq -n 100 --no-pager
journalctl -u wg-quick@wg0 -n 100 --no-pager

Docker container logs are inspected separately:

docker logs nas-api --tail 100
docker logs nas-postgres --tail 100
Checking disk usage and available space

Check all mounted filesystems:

df -h

Inspect space used by a directory:

sudo du -xhd1 /var | sort -h

Check Jellyfin storage specifically:

sudo du -sh /var/lib/jellyfin
sudo du -sh /var/cache/jellyfin

Check Docker disk usage:

docker system df

Unexpectedly full root filesystems can cause applications to fail even when large NAS storage drives still have plenty of available capacity.

Backing up the PostgreSQL database

PostgreSQL is backed up using a logical database dump rather than copying its live container storage.

docker exec nas-postgres \
pg_dump -U nasuser -Fc nasdb > nasdb-backup.dump

Verify that the dump exists:

ls -lh nasdb-backup.dump

Inspect the dump table of contents:

pg_restore -l nasdb-backup.dump | head

A database dump should be created or verified before major schema, container, or application changes.

Checking automated backups

The server performs scheduled backups to a separate backup filesystem.

First confirm that the backup filesystem is mounted:

findmnt | grep backup

Check available space:

df -h

Inspect the backup timer:

systemctl list-timers --all | grep backup

Check the corresponding backup service:

systemctl status nas-backup.service --no-pager

Review recent backup logs:

journalctl -u nas-backup.service -n 100 --no-pager

Confirm that recent snapshots exist and that the latest reference points to the newest successful backup.

ls -lah /path/to/backup/snapshots

Backups should be checked periodically rather than assuming that a scheduled task is succeeding simply because the timer exists.

Validating services after a reboot

After a reboot or power interruption, verify the infrastructure from the lowest layer upward.

Check storage:

findmnt
df -h

Check Samba:

systemctl status smbd --no-pager

Check Docker:

docker ps

Check Jellyfin:

systemctl status jellyfin --no-pager

Check DNS:

systemctl status dnsmasq --no-pager

Check Caddy:

systemctl status caddy --no-pager

Check WireGuard:

sudo wg show

Check scheduled tasks:

systemctl list-timers --all

Finally test client-facing services:

https://jellyfin.home.arpa
https://moviesearch.home.arpa
Basic recovery checklist

When several services appear to be unavailable at once, check the infrastructure in dependency order rather than restarting applications immediately.

1. Confirm the server is reachable.

2. Confirm storage filesystems are mounted.

3. Confirm available disk space.

4. Check native Linux services (Samba, Jellyfin, dnsmasq, Caddy, WireGuard)

5. Check Docker.

6. Check PostgreSQL.

7. Check the FastAPI application.

8. Verify local DNS resolution.

9. Verify reverse-proxy routing.

10. Test from a client device.

11. Review system and application logs.

12. Verify the most recent backup before making major recovery changes.

The general troubleshooting sequence is: Storage -> OS -> Native Services -> Docker -> Database -> Applications -> DNS/Reverse Proxy -> Client

Security

The NAS is private home infrastructure, so the security model focuses on keeping storage, services, and administrative interfaces limited to trusted devices and networks.

  • Limited exposure: Samba, services, databases, and backends are not intentionally exposed directly to the public internet.
  • Remote access: WireGuard provides encrypted access to the network rather than exposing shares or services externally.
  • Local DNS: Internal services use the reserved .home.arpa namespace rather than public DNS records.
  • Reverse Proxy: Caddy provides control for web services instead of requiring direction connection to application ports.
  • Service binding: Where appropriate, applications listen on local interfaces and are accessed through the reverse proxy.
  • Source of truth: Media files remain on dedicated storage rather than inside application containers or databases.
  • Service separation: Docker Compose separates FastAPI and PostgreSQL from native OS services.
  • Database credentials: Application credentials are supplied through environment configuration rather than source files.
  • Filesystem permissions: Samba, Jellyfin, Docker bind mounts, and accounts are given only the accesss they require.
  • Administrative access: SSH is used for maintenance, deployment, and troubleshooting from trusted systems.
  • Backups: Server configuration and application state are copied to a separate disk using an automated snapshot process.
  • Disk monitoring: SMART information and available storage space are monitored to identify storage problems early.
  • Public documentation: Credentials, keys, internal addresses, and other sensitive implementation details are intentionally omitted from public documentation.

Maintenance

The NAS is maintained through regular checks of storage, native services, Docker containers, databse state, DNS, backups, and application health.

The goal is to keep the infrastructure understandable and recoverable rather than relying on a collection of services that happen to work.

  • Check storage health: Confirm drives are mounted, available, and review SMART information for attached disks.
  • Check Samba: Verify that network shares remain reachable and that filesystem permissions behave as expected.
  • Review Docker services: Confirm that the FastAPI and PostgreSQL containers are healthy.
  • Review native services: Check Jellyfin, Samba, WireGuard, dnsmasq, Caddy, and other systemd-managed services.
  • Verify database state: Review PostgreSQL, tables, scanner history, and indexed file counts.
  • Check scanner results: Investigate unexpectedly large numbers of added, changes, or removed files.
  • Test application endpoints: Confirm that search and streaming functions remain operational.
  • Verify DNS: Confirm that expected .home.arpa hostnames resolve correctly from both local and VPN clients.
  • Verify reverse proxy routes: Confirm that Caddy can reach the appropriate backend services.
  • Review logs: Use systemd journals, Docker logs, and application logs when services behave unexpectedly.
  • Verify backups: Confirm the nightly backup completed, the backup disk is mounted, and the newest snapshot is available.
  • Back up before major changes: Create or verify a current backup before schema, OS, container, or other major changes.
  • Document changes: Record significant configuration changes, troubleshooting steps, and recovery procedures.
  • Update intentionally: Apply Debian, Docker, and application updates with service verification afterwards.
  • Validate after reboot: Confirm storage mounts, Samba shares, Docker containers, PostgreSQL, Jellyfin, WireGuard, DNS, Caddy, timers, and application endpoints after a reboot.

The maintenance process is intentionally layered: Verify storage, then OS services, then Docker and the database, then applications, and finally DNS and client-facing access.

GitHub YouTube in