Recently, during a routine performance check on the backend of a side project I developed, I noticed that the automated backup cron jobs had been silently failing for weeks. Although I only noticed the errors when the disk space filled up, I later discovered that backups weren’t actually going anywhere due to a simple configuration error: the rsync command was targeting an independent tmpfs mount point. This incident once again reminded me how insidious data backup issues can be in self-hosting environments and how critical regular checks are.
Self-hosting covers a wide range of applications, from personal projects to small-scale commercial applications. While it provides full control over our own servers or VPSs, it also places all responsibility on us. One of the most fundamental and often overlooked components of this responsibility is undoubtedly data backup. Backup, whose value is often not understood until a problem occurs, should actually be at the center of every infrastructure strategy.
Why Does Data Backup Often Get Overlooked?
Backup is often perceived as a “set it and forget it” task, but this approach carries serious risks. As I experienced, once backup systems are set up, they are often left untouched for long periods, and whether they are working is rarely checked. This situation, especially in self-hosting environments, can cause a small configuration change or a disk space problem to completely break the backup chain.
In modern systems, the interaction between dependencies and services is becoming increasingly complex. While we might think a database backup has completed successfully, we might not realize that related log files or configurations are missing. Such omissions can prevent the system from becoming operational again in a full disaster recovery scenario. Therefore, we should view the backup process not just as copying files, but as taking a snapshot of the entire system.
Backup Strategies: What Are Full, Differential, and Incremental Approaches?
Data backup strategies are fundamental choices that affect many factors, from storage space to recovery time. Generally, we can talk about three main approaches: Full, Differential, and Incremental backups. Each has its unique advantages and disadvantages, and making the right choice is critical for efficient resource utilization and quick recovery when needed.
Full backup refers to taking a complete copy of all data. While this method might seem the most secure option, it requires a significant amount of storage space and a long backup time for large datasets. On the other hand, the recovery process is the simplest because you can restore all data from a single backup set. Differential backup, however, copies all data that has changed since the last full backup. This requires less storage space and a shorter backup time compared to a full backup, but for recovery, you need both the last full backup and the last differential backup. Incremental backup copies only the data that has changed since the last any backup (full or incremental). This method offers the least storage space and the shortest backup time, but the recovery process is the most complex; the entire incremental backup chain and the last full backup must be restored in sequence.
graph TD; A["Full Backup"] --> B["All Data Copied"]; B --> C["High Storage Cost"]; B --> D["Easy Recovery"]; E["Differential Backup"] --> F["Changes Since Last Full Backup"]; F --> G["Medium Storage Cost"]; F --> H["Recovery with Full + Differential"]; I["Incremental Backup"] --> J["Changes Since Last Backup"]; J --> K["Low Storage Cost"]; J --> L["Difficult Recovery with Entire Chain"];
I generally prefer a combination of weekly full backups and daily incremental backups for critical data. This approach optimizes storage costs and provides the flexibility to achieve a specific Recovery Point Objective (RPO). Especially when working with a production ERP, hourly incremental backups of operational data were an indispensable strategy to ensure business continuity in case of potential data loss.
How to Determine Backup Targets (RPO/RTO) in a Self-Hosting Environment?
When defining a backup strategy, Recovery Point Objective (RPO) and Recovery Time Objective (RTO) are vitally important concepts. RPO refers to the maximum amount of data (in terms of time) that can be lost in a disaster. For example, an RPO of 4 hours means that in the worst-case scenario, you could lose the last 4 hours of data. RTO, on the other hand, specifies how long it should take for the system to become operational again after a disaster. These two objectives directly influence your backup frequency, storage options, and the complexity of your recovery processes.
For my own side projects, I generally aim for an RPO of 4-6 hours and an RTO of 24 hours. These targets ensure that data losses remain at an acceptable level while keeping costs and operational complexity reasonable. However, in a client project, especially for a system handling financial transactions, the RPO might be measured in minutes and the RTO in a few hours. In such cases, more aggressive strategies like continuous replication or very frequent incremental backups come into play.
When determining RPO, I consider the rate of data change and the cost of data loss. If data is constantly changing (e.g., a message queue or a heavily transacted database), the RPO needs to be kept low. For RTO, factors such as system complexity, the amount of data to be recovered, and the capabilities of the recovery team come into play. Automated recovery tools and a well-documented disaster recovery plan can significantly reduce RTO.
Special Cases in Backing Up Services Like PostgreSQL and Redis
In self-hosting environments, we often use specific data storage services, and their backup approaches differ from simple file copying. PostgreSQL and Redis, in particular, have their own special considerations regarding data integrity and performance. When backing up these services, it’s necessary to ensure not only the data itself but also its consistency and recoverability.
For PostgreSQL, the pg_basebackup tool is one of the safest ways to take a physical copy of the database. Additionally, the Write-Ahead Log (WAL) archiving mechanism provides point-in-time recovery (PITR) capability, meaning the ability to revert to a specific moment in time. I typically take regular full backups with pg_basebackup and continuously archive WAL files to a separate storage area. This gives me the flexibility to even revert the last transaction in case of data loss. However, it’s important to monitor WAL bloat; excessive WAL file accumulation can lead to disk space issues and affect backup processes. Also, care must be taken to ensure that PostgreSQL’s connection pool tuning settings do not cause connection restrictions during backup.
# Example of taking a PostgreSQL base backup
pg_basebackup -h localhost -p 5432 -U backup_user -D /var/lib/postgresql/backups/base_backup_$(date +%Y%m%d) -Ft -X fetch -P -vOn the Redis side, the situation is a bit different. Redis is an in-memory database that keeps data in memory and uses RDB snapshots or AOF (Append Only File) persistence mechanisms for durability. RDB snapshots write a snapshot of the database to disk at regular intervals. AOF, on the other hand, logs every write operation to Redis in a log file. My preference is usually to use both: I aim for a lower RPO with AOF, while creating faster recovery points with RDB snapshots. Redis’s OOM eviction policy choices can also be critical during backup; an incorrect policy can lead to data loss if memory fills up.
# Redis persistence settings (in redis.conf)
save 900 1
save 300 10
save 60 10000
appendonly yes
appendfsync everysecBackups made without understanding the unique mechanisms of these services can often lead to incomplete or inconsistent data. Therefore, it is important to carefully review the official documentation for each service and follow best practices.
Backup Verification and Recovery Drills: Indispensable Steps
The weakest link in a backup plan is not testing whether the backups actually work. There’s a big difference between saying “I have a backup” and “I can successfully restore my backup.” Therefore, backup verification and regular recovery drills are an indispensable part of my operational processes. If necessary, even setting up a separate test environment for these drills is worth the effort.
In my practice, an automated verification process kicks in after backup operations are complete. This process checks the integrity of the backed-up files (e.g., checksum comparison), attempts a simple restore of database backups, and verifies that critical services come back online. For example, restoring a PostgreSQL backup to a separate Docker container and running a simple SQL query on it is an effective way to confirm the backup is sound.
# Example command sequence for verifying a PostgreSQL backup
# Copy the backup file to a temporary directory
cp /path/to/backup/db.sql.gz /tmp/db_restore_test/
gzip -d /tmp/db_restore_test/db.sql.gz
# Start a Docker container (with an empty PostgreSQL)
docker run --rm --name pg_restore_test -e POSTGRES_PASSWORD=mysecretpassword -d postgres:14
# Restore the data
docker exec -i pg_restore_test psql -U postgres < /tmp/db_restore_test/db.sql
# Verify the database is working with a simple query
docker exec pg_restore_test psql -U postgres -c "SELECT COUNT(*) FROM your_table;"
# Stop the container
docker stop pg_restore_testBeyond backup verification are recovery drills. At regular intervals (e.g., quarterly), I simulate a real disaster scenario and perform a system setup and data recovery process from scratch, end-to-end. These drills not only reveal whether the backups work but also whether my disaster recovery documentation is up-to-date and how proficient my team (or I myself) am with the recovery process. In a production ERP, by regularly conducting these drills, we would test how quickly even the most critical systems could be brought back online in an unexpected situation. Hidden network-level issues encountered during these drills, such as MTU/MSS mismatches or DNS negative caching, helped us improve our overall system resilience.
Automation and Monitoring: Preventing Manual Errors
Manual backup processes leave room for human error and are not sustainable. Therefore, automating backup processes as much as possible and continuously monitoring them is critically important in self-hosting environments. Automation ensures that backups run regularly and without errors, while monitoring allows us to detect potential problems early.
cron jobs or systemd units are indispensable for backup automation. systemd timers can offer a more flexible and reliable alternative to cron; for example, they can trigger timers even on system boot and provide more detailed logging capabilities. In one of my projects, I set up a pipeline using systemd units to back up a specific database every 4 hours and move it to a different storage location. This allowed me to ensure data security without the need for manual intervention.
# Example systemd timer unit (backup.timer)
[Unit]
Description=Run database backup every 4 hours
[Timer]
OnCalendar=*-*-* *:00:00
Persistent=true
[Install]
WantedBy=timers.target# Example systemd service unit (backup.service)
[Unit]
Description=Database backup service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup_script.shMonitoring is the complement to automation. Visualizing the status of backup processes (number of successes/failures, backup duration, disk space usage) with tools like Prometheus and Grafana allows us to quickly identify potential problems. Setting up alerts that search for specific keywords (e.g., “backup failed”) in journald logs is also an effective method. Additionally, preventing unauthorized access attempts to backup servers with tools like fail2ban strengthens our overall security posture. On one server, when brute-force attempts to SSH began, fail2ban automatically blocked them, protecting the backup mechanisms.
Disaster Recovery Plan and Backup Location Diversity
Backup is only one part of a disaster recovery plan. In the event of a real disaster (server failure, data center outage, cyberattack), simply having backups is not enough; it’s also necessary to have a clear plan on how to access these backups and how to restore the system. An important component of this plan is the diversity of backup locations.
Backups kept in a single physical or logical location can become completely useless if something happens to that location. Therefore, approaches like the “3-2-1 rule” are popular: 3 copies of the data, on 2 different media, and 1 copy offsite (remote location). In my own infrastructure, I send one copy of the data on my main server to a separate disk partition on the same server, another copy to a different storage server on the network, and finally, to an S3-compatible cloud storage service. This allows me to access data even in the event of a regional outage or a local disk failure.
In the disaster recovery plan, access methods to backup locations should also be detailed. For example, VPN topologies or Zero Trust Network Access (ZTNA) solutions can be used for accessing offsite backups. Additionally, encrypting backups and securely storing access credentials are indispensable for cybersecurity. In a production ERP project, we aimed to be prepared for a potential outage by regularly updating our disaster recovery plan and defining detailed steps for different scenarios. One of the most critical points of this plan was keeping data backups in different geographical locations and enforcing multi-factor authentication for accessing these locations.
Conclusion
Self-hosting brings significant responsibilities along with the freedom and control it provides. Data backup is paramount among these responsibilities and is often the most neglected area. From my own experiences, I’ve seen that the mere existence of a backup system is not enough; it’s essential to constantly question its reliability, regularly verify it, and test it against potential scenarios.
Correctly defining backup strategies, realistically setting RPO and RTO targets, understanding the nuances of specialized services like PostgreSQL or Redis, and most importantly, automating and monitoring backup processes, forms the best line of defense against unexpected problems. Let’s remember that backup is not a product, but a process, and it requires continuous attention and maintenance. The next step might be to review your own self-hosting environment’s backup plan and schedule a recovery drill.