Solution
High Availability, backup and recovery
Design of HA clusters, backup repositories and Disaster Recovery platforms on-premises and in the cloud. Replication, failover, retention and restore procedures sized for RTO/RPO targets and the data volume to recover.
Architecture and implementation
Copies and retention
3–2–1–1–0 architecture, independent repositories, offsite copies and immutable retention. Design of access, key recovery and historical restore points.
High Availability and replication
Active-active or active-passive topologies, synchronous or asynchronous replication, traffic distribution and coordinated application and storage failover.
Recovery and restoration
Cross-site or cross-region DR, point-in-time recovery, automated reconstruction and drills measuring recovery time and the recovered data point.
Related projects
Designing recovery around RTO and RPO
Recovery time and acceptable data loss guide storage choices, backup frequency and redundancy. These targets are defined per service, including its dependencies and data volume.
- RPORecovery Point ObjectiveAcceptable data loss
- RPO sets the maximum acceptable data loss in minutes or hours. With a 15-minute RPO and an incident at 10 am, backups or replicas must recover transactions recorded through at least 9.45 am.
- RTORecovery Time ObjectiveTime to restore service
- RTO sets the maximum acceptable service interruption. With a two-hour RTO and an incident at 10 am, the application must accept connections and transactions again by noon. This includes detection, failover or restoration, and functional checks.
- Earliest acceptable recovery point
- Service interruption
- Latest acceptable service recovery
RPO · At most 15 min of data loss
RTO · At most 2 h of interruption
These values illustrate design targets. Recovery drills measure whether they are met.
The recoverable point depends on completed backups and available transaction logs. A job scheduled every 15 minutes can fail or finish late. Recovery drills therefore verify the transactions actually recovered, followed by authentication, application queries and connections to external APIs.
Technical references AWS Well-Architected · RTO / RPO
Hot, warm and cold storage by data access
Production database tables, recent backups and archives are read at different frequencies. Hot, warm and cold describe these access patterns and the required retrieval time. Physical media and access interfaces are then selected for the workloads involved.
| Tier | Access and examples | Implementation options | Cost and retrieval |
|---|---|---|---|
| Hot | Tables and indexes used by transactions, active logs and data served by APIs | Local NVMe SSDs, all-flash array volumes or cloud storage provisioned for I/O | Budget driven by IOPS and write latency. Access without prior retrieval from an archive |
| Warm | History queried periodically and recent backups used for testing or restores | HDD pools, hybrid systems or flash repositories for concurrent restores | Balance of capacity and restore throughput. Data remains directly accessible in the chosen repository |
| Cold | Older backups, archived datasets and long-term retention | Object retention classes, offline tape, or HDD and flash repositories according to required access time | Retention cost takes priority. Retrieval or archive charges and any delay depend on the service |
A SQL database uses flash volumes for transactions. Recent backups are written to an HDD or flash repository, shown here as alternatives. A retention copy is transferred to cloud storage. Hot, warm and cold describe how data is used, while media selection follows performance and recovery-time requirements.
Azure Storage
SQL Server
Components and flows
- The database reads data pages and writes its transaction log on a local or array-backed flash volume
- An HDD repository prioritises capacity, while flash can support more concurrent restores and tests
- Azure Blob Cold and Archive are retention options with different retrieval procedures
- The Cold tier stays online, while Archive requires rehydration before restore
- Retention and copy jobs are configured separately from physical media
Flash, SSDs and the NVMe protocol
NAND flash is the memory retaining data in an SSD. NVMe is the protocol used to communicate with storage, commonly over PCIe inside a server. NVMe over Fabrics extends access across a storage network, for example through NVMe/TCP. SSDs also exist with SATA or SAS interfaces.
Flash for transactional databases
An OLTP database records the application’s operational transactions. Index and data-page access can generate many small reads, while its log requires durable writes. Flash removes the waiting caused by mechanical HDD movement. Sizing measures latency, input/output operations per second (IOPS) and throughput under the actual SQL workload.
HDD or flash for backup
HDDs are used for capacity-oriented backup repositories. Flash can accelerate parallel reads during concurrent restores or tests. A rarely read copy can remain cold even on SSD. Media selection follows retention volume and measured recovery throughput, including network and backup-software limits.
NVM Express | NVMe / NAND · NVM Express | NVMe over Fabrics
Online cold storage and archives requiring rehydration
Azure Blob Cold remains online. Azure Blob Archive requires rehydration before reading, which can take up to 15 hours depending on priority. Google Cloud Archive remains online without that step. Within Azure Blob, Hot has lower access charges, while Cold reduces stored-capacity cost. Cost calculations include stored capacity, reads and transfers, as well as minimum billable storage durations. These conditions contribute to RTO planning.
Microsoft | Azure Blob access tiers · Google Cloud | Storage classes
Block, file and object storage by application interface
A SQL engine can write its files on a block volume, a document application can open an SMB share, and a pipeline can upload Parquet files to an object bucket. These models define how software accesses data. Each can use HDD or flash, with performance determined by the service.
The SQL database writes files on a block volume mounted by its host. An application using file paths accesses an SMB or NFS share. The data pipeline uploads objects to an Azure Blob container over HTTPS. HDD or flash media are selected separately according to service performance.
- Azure Storage
Python- SQL Server
Components and flows
- Block access with a host-mounted volume, a filesystem and block I/O through a local disk or SAN
- File access with a directory tree and shared access through SMB or NFS
- Object access with a key, metadata and API requests to upload or retrieve data
- Hot, warm and cold classifications are selected separately from the access model
| Interface | Organisation and access | Workloads | Operational considerations |
|---|---|---|---|
| Block storage | An addressable block volume presented to a server. Local access or iSCSI, Fibre Channel or NVMe-oF | VM volumes, SQL data and log files, operating-system disks | The host or engine organises data on the volume. A volume shared by multiple hosts requires cluster-compatible write coordination |
| File storage | Files organised in directories and opened by path. SMB or NFS shares on NAS or cloud file services | Application files, shared directories, scientific datasets and selected VM or database deployments | The file service manages names, permissions and locks. Performance also depends on protocol and concurrent access |
| Object storage | Objects addressed by key with associated metadata. API access through services such as S3, Azure Blob or Google Cloud Storage | Parquet data lakes, AI models and datasets, media and backup copies | Applications read and write through API operations. Versioning, retention and access classes follow the service’s supported features |
Selection starts with the application’s operations. A SQL engine requires durable writes and specific locking behaviour. Vendors support selected NFS or SMB deployments. Mounting an object bucket through a file gateway requires those operations to be checked, because the object API and mounted filesystem do not automatically provide equivalent guarantees.
Technical referencesMicrosoft | Storage services · Microsoft | Blob Storage
Organising backups with the 3–2–1–1–0 rule
A backup strategy combines storage diversity, geographic separation, deletion protection and verified restores. One copy can satisfy both the offsite and immutable criteria.
3 Copies
Production data and two backup copies
2 Storage types
For example, local disk and cloud object storage
1 Offsite copy
Separated from the production site and its failures
1 Protected copy
Offline, air-gapped or immutable
0 Unresolved errors
Backup verification and recovery testing
For an on-premises platform, this architecture combines a local repository for fast restores with an offsite cloud copy under locked retention. The three copies include production data. An offline tape can add a fourth copy, disconnected from the network.
- Azure Storage
- SQL Server
Components and flows
- Application-consistent backup to a disk repository separate from production storage
- Offsite copy to Azure Blob Storage with locked WORM retention and separate administration
- Local disk and cloud object storage provide two storage types, with optional offline tape
- Restore copies into an isolated network, then verify data and application functions
- The “0” concerns verification errors to resolve, without promising zero risk
Immutable retention and offline copies
An immutable copy remains accessible, but modification and deletion are blocked during configured retention. An offline copy is disconnected. In both cases, the catalogue, encryption keys and recovery access must remain recoverable independently of production.
Replication, snapshots and backup
A replica receives changes from the primary database, including accidental deletions. A snapshot preserves an earlier volume state but remains exposed to the loss of its storage array. A backup in an independent repository retains earlier versions for the retention period and supports restoration after production storage is lost.
Technical references Veeam · 3–2–1–1–0 · Microsoft · Azure Blob WORM · Microsoft · Redundancy, replication and backup
Keeping an application running with HA
For a web application, High Availability (HA) means running multiple instances on separate hosts. A load balancer distributes HTTP requests between them and removes instances that stop passing health checks. For a single-primary database, failover promotes a replica to primary and redirects new connections to it.
Instances and traffic
Health checks query an application URL to verify that it can process a request. The load balancer itself runs as multiple instances or as a managed service distributed across zones. User sessions are held in a replicated shared service or in tokens each instance can validate, allowing another host to handle the next request.
Writes and arbitration
In a single-primary cluster, quorum sets the number of votes needed to authorise primary selection. Fencing blocks the former primary from storage or stops its server before a replica is promoted. These mechanisms prevent two servers from writing independently to the same database after a network partition. Storage-array replication and SQL-engine write coordination are configured at their respective layers.
Capacity and dependencies
Hosts at the surviving site need the RAM, CPU, network throughput and input/output operations per second (IOPS) required to absorb load after failover. The storage network uses separate switches and SAN paths. DNS and authentication remain accessible from the recovery site. Tests under load verify application responses while a host, network link or site is unavailable.
Two nearby sites host active application instances and Pure Storage FlashArray arrays synchronised through ActiveCluster. The load balancer removes failed instances from routing according to its health checks. Pure1 Cloud Mediator determines which array retains access to the volumes when the replication link fails. The application cluster applies its own quorum and fencing mechanisms.
Linux
Pure Storage
Components and flows
- Redundant entry point and application health checks; failed instances are removed from routing
- Instances distributed across two sites, with sufficient capacity on the surviving site and consistent session handling
- Synchronously replicated volumes; intersite latency contributes to write acknowledgement time
- Pure1 Cloud Mediator for storage arbitration; quorum and fencing also designed at application-cluster level
- Active-active storage without uncoordinated database writers; independent historical backups
With synchronous replication, a write waits for acknowledgement from remote storage or a remote server according to the configured mode. Intersite round-trip latency therefore increases write response time. With asynchronous replication, the primary acknowledges without waiting for the remote replica, which can be missing the latest transactions if the primary becomes unrecoverable. Backups separately retain the historical versions needed to undo a deletion or restore a corrupted database.
Technical references Microsoft · Redundancy, replication and backup · Microsoft · Load balancer reliability · Pure Storage · ActiveCluster
Designing Disaster Recovery across two regions
Disaster Recovery (DR) prepares an application to restart after losing its datacenter or cloud region. A second region holds the replicas, backups or instances required for recovery. Availability zones distribute instances across datacenters within one region and do not cover loss of the entire region.
An application and its PostgreSQL database run in region A. Region B maintains a replica and a preconfigured application platform, without production traffic. Failover combines fencing the former primary, promoting the replica and reopening the service; backups are managed separately.
Azure
Docker
PostgreSQL
Terraform
Components and flows
- Two distinct regions; a containerised application and self-managed PostgreSQL in this Azure example
- Asynchronous WAL replication; observed lag and the durability of received data determine the recoverable point
- Detection and decision, then fencing of the former primary before promotion to prevent two writable primaries
- Standby capacity, secrets and networking prepared; routing changes after the recovered service is validated
- Return to the original site after resynchronisation through a planned failback
The recovery region receives network configuration, service identities, secrets and application images. Cloud quotas and reserved resources must support the planned instances. The runbook describes failover, operation from the recovery region and failback to the original region after data resynchronisation.
| Architecture | State before the incident | Design trade-off |
|---|---|---|
| Backup / Restore | Retained copies; platform must be rebuilt | Few running standby resources; recovery dominated by transfer and rebuild time |
| Active-passive · Warm standby | Continuously updated replica and preconfigured platform | Ongoing standby cost; faster recovery with possible data lag |
| Active-active · HA | Multiple instances already serve traffic | Extra capacity and coordination; application availability must be distinguished from storage availability |
These architectures illustrate implementation options. Tests measure replication lag, recovery-site capacity and application recovery time against the project targets.
Technical references Microsoft · Multi-region disaster recovery · PostgreSQL · Standby and replication
Restore the data, rebuild the service
A file restore recovers a document or directory. A database restore recovers tables and transactions. Application recovery also rebuilds the servers, access and APIs it depends on. After a compromise, backups are first restored and checked in an isolated network before production connections are enabled.
After data corruption or compromise, recovery takes place in an isolated environment, on-premises or in another cloud region. A pre-incident backup, database logs, infrastructure code and recovery access are used to rebuild a consistent system before reopening the service.
- Azure Storage
- Docker
- PostgreSQL
- Terraform
Components and flows
- Select a clean recovery point; use a base backup and a complete log chain for PITR
- Recreate networks, instances and services; recover keys and access through a separate channel
- Restore the database, then start applications; block external integrations during testing
- Technical and application checks, measurement of recovery time and recovered data point, then traffic release
- On failure, correct the issue or select another recovery point; never release traffic after an incomplete test
Choose the right restore sequence
Full, incremental or differential
A full backup copies all data covered by the job. An incremental backup retains changes since the preceding backup, so the increments required for restoration must remain available. A differential contains all changes since the latest full backup. Restoration then uses that full backup and the latest differential. The choice depends on the volume copied per run and the number of backup files read during recovery.
Restoring a database to a specific point with PITR
For PostgreSQL, a base backup and a continuous write-ahead log (WAL) sequence support Point-in-Time Recovery. SQL Server uses a full backup, optionally a differential, then transaction-log backups under an appropriate recovery model. Recovery can stop before an accidental deletion; later transactions are then absent and must be reconciled separately.
Snapshots and instant recovery
An application-consistent snapshot supports selected rollback scenarios. Where supported, instant recovery can start a VM from a backup repository before migrating it to production storage. Repository availability and performance remain critical; booting the VM does not complete recovery.
End-to-end validation and recovery drills
Tests cover backup readability, database integrity, authentication and representative application transactions. Integrations that might send messages or modify external systems remain isolated. The record measures the recovered data point and total recovery time, then identifies gaps against targets and required corrections.
Transferring 1 TB over a 1 Gbit/s link takes roughly 2 hours 15 minutes at theoretical throughput. Decompression, log replay and application tests extend recovery time further. Sizing also accounts for cloud archive retrieval time, transfer charges and recovery-storage IOPS.
Technical references PostgreSQL · Point-in-time recovery · Microsoft · SQL Server restore sequences
Design and integration
- SSD/NVMe and HDD storage sizing with block, file or object access
- Repository architecture, retention, immutability and separate access
- Consistent backups of databases, VMs, files and application configurations
- HA clusters, load balancing, replication and failover procedures
- DR automation, isolated restoration and measurement against RTO/RPO targets
- Monitoring of backup jobs, replication lag and recovery capacity
Systems involved
- Backups, transaction logs and PITR for PostgreSQL and SQL Server
- Linux servers, virtualised environments and containerised applications
- SAN/NAS arrays, disk repositories, tape and cloud object storage
- Cloud platforms, networking and infrastructure as code
Service lineServers and hosting →
To go deeper
Five attack scenarios and the controls that limit their consequences
Exposed services, accounts, payments, suppliers and recovery need different protections. Priority follows the activities the organisation must preserve.
Testing whether backups can restore the business service
A complete restore checks the data, applications and dependencies. Its measured result shows how recovery capacity compares with the organisation’s needs.
Discuss your project
A discussion of your project’s design, development and production deployment.