Technical Disaster Recovery Plan — Orbit Scheduling (SaaS Platform)

System / platform: Orbit Scheduling (multi-tenant appointment SaaS) Owner: Platform Reliability Team Date: 2 June 2026 Last tested: 18 May 2026 Next review: 1 September 2026

---

1. Scope and critical systems

This plan covers the production services that keep Orbit's booking flow working. It does not cover the internal analytics warehouse (recovered from source systems) or the marketing site (static, rebuilt from Git).

System	Function	Criticality	Depends on
Booking API	Creates and manages appointments	Critical	Primary DB, Auth
Primary database (PostgreSQL)	System of record for bookings	Critical	Object storage (backups)
Auth service	Logs users in	Critical	Primary DB
Notification worker	Sends booking emails/SMS	High	Booking API, queue
Object storage	Uploaded files + DB backups	High	None

2. Recovery targets (RTO / RPO)

System	RTO (max downtime)	RPO (max data loss)
Booking API	30 minutes	5 minutes
Primary database	30 minutes	5 minutes
Auth service	30 minutes	5 minutes
Notification worker	4 hours	1 hour

3. Backup strategy

• What: PostgreSQL (continuous WAL streaming + hourly snapshots), object storage, infrastructure config (Terraform in Git), application secrets (managed vault, exported nightly).
• Frequency: Streaming replica kept under 5 minutes behind primary; hourly snapshots; one daily copy shipped to a second cloud region.
• Location: Primary in eu-west-1. Snapshots and the daily copy live in eu-central-1, in a separate cloud account so a compromise of production cannot delete them.
• Retention: 7 daily, 4 weekly, 6 monthly.
• Restore verification: A monthly automated job restores the latest daily copy into a scratch environment and runs a row-count and integrity check. The 18 May restore completed in 22 minutes and passed.

4. Failover and recovery procedures

Scenario A — Primary database failure

1. Alert fires from the database health check; on-call confirms primary is unreachable.
2. On-call declares an incident in the reliability channel and pages the incident commander.
3. Recovery lead promotes the eu-west-1 streaming replica to primary.
4. Repoint Booking API and Auth at the promoted primary via the connection config and restart pods.
5. Confirm test bookings read and write successfully; check replica lag log to verify data loss is within the 5-minute RPO.
6. Post status to the status page; close the incident once stable for 15 minutes.

Scenario B — Full eu-west-1 region outage

1. Trigger: cloud provider confirms a regional outage or all eu-west-1 health checks fail for 10+ minutes. The incident commander decides to fail over.
2. Restore the latest daily database copy in eu-central-1 and bring up the standby application stack there.
3. Update DNS to route booking traffic to the eu-central-1 load balancer.
4. Validate sign-in, create-booking, and notification send for two test tenants.
5. Communicate the regional failover to customers; monitor until the primary region is healthy, then plan a controlled fail-back.

5. Roles and contacts

Role	Responsibility	Name	Contact (incl. out-of-hours)
Incident commander	Declares the disaster, owns decisions	Priya Nadkarni	On-call pager #1
Recovery lead	Executes the technical steps	Tom Aldridge	On-call pager #2
Communications	Status page + customer updates	Jordan Lee	Comms channel / phone
Escalation	Cloud vendor + leadership	Sara Okonkwo	Director on-call line

6. Test schedule

Test type	What it proves	Frequency	Last run
Tabletop exercise	Roles, comms, decision flow	Quarterly	14 April 2026
Restore drill	Backups are restorable + complete	Monthly	18 May 2026
Failover drill	Real RTO is met	Twice a year	9 March 2026 (DB failover in 19 min)

7. Recovery checklist

• Incident declared and commander assigned
• Scope of impact confirmed (Booking API + DB, all tenants)
• Latest good replica identified, lag confirmed under 5 minutes
• Database failover procedure started
• Service restored and create-booking verified
• Data integrity confirmed (row-count match)
• Customers informed via status page
• Incident closed; recovery time 19 min, data loss under 5 min recorded
• Post-incident review scheduled; plan updated