3.2 KiB
IT PRO PARTNER — DOCUMENT 4 OF 4 DR Testing & Validation Schedule The cadence that turns committed RTO/RPO targets into verified ones Field Value Document status ACTIVE Date July 9, 2026 Governing rule A target in the Master Plan is “unverified” until it has passed a live test recorded here.
-
Test Cadence Test Frequency Owner Backup job success check Daily (automated) Monitoring/alerting Backup age / RPO compliance check Daily (automated) Monitoring/alerting Single database restore Monthly Network Services Team Single app stack restore Quarterly Network Services Team Core failover test (Core → app1-bu) Quarterly Network Services Team Core failback test (app1-bu → Core) Quarterly, immediately after failover test Network Services Team Full DR tabletop exercise Twice yearly Full team + stakeholders Provider/S3 outage simulation Twice yearly Network Services Team Do not skip the failback test: A failover test without an immediate failback test only proves half the design. Schedule them back-to-back so failback is never left untested for a full quarter.
-
Restore Test Log — Required Fields Every restore test, whether scheduled or incident-driven, gets logged with: Date and tester name Server/application tested Backup artifact path and timestamp used Restore duration (actual, compared against the RTO target) Data validation performed against a known record, file, or document Issues encountered Remediation ticket, if needed Date Tester System Backup Used Duration RTO Target Met? Validation Passed? Issues / Ticket
-
Failover / Failback Test Log Date Trigger Simulated Detection Time Time to Active Failback Time Split-Brain Check Passed? Issues / Ticket
-
Immediate Priority Test Queue Given uptime is the top priority, run these before anything else on the cadence above: Standby capacity load test on app1-bu (validates DR Master Plan v2, Section 4). First Core failover + failback test, end to end, to establish a real baseline against the 5-minute RTO target. Verify MariaDB dump automation on wphost02 and fleettracker360, then restore one dump from each. Restore one Docker volume on each of app1, app2, app3 into a test environment. Confirm Object Lock and write-only IAM are actually enabled on the Wasabi bucket (Backup & Security Hardening Standard, Section 2), not just documented. Stand up and validate the secondary backup destination replication job.
-
Sign-off A target moves from “committed” to “verified” in the Master Plan only after it has a passing entry in this log, dated within the current quarter. Re-verify quarterly — infrastructure and application changes can silently break a previously-passing test. Target Status Last Verified Verified By Core RTO: 5 minutes Unverified — pending first test
Core RPO: 15 minutes Unverified — pending first test
app1/2/3 RTO: 60 minutes Unverified — pending first test
wphost02 / fleettracker360 RTO: 2 hours Unverified — pending first test
Backup immutability (Object Lock) Unverified — pending confirmation
Secondary backup destination Not yet built
This document is the source of truth for whether the DR plan actually works, not just whether it is well written. Keep it current.