144 lines
3.2 KiB
Markdown
144 lines
3.2 KiB
Markdown
IT PRO PARTNER — DOCUMENT 4 OF 4
|
|
DR Testing & Validation Schedule
|
|
The cadence that turns committed RTO/RPO targets into verified ones
|
|
Field
|
|
Value
|
|
Document status
|
|
ACTIVE
|
|
Date
|
|
July 9, 2026
|
|
Governing rule
|
|
A target in the Master Plan is “unverified” until it has passed a live test recorded here.
|
|
|
|
1. Test Cadence
|
|
Test
|
|
Frequency
|
|
Owner
|
|
Backup job success check
|
|
Daily (automated)
|
|
Monitoring/alerting
|
|
Backup age / RPO compliance check
|
|
Daily (automated)
|
|
Monitoring/alerting
|
|
Single database restore
|
|
Monthly
|
|
Network Services Team
|
|
Single app stack restore
|
|
Quarterly
|
|
Network Services Team
|
|
Core failover test (Core → app1-bu)
|
|
Quarterly
|
|
Network Services Team
|
|
Core failback test (app1-bu → Core)
|
|
Quarterly, immediately after failover test
|
|
Network Services Team
|
|
Full DR tabletop exercise
|
|
Twice yearly
|
|
Full team + stakeholders
|
|
Provider/S3 outage simulation
|
|
Twice yearly
|
|
Network Services Team
|
|
Do not skip the failback test: A failover test without an immediate failback test only proves half the design. Schedule them back-to-back so failback is never left untested for a full quarter.
|
|
2. Restore Test Log — Required Fields
|
|
Every restore test, whether scheduled or incident-driven, gets logged with:
|
|
Date and tester name
|
|
Server/application tested
|
|
Backup artifact path and timestamp used
|
|
Restore duration (actual, compared against the RTO target)
|
|
Data validation performed against a known record, file, or document
|
|
Issues encountered
|
|
Remediation ticket, if needed
|
|
Date
|
|
Tester
|
|
System
|
|
Backup Used
|
|
Duration
|
|
RTO Target Met?
|
|
Validation Passed?
|
|
Issues / Ticket
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3. Failover / Failback Test Log
|
|
Date
|
|
Trigger Simulated
|
|
Detection Time
|
|
Time to Active
|
|
Failback Time
|
|
Split-Brain Check Passed?
|
|
Issues / Ticket
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
4. Immediate Priority Test Queue
|
|
Given uptime is the top priority, run these before anything else on the cadence above:
|
|
Standby capacity load test on app1-bu (validates DR Master Plan v2, Section 4).
|
|
First Core failover + failback test, end to end, to establish a real baseline against the 5-minute RTO target.
|
|
Verify MariaDB dump automation on wphost02 and fleettracker360, then restore one dump from each.
|
|
Restore one Docker volume on each of app1, app2, app3 into a test environment.
|
|
Confirm Object Lock and write-only IAM are actually enabled on the Wasabi bucket (Backup & Security Hardening Standard, Section 2), not just documented.
|
|
Stand up and validate the secondary backup destination replication job.
|
|
5. Sign-off
|
|
A target moves from “committed” to “verified” in the Master Plan only after it has a passing entry in this log, dated within the current quarter. Re-verify quarterly — infrastructure and application changes can silently break a previously-passing test.
|
|
Target
|
|
Status
|
|
Last Verified
|
|
Verified By
|
|
Core RTO: 5 minutes
|
|
Unverified — pending first test
|
|
|
|
|
|
Core RPO: 15 minutes
|
|
Unverified — pending first test
|
|
|
|
|
|
app1/2/3 RTO: 60 minutes
|
|
Unverified — pending first test
|
|
|
|
|
|
wphost02 / fleettracker360 RTO: 2 hours
|
|
Unverified — pending first test
|
|
|
|
|
|
Backup immutability (Object Lock)
|
|
Unverified — pending confirmation
|
|
|
|
|
|
Secondary backup destination
|
|
Not yet built
|
|
|
|
|
|
|
|
This document is the source of truth for whether the DR plan actually works, not just whether it is well written. Keep it current. |