Disaster recovery exercise: what if the primary site and admin access are unavailable?

Run a tabletop and an isolated recovery drill. Verify emergency access, independent backups, recovery time and actual data loss.

01

The scenario: administrators are locked out too

At 9 a.m., Boreal Workshop loses its primary site. The VPN is down, regular admin accounts cannot sign in and the last visible copy depends on the same directory. The business goal is simple: an order clerk signs in, finds an existing order and saves a new synthetic order. The drill passes when that workflow is accepted and measured, even if the rest of the infrastructure is still unavailable.

Start with a tabletop, then restore into an isolated network. The scenario is fictional and calls for no induced production outage or production change.

02

1. Tabletop: make decisions without the primary site

Assign a recovery lead, facilitator, recorder and business owner who accepts the service. Keep the runbook, contacts and objectives accessible outside the primary site and tenant. The record captures UTC timestamps, decisions, owners and evidence. Set an escalation deadline and a provider identity verification process in case planned access fails.

Present three developments: the VPN is down; the usual approver is unavailable; the latest backup cannot be read or is too recent to trust after an incident. Each time ask who can authorize action, which dependency failed, what fallback works and what evidence closes the step. If the answer needs the VPN or a secret stored in the unavailable tenant, log a gap to fix before the pilot.

03

2. Prove emergency access and independent copies

List separate access paths for the recovery platform, backups, decryption keys and identity. An Entra emergency account does not automatically have rights to a hypervisor, vault, subscription or separate tenant. For Entra, Microsoft recommends at least two non-federated cloud-only emergency accounts with strong authentication independent of normal admins, such as FIDO2 keys. Test the expected path from a secured workstation and independent network, monitor sign-ins and record activity without exposing secrets. Use scoped test accounts in the pilot and verify existing emergency procedures separately.

Check that at least one copy independent of the failure domain can be read using the fallback path and that recovery keys are accessible. A synchronized replica may carry deletions or corruption. Compare an offline or immutable copy with an earlier known-good point. Inspect inventory, retention, point age, chain and key availability without changes; only a restore proves usability. During an incident, have the response team approve the recovery point first.

04

3. Restore dependencies in an isolated pilot

Lab changes only: create a network without production routes or peering, test DNS, dedicated management access and explicit outbound requirements. Use synthetic data or approved sanitized copies. Prevent mail, webhooks, replication and jobs from reaching production before any workload boots. Stop the pilot if a route or integration escapes containment.

Suggested order: emergency access and compute; network, DNS and time; identity, keys and certificates in dependency order; data; application; business workflow. Document the bootstrap route if identity recovery depends on networking or storage. Follow vendor recovery procedures for directories and replicated databases, and never reconnect a restored copy to the live network.

Record hypervisor and agent versions and backup formats. Confirm point compatibility, restore permissions, capacity, quotas and standby licensing before the drill. For example, with Azure Site Recovery, select Test Failover and an isolated Azure test network, record the recovery-point type and use test cleanup afterwards. Check support and compute, storage and service costs for the actual workload. Replication does not prove the recovery subscription can start your VMs.

05

4. Validate the network, application and order workflow

From a test VM, use the lab DNS resolver and TLS name. These PowerShell checks are read-only and expose no password. HTTP 200 covers only the HTTPS probe; also confirm DNS resolves to the isolated network. Keep certificate validation enabled.

Sign in as a test user in a fresh session, retrieve the synthetic order and compare it to the baseline. In the lab only, save a synthetic order and sign in again to check persistence. Confirm an unauthorized user is denied and no real email or event was sent. Have the business owner accept the evidence.

PowerShell — read-only checks
# From an authorized test VM, using the lab DNS resolver.
Resolve-DnsName -Name 'orders.recovery.example' -Server '192.0.2.53' -DnsOnly
Test-NetConnection -ComputerName 'orders.recovery.example' -Port 443
$r = Invoke-WebRequest -Uri 'https://orders.recovery.example/ready' -TimeoutSec 10 -MaximumRedirection 0 -ErrorAction Stop
if ($r.StatusCode -ne 200) { throw 'Unexpected readiness response' }
$r.StatusCode
06

5. Measure RTO and RPO, then close the pilot

RTO is the target time to restore service. Measure from simulated disruption to business acceptance, including emergency access. RPO is the tolerated data loss in time: compare interruption time with the latest consistent transaction actually recovered. Backup completion time is insufficient. Use timestamped synthetic checkpoints and record any uncertainty in the measurement.

On failure, preserve evidence, stop and clean up lab resources without altering copies or widening access. On success, remove test VMs and temporary access, check for residual routes or DNS and reconcile costs. Lab cleanup does not replace a production failback plan. Assign an owner and retest date to every gap.

Fictional metricTargetObservedDecision
Service accepted4 h9:00 → 11:40 = 2 h 40 minPass
Data loss30 min9:00 − 8:18 = 42 minFail: reduce interval or fix chain, then retest
07

Official sources