Automate the Full AWS DRS Agent Lifecycle Install, Heal, and Offboard with SSM

Why Manual DRS Agent Management Breaks at Scale

If you protect hundreds of EC2 instances with AWS Elastic Disaster Recovery (DRS), managing the replication agent manually is a losing battle. Agents stall after network changes, descoped servers keep replicating and billing, and problems between drills go unnoticed until it's too late.

This tutorial walks through a production-tested automation that handles the full agent lifecycle—install, heal, and offboard—using AWS Systems Manager (SSM) Automation documents triggered by EventBridge or a schedule. No Lambda functions to maintain. One EC2 tag controls everything.

The logic lives in three SSM Automation documents, and the entire workflow is available in the aws-samples/sample-drs-agent-lifecycle-automation GitHub repository. For a deeper look at how AWS designs resilient architectures at scale, see this insight on Meta's AI storage blueprint.

Core Implementation: Three Documents, One Tag

The automation uses a single EC2 tag key (default: DR) with three values:

Onboarding: Idempotent Install and Repair

State Manager runs the DRS-Agent-Onboard document every 30 minutes against tag:DR=yes instances. It's idempotent—healthy instances are skipped.

# state-manager-association.yaml (excerpt)
MaxConcurrency: 10
MaxErrors: 100%  # one failure doesn't stop the fleet

A concurrency lock (flock on Linux, named mutex on Windows) prevents race conditions when runs overlap. The failback tag AWSDRS:AllowLaunchingIntoThisInstance is applied only when the agent is confirmed present.

Healing: Event-Driven Stall Recovery

When DRS detects a stalled replication, it emits an event in the DR Region. An EventBridge forwarding rule filters on STALLED state and sends it cross-Region to the source Region:

{
  "source": ["aws.drs"],
  "detail-type": ["DRS Source Server Replication Status Changed"],
  "detail": {
    "state": ["STALLED"]
  }
}

The heal document restarts the agent service first (most stalls are transient). If restart fails, it re-runs the installer in place, preserving driver state to avoid a full-disk rescan. Non-agent stalls (bandwidth throttling, staging disk issues) escalate to operators via SNS.

Offboarding: Honest Billing Status

When you set DR=no or delete the tag, EventBridge fires a Tag Change on Resource event. The offboarding document applies safeguards:

1. Terminated-instance guard — EC2 termination strips tags; don't delete the recovery point.
2. Tag re-check — if the tag was restored, exit.
3. Approval gate — if AutoOffboardEnabled=false, notify and stop.
4. State recording — capture replication state, lag, last snapshot.
5. Recovery-instance check — flag active recovery instances (separate EC2 billing).

After teardown, the outcome reflects what actually succeeded:

Tagging Commands

# Protect an instance
aws ec2 create-tags --resources i-0123456789abcdef0 \
  --tags Key=DR,Value=yes --region us-west-1

# Descope (triggers offboarding)
aws ec2 create-tags --resources i-0123456789abcdef0 \
  --tags Key=DR,Value=no --region us-west-1

# Or remove the tag entirely (also triggers offboarding)
aws ec2 delete-tags --resources i-0123456789abcdef0 \
  --tags Key=DR --region us-west-1

Safety Guards and Limitations

The automation includes nine safety guards, but you should understand the trade-offs:

• Stall-detection latency counts against your RPO. The 15-minute RPO is a customer target, not an AWS guarantee. Layer 2 (EventBridge heal) depends on DRS detecting the stall first—lag must accumulate before the server is marked stalled.
• Reinstall can trigger a block-level rescan whose duration scales with disk size. On large volumes, this may temporarily exceed the RPO window.
• Layer 3 (State Manager backstop) runs every 30 minutes, which exceeds a 15-minute RPO. It's a safety net, not primary defense.
• Pause during DR events. Use the stand-down flag to prevent automation from acting during failover/failback:

aws ssm put-parameter --name /drs/control/event-in-progress \
  --value true --overwrite --region us-west-1

• Cleanup doesn't stop replication. Removing the automation leaves DR=yes instances replicating. You must stop replication through the DRS console or API.

Next Steps

1. Clone the repo and deploy in audit mode (auto_install_enabled: false).
2. Tag a few instances DR=yes and let onboarding run for a day.
3. Review S3 audit reports—they show what would be installed without changing anything.
4. Turn on installation, then test offboarding on one instance.
5. For broader frontend trends, check out this guide to CSS border-shape.

Conclusion

You now have a complete, event-driven automation for the AWS DRS agent lifecycle: install, heal, and offboard—all controlled by one EC2 tag. The architecture uses SSM Automation documents, EventBridge cross-Region forwarding, and honest billing status reporting so you never wonder whether replication costs actually stopped.

Start in audit mode, scale gradually, and use the stand-down flag during declared DR events. The full source is in the aws-samples/sample-drs-agent-lifecycle-automation repository—open an issue if you hit edge cases.

Source: AWS Architecture Blog