Building Resilient Real-Time Streaming Workers with DynamoDB Leases

Why WebSocket Fleets Are Hard to Manage

WebSocket connections are stateful, long-lived, and must be owned by exactly one worker at a time. When a worker crashes, its connections become orphaned and data stops flowing. Traditional HTTP load balancing doesn't solve this because the coordination must happen at the application layer.

This tutorial walks you through a production-ready pattern using Amazon DynamoDB conditional writes as a distributed lease mechanism. You'll learn how to build a self-healing WebSocket fleet on Amazon ECS with AWS Fargate that recovers from failures in seconds, not minutes.

The core insight: DynamoDB's atomic conditional writes give you distributed locking without running a separate coordination service like ZooKeeper or etcd.

The Lease Lifecycle: Acquire, Renew, Release, Expire

A lease is simply a DynamoDB item that tracks who owns a connection and when that ownership expires. Here's the schema:

# DynamoDB item structure for a connection lease
{
    "Pk": "CONN#meeting-123",          # Partition key: connection ID
    "desired_state": "STARTED",        # STARTED or STOPPED
    "ws_url": "wss://upstream.example.com/stream",
    "lease_owner": "worker-abc",       # Current owner
    "lease_expires_at_ms": 1712345678901,  # Epoch ms
    "last_seq": 42                     # For resumption
}

Acquiring a Lease (Atomic)

The conditional expression is the critical piece. It succeeds only if no lease exists OR the existing lease has expired:

import boto3
from botocore.exceptions import ClientError

async def try_acquire_lease(table, conn_id, worker_id, lease_seconds=20):
    now_ms = int(time.time() * 1000)
    expires_ms = now_ms + (lease_seconds * 1000)
    
    try:
        table.update_item(
            Key={"Pk": f"CONN#{conn_id}"},
            UpdateExpression="SET lease_owner = :w, lease_expires_at_ms = :e",
            ConditionExpression=(
                "attribute_not_exists(lease_owner) OR "
                "lease_expires_at_ms < :now"
            ),
            ExpressionAttributeValues={
                ":w": worker_id,
                ":e": expires_ms,
                ":now": now_ms
            }
        )
        return True  # We own it
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False  # Someone else owns it
        raise

Renewing (Heartbeat)

Every 5 seconds, the owner renews. If renewal fails, the worker knows it lost ownership and exits cleanly:

async def renew_lease(table, conn_id, worker_id, lease_seconds=20):
    now_ms = int(time.time() * 1000)
    try:
        table.update_item(
            Key={"Pk": f"CONN#{conn_id}"},
            UpdateExpression="SET lease_expires_at_ms = :e",
            ConditionExpression="lease_owner = :w",
            ExpressionAttributeValues={
                ":e": now_ms + lease_seconds * 1000,
                ":w": worker_id
            }
        )
        return True
    except ClientError:
        return False  # Lost ownership

Orphan Reconciliation

A GSI on (desired_state, lease_expires_at_ms) lets any worker find connections that should be active but have no owner:

# Query GSI for orphaned connections
orphans = table.query(
    IndexName="desired_state-lease_expires_at_ms-index",
    KeyConditionExpression="desired_state = :s AND lease_expires_at_ms < :now",
    ExpressionAttributeValues={":s": "STARTED", ":now": now_ms}
)
# Each worker tries to acquire; conditional write ensures only one wins
for item in orphans["Items"]:
    await try_acquire_lease(table, item["Pk"], worker_id)

Graceful Shutdown

On SIGTERM, release leases immediately by setting lease_expires_at_ms = 0 so other workers pick them up on the next reconciliation cycle instead of waiting for natural expiry:

async def release_lease(table, conn_id, worker_id):
    table.update_item(
        Key={"Pk": f"CONN#{conn_id}"},
        UpdateExpression="SET lease_expires_at_ms = :zero",
        ConditionExpression="lease_owner = :w",
        ExpressionAttributeValues={":zero": 0, ":w": worker_id}
    )

Production Warnings and Tuning

Clock Skew Is Your Enemy

The lease expiration check compares against the worker's local clock, not DynamoDB's server clock. On AWS Fargate, the Amazon Time Sync Service keeps skew within a few milliseconds—fine for a 20-second lease. Outside AWS, verify NTP is configured or increase lease duration by your maximum expected skew.

Cost Reality Check

Heartbeat writes dominate cost. At 5-second intervals:

For sustained high counts, use provisioned capacity with Auto Scaling. Doubling heartbeat to 10s halves WCU, but doubles your failover window—trade-off accordingly.

Scale Limits

• The per-connection check_desired_state loop is fine for small fleets. At scale, batch with BatchGetItem to cut RCU from N reads/cycle to 1.
• GSI queries during reconciliation use eventually consistent reads by default—halves RCU cost.
• Start with MAX_CONNECTIONS=100 per task and increase while monitoring memory (each WebSocket ~2-5 MB).

What This Pattern Doesn't Solve

• Data replay: If a worker dies mid-stream, messages between failure and reacquisition are lost unless upstream buffers or supports offset-based resumption via last_seq.
• Cross-region failover: The pattern is single-region. Multi-region requires DynamoDB Global Tables plus careful conflict resolution.
• Thundering herd: When many leases expire simultaneously, all workers query the GSI at once. Add jitter to reconciliation intervals.

Key Takeaways and Next Steps

You've seen how DynamoDB conditional writes eliminate the need for a separate coordination service while providing exactly-once ownership semantics. The full pattern—acquire, renew, release, reconcile—handles the complete failure spectrum: crashes, rolling deployments, and scale-in events.

Start here: Deploy the reference implementation with Terraform, validate the lease lifecycle with a handful of test connections, then gradually increase MAX_CONNECTIONS while watching CloudWatch Container Insights.

Level up:

• Add AWS X-Ray distributed tracing to visualize the acquisition path across workers.
• Implement offset-based resumption using the last_seq field to close data gaps after failover.
• Explore DynamoDB Streams to trigger reconciliation reactively instead of polling.

For a broader view of how this fits into modern AWS architecture patterns, see our analysis of AWS Architecture Trends 2024. And if you're interested in how seemingly trivial design decisions can have outsized ethical implications, our piece on lessons from April Fools pranks in web design is a worthwhile read.

Source: AWS Architecture Blog