Test Your Apps Resilience Simulating SQS Failures with AWS FIS

Why Test SQS Failure Handling?

When your application loses access to Amazon SQS—due to IAM misconfiguration, network partition, or a bad deployment—the symptoms are similar: sends and receives start failing. The difference between a brief blip and a cascading outage lies in how your services handle those failures. Do they fail fast? Open circuit breakers? Buffer locally? Without testing, you're guessing.

AWS Fault Injection Service (FIS) lets you safely inject controlled failures to validate your resilience mechanisms and observability. This tutorial walks through a progressive experiment that denies SQS data-plane access, then restores it, so you can see exactly what breaks and what recovers.

Key principle: Test your application's behavior, not SQS itself. The goal is to learn whether your recovery patterns work and whether you'd notice if they didn't.

Step-by-Step: Building the Experiment

1. Define a Measurable Hypothesis

Start with a clear statement of expected behavior. For example:

When our order processing app loses SQS access for 5 minutes, we expect the producer to open its circuit breaker within 30 seconds, buffer messages to durable storage, and replay them on recovery. We'll measure success by NumberOfMessagesSent returning to baseline and ApproximateNumberOfMessagesVisible draining to near zero within 15 minutes.

If you've never tested this failure mode, frame it as discovery: "We'll block access for 2 minutes and observe how the producer handles failed sends and whether the consumer recovers unaided." Write it down before you start.

2. Prerequisites

• Instrumented producer and consumer – your application must emit metrics for circuit-breaker state, failed sends, dropped messages, and fallback writes.
• IAM role for FIS – trusted by fis.amazonaws.com, with permissions to run SSM Automation and pass a role.
• IAM role for SSM Automation – able to read/modify SQS queue policies (sqs:GetQueueAttributes, sqs:SetQueueAttributes).
• SQS queues tagged FIS-Ready: True – scopes the automation to non-production queues.
• CloudWatch dashboard and alarms – combine application and queue metrics; set a stop condition on a customer-impact signal (e.g., failed orders per minute).
• Rollback plan – in case the deny policy can't be removed.

⚠️ Critical: Never deny sqs:*. An explicit Deny overrides every Allow, including the automation's own permission to clean up. Scope the deny to data-plane actions only (SendMessage, ReceiveMessage, DeleteMessage, ChangeMessageVisibility, PurgeQueue).

3. Configure the SSM Automation Document

The automation follows four steps:

1. getTargetQueues – finds queues tagged FIS-Ready: True.
2. applyDenyAllPolicyToQueues – adds a scoped deny statement to each queue's resource policy.
3. waitForDuration – sleeps for the impairment duration (ISO 8601, e.g., PT2M).
4. removeDenyAllPolicyFromQueues – removes the FISTemporaryDeny statement.

You can make the deny self-expiring by adding a DateLessThan condition on aws:CurrentTime.

4. Chain Phases with FIS

The FIS template runs four impairment phases with recovery periods between them:

Stop conditions halt the experiment if a CloudWatch alarm fires. Choose an alarm on a customer-impact metric (e.g., application error rate), not on queue metrics that are supposed to move during impairment.

5. Run and Observe

Use the AWS CLI or console to start the experiment:

aws fis start-experiment --experiment-template-id <template-id>

During impairment, SQS operations return AccessDenied (403, non-retryable). Watch producer and consumer separately:

Producer side:

• First failed SendMessage: does it fail fast or hang?
• After 3–5 failures: does the circuit breaker open?
• When send gives up: does it buffer to durable storage or drop silently?

Consumer side:

• First failed ReceiveMessage: does it back off or hammer?
• Backlog accumulation: does ApproximateNumberOfMessagesVisible go flat?
• Recovery: does the backlog drain steadily?

💡 Note: The DLQ won't fill during impairment because ReceiveMessage is denied—redrive depends on maxReceiveCount. Watch the DLQ during recovery instead.

6. Analyze Results

Compare what happened against your hypothesis:

• Did circuit breakers activate within your target (e.g., 30 seconds)?
• Were messages preserved? Reconcile attempted sends vs. processed messages + DLQ + fallback storage.
• Did recovery drain the backlog steadily, or did it stall?
• Did longer phases reveal new failure modes (thread-pool exhaustion, memory pressure)?

Every gap between hypothesis and reality is a fix to make.

Common Pitfalls and Best Practices

Retry Logic

Don't retry everything. Retry only transient errors (throttling, timeouts, 5xx) with exponential backoff and jitter. Fail fast on non-retryable errors like AccessDenied (403). The AWS SDKs have configurable retry behavior—use it.

Circuit Breakers

A circuit breaker stops attempts after a threshold of consecutive failures, then lets a single test call through after a recovery timeout. Choose the open-state behavior deliberately: shedding or buffering load is safer than silently switching to an alternate path.

Dead Letter Queues

Configure a DLQ for every queue. It's a consumer-side safety net for poison messages, not a producer overflow buffer. Set maxReceiveCount to 3–5. Ensure consumers are idempotent.

Monitoring Strategy

Build a CloudWatch dashboard combining:

• Queue metrics: NumberOfMessagesSent, NumberOfMessagesReceived, ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage.
• Application metrics: error rates by type (distinguish AccessDenied), circuit-breaker state, DLQ depth, end-to-end latency.

Alarm on ApproximateAgeOfOldestMessage exceeding your SLA as a production alert—but not as an experiment stop condition.

Cleanup

• Verify the deny policy is gone: aws sqs get-queue-attributes --queue-url <url> --attribute-names Policy.
• Process any DLQ messages from the experiment.
• Document findings: what matched, what didn't, what you're fixing.

🔗 Further Reading: For a deeper dive into how modern AI models optimize inference throughput, check out Holotron-12B: The Hybrid SSM Model That Doubles AI Agent Throughput. And if you're curious about hardware-level bottlenecks, see Beyond Matrix Math: How NVIDIA Blackwell Ultra Tackles the Softmax Bottleneck.

Conclusion

Testing SQS failure handling with AWS FIS turns assumptions into evidence. Start with the 2-minute phase in a non-production environment, fix what breaks, then run the full sequence as your application evolves. The gaps between your hypothesis and results are your work list—and the best time to find them is before a real outage.

Next steps:

• Extend the experiment with partial failures (impair only a subset of queues).
• Combine with other failures (EC2 termination, network latency) for compound scenarios.
• Explore AWS Resilience Hub for continuous assessment.
• Review the AWS FIS samples on GitHub for working examples.

Source: Testing application resilience with Amazon SQS and AWS Fault Injection Service