0%
AWS in Practice: Services, Limits, and Failure Modes
Account Foundations
Compute
Storage
Databases
Networking
Delivery and Visibility
Limits, Cost and Recovery
SQS, SNS, and EventBridge
Amazon SQS is the oldest AWS service and still one of the most used. It gives a producer somewhere to put messages and a consumer somewhere to take them from, with storage, retries and scaling handled for you. Queues decouple services so that a slow or failed consumer does not slow the producer, absorb bursts so that downstream systems process work at their own pace, and spread work across many workers. Most production problems with SQS come not from the service itself but from misunderstanding how messages become invisible, reappear and get deleted, which this section covers in detail.
A queue is also a buffer between systems with different capacities. A web tier can accept ten thousand orders a minute during a flash sale while the fulfilment system behind it handles two thousand. Without a queue, the fulfilment system is overwhelmed or the web tier must reject orders. With a queue, orders accumulate and drain over the following minutes, and the only visible effect is a delay in confirmation emails. That buffering is the most common reason teams introduce SQS, and it shapes the metrics they should watch, which measure delay rather than errors.
Send, receive, delete
A producer sends a message, up to 256 KB of text, to a queue. A consumer calls ReceiveMessage, which returns up to ten messages along with a receipt handle for each. The messages are not removed: they become invisible to other consumers for the visibility timeout, 30 seconds by default. The consumer processes each message and then calls DeleteMessage with the receipt handle. If the consumer crashes, or simply takes longer than the timeout, the message becomes visible again and another consumer receives it. This is how SQS guarantees that work is not lost, and it is also why every consumer must be ready to see a message more than once.
The receive, process, delete cycle has no transaction around it. A consumer that writes to a database and then crashes before deleting will reprocess the message after the timeout, repeating the database write. A consumer that deletes first and then crashes loses the work. The standard design is to process first, delete after success, and make processing idempotent, so that a repeat does no harm. Every SQS consumer in production should be reviewed with one question: what happens if this message is delivered twice, or ten minutes late?
Choosing the visibility timeout
The visibility timeout should be longer than the time a consumer needs to process a message, including retries inside the handler, with margin for slow cases. AWS suggests at least six times the expected processing time for Lambda consumers, which retry and batch. A timeout that is too short causes duplicate processing: the message reappears while the first consumer is still working on it, a second consumer takes it, and both complete it. A timeout that is too long delays recovery: when a consumer crashes, the message stays invisible for the full timeout before anyone else can retry it.
For work whose duration varies widely, such as video transcoding or report generation, a consumer can extend the timeout while it works by calling ChangeMessageVisibility, like a heartbeat. The consumer starts with a modest timeout, then extends it every so often while processing continues, so a crash is still detected quickly but long jobs are not duplicated. The maximum total visibility period is twelve hours from the first receive, which bounds the longest job a single message can represent; longer work should be split or tracked in a separate state store.
The visibility timeout can be set on the queue, applying to every receive, or on each ReceiveMessage call, overriding the queue default for that consumer. Different consumers of the same queue rarely need different timeouts, so the queue setting is usually enough, but a consumer that handles a class of slow messages can request a longer timeout explicitly. A consumer can also set visibility to zero on a message it decides not to handle right now, returning it to the queue immediately for another consumer rather than waiting for the timeout to expire.
Long polling
ReceiveMessage with a wait time of zero returns immediately, even when the queue is empty, which leads consumers to poll in tight loops, paying for each empty request and burning CPU. Long polling, with a wait time of up to 20 seconds, holds the request open until a message arrives or the wait expires. It cuts empty responses dramatically, reduces cost and lowers latency, because messages are returned as soon as they arrive rather than at the next poll. Setting the queue's receive wait time attribute to 20 seconds makes long polling the default for all consumers.
Long polling also samples all of the queue's servers rather than a subset, so it avoids the rare false empty response that short polling can produce when a message sits on a server the request did not check. For low traffic queues this matters: a consumer using short polling may report the queue empty while messages wait. There are few reasons not to use long polling, and the main one, a consumer that must return promptly to do other work between polls, is better handled by running the poller on its own thread.
Consumer concurrency deserves a deliberate choice too. A single consumer process can run several threads or async tasks, each receiving and processing messages, and a fleet of processes multiplies that. The total must stay within what downstream systems can handle, such as database connections or a partner API's rate limit, and should scale with backlog rather than staying fixed. Autoscaling consumer fleets on the queue's backlog per instance, a target tracking metric computed from visible messages divided by running consumers, keeps processing time steady through traffic swings.
Message size, retention and delays
Messages live in a queue for the retention period, four days by default and up to fourteen, after which SQS deletes them whether or not they were processed. A consumer outage longer than the retention period silently loses work, so retention should be set well beyond the longest plausible outage, and the age of the oldest message should be alarmed long before it approaches the limit. Payloads larger than 256 KB can be stored in S3 with only a pointer in the message, a pattern the extended client libraries implement. Delay queues and per message delay timers postpone delivery by up to fifteen minutes, which helps for simple backoff or scheduled retries.
Messages also carry attributes, up to ten name and value pairs outside the body, which consumers can read without parsing the payload and which SNS and EventBridge can use for filtering upstream. System attributes record the sent timestamp, the approximate receive count and the first receive timestamp. The receive count is especially useful: a consumer can log it on every message, and a message received five or six times is a strong signal of a problem, either a poison message or a timeout too short for the work, worth investigating before it reaches the dead letter queue.
Scaling and throughput
Standard queues support nearly unlimited throughput: thousands of messages per second per action, scaling automatically with no provisioning. Consumers scale out by running more pollers, and batch actions, sending, receiving and deleting up to ten messages per call, cut both request counts and cost by up to ten times. Pricing is per request, with every 64 KB chunk of payload counted as a request, so batching and long polling are the two biggest cost levers. A queue handling a billion small messages a month costs a few hundred dollars when batched and several times that when not.
Encryption is on by default with SQS managed keys, and customer managed KMS keys add control over who can decrypt, at the cost of KMS requests. The data key reuse period, five minutes by default and up to a day, controls how often SQS calls KMS, and raising it reduces KMS costs and throttling for busy queues. Access is controlled through IAM policies and queue policies, and queues shared across accounts should name specific principals and use conditions such as the source ARN, rather than allowing any principal to send.
Delete only after processing succeeds, make processing idempotent, set the visibility timeout well above the slowest normal processing time, extend it as a heartbeat for long jobs, use 20 second long polling, and set retention far beyond any plausible outage.
Loading problem...