How to process SQS queue with lambda function not via scheduled events?
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Introduction
You do not need a scheduled Lambda to poll SQS yourself. The normal serverless pattern is to connect the queue to Lambda with an event source mapping so that Lambda polls the queue on your behalf and invokes your function when messages are available.
The Right Architecture: SQS as a Lambda Trigger
The key idea is simple:
- producers send messages to SQS
- Lambda is configured with the queue as an event source
- AWS polls the queue and batches messages into Lambda invocations
So the function is still event-driven, just not HTTP-triggered. There is no cron job involved.
This is what most people mean when they say "process SQS with Lambda."
What the Lambda Event Looks Like
When SQS triggers the function, Lambda passes a batch of messages in the event payload:
Each record contains metadata such as:
- '
messageId' - '
body' - '
attributes' - '
receiptHandle'
Your function does not delete the message manually in the normal success case. Lambda and the event source mapping handle deletion after a successful invocation.
Creating the Trigger
The wiring can be done in the AWS console, CloudFormation, CDK, Terraform, or the CLI. The important configuration is the event source mapping between the queue ARN and the Lambda function.
Conceptually, the setup is:
- create the SQS queue
- create the Lambda function
- grant the function permission to consume the queue
- add the SQS trigger to the function
After that, publishing a message to the queue is enough to cause Lambda processing.
Batch Size and Visibility Timeout Matter
Two SQS settings influence behavior immediately:
- batch size
- visibility timeout
Batch size controls how many messages Lambda tries to deliver in one invocation. Larger batches can improve throughput, but they also make retries and failure handling coarser.
Visibility timeout must be longer than the Lambda processing time. If it is too short, a message can become visible again while the original invocation is still working on it, causing duplicate processing.
That rule is one of the most important operational details in SQS plus Lambda systems.
Handle Partial Failure Correctly
The naive mistake is to raise one exception for the whole batch when only one message failed. Modern SQS plus Lambda integrations support partial batch failure reporting so successfully processed messages are not retried unnecessarily.
A simple Python example:
This pattern is much better than failing the entire batch if one message is bad.
Idempotency Is Still Required
SQS and Lambda are designed for at-least-once delivery. That means duplicates are possible even when everything is configured correctly.
So your handler should be idempotent:
- safe to retry
- safe to process the same logical message twice
- safe under partial failures and timeouts
This is not a special corner case. It is part of the contract of the integration.
Use a DLQ or Redrive Strategy
If a message keeps failing, you need somewhere for it to go. That is where a dead-letter queue or a configured redrive policy helps.
Without that, the same bad message can poison the queue and keep triggering Lambda retries.
A healthy production setup usually includes:
- main SQS queue
- Lambda event source mapping
- failure handling in code
- dead-letter or redrive path for persistent failures
Why Scheduled Polling Is Usually Worse
A scheduled Lambda that wakes up, polls SQS, and manually deletes messages is usually the wrong design because:
- Lambda already knows how to poll SQS
- you add extra code for batching and deletion
- you increase the chance of race conditions and visibility-timeout bugs
- you lose the built-in scaling behavior of the native integration
In other words, the scheduled approach is usually more code for a worse result.
Common Pitfalls
- Using EventBridge or a cron schedule to poll SQS manually instead of using an SQS trigger.
- Setting the SQS visibility timeout too short for the Lambda runtime.
- Treating one bad message as a reason to fail the whole batch.
- Forgetting that duplicates can happen and writing non-idempotent handlers.
- Running without a dead-letter or redrive strategy for poison messages.
Summary
- The standard way to process SQS with Lambda is an event source mapping, not a scheduled poller.
- Lambda polls SQS for you and invokes the function with message batches.
- Batch size, visibility timeout, and partial batch failure handling are the key settings.
- Handlers must be idempotent because delivery is at least once.
- A dead-letter or redrive path is important for operational stability.

