AWS Lambda
SQS Queue
Serverless Architecture
Event-Driven Computing
Cloud Computing

How to process SQS queue with lambda function not via scheduled events?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

You do not need a scheduled Lambda to poll SQS yourself. The normal serverless pattern is to connect the queue to Lambda with an event source mapping so that Lambda polls the queue on your behalf and invokes your function when messages are available.

The Right Architecture: SQS as a Lambda Trigger

The key idea is simple:

  • producers send messages to SQS
  • Lambda is configured with the queue as an event source
  • AWS polls the queue and batches messages into Lambda invocations

So the function is still event-driven, just not HTTP-triggered. There is no cron job involved.

This is what most people mean when they say "process SQS with Lambda."

What the Lambda Event Looks Like

When SQS triggers the function, Lambda passes a batch of messages in the event payload:

python
1def handler(event, context):
2    for record in event["Records"]:
3        body = record["body"]
4        print(body)
5
6    return {"statusCode": 200}

Each record contains metadata such as:

  • 'messageId'
  • 'body'
  • 'attributes'
  • 'receiptHandle'

Your function does not delete the message manually in the normal success case. Lambda and the event source mapping handle deletion after a successful invocation.

Creating the Trigger

The wiring can be done in the AWS console, CloudFormation, CDK, Terraform, or the CLI. The important configuration is the event source mapping between the queue ARN and the Lambda function.

Conceptually, the setup is:

  1. create the SQS queue
  2. create the Lambda function
  3. grant the function permission to consume the queue
  4. add the SQS trigger to the function

After that, publishing a message to the queue is enough to cause Lambda processing.

Batch Size and Visibility Timeout Matter

Two SQS settings influence behavior immediately:

  • batch size
  • visibility timeout

Batch size controls how many messages Lambda tries to deliver in one invocation. Larger batches can improve throughput, but they also make retries and failure handling coarser.

Visibility timeout must be longer than the Lambda processing time. If it is too short, a message can become visible again while the original invocation is still working on it, causing duplicate processing.

That rule is one of the most important operational details in SQS plus Lambda systems.

Handle Partial Failure Correctly

The naive mistake is to raise one exception for the whole batch when only one message failed. Modern SQS plus Lambda integrations support partial batch failure reporting so successfully processed messages are not retried unnecessarily.

A simple Python example:

python
1def handler(event, context):
2    failures = []
3
4    for record in event["Records"]:
5        try:
6            process_message(record["body"])
7        except Exception:
8            failures.append({"itemIdentifier": record["messageId"]})
9
10    return {"batchItemFailures": failures}
11
12
13def process_message(body):
14    print("processing:", body)

This pattern is much better than failing the entire batch if one message is bad.

Idempotency Is Still Required

SQS and Lambda are designed for at-least-once delivery. That means duplicates are possible even when everything is configured correctly.

So your handler should be idempotent:

  • safe to retry
  • safe to process the same logical message twice
  • safe under partial failures and timeouts

This is not a special corner case. It is part of the contract of the integration.

Use a DLQ or Redrive Strategy

If a message keeps failing, you need somewhere for it to go. That is where a dead-letter queue or a configured redrive policy helps.

Without that, the same bad message can poison the queue and keep triggering Lambda retries.

A healthy production setup usually includes:

  • main SQS queue
  • Lambda event source mapping
  • failure handling in code
  • dead-letter or redrive path for persistent failures

Why Scheduled Polling Is Usually Worse

A scheduled Lambda that wakes up, polls SQS, and manually deletes messages is usually the wrong design because:

  • Lambda already knows how to poll SQS
  • you add extra code for batching and deletion
  • you increase the chance of race conditions and visibility-timeout bugs
  • you lose the built-in scaling behavior of the native integration

In other words, the scheduled approach is usually more code for a worse result.

Common Pitfalls

  • Using EventBridge or a cron schedule to poll SQS manually instead of using an SQS trigger.
  • Setting the SQS visibility timeout too short for the Lambda runtime.
  • Treating one bad message as a reason to fail the whole batch.
  • Forgetting that duplicates can happen and writing non-idempotent handlers.
  • Running without a dead-letter or redrive strategy for poison messages.

Summary

  • The standard way to process SQS with Lambda is an event source mapping, not a scheduled poller.
  • Lambda polls SQS for you and invokes the function with message batches.
  • Batch size, visibility timeout, and partial batch failure handling are the key settings.
  • Handlers must be idempotent because delivery is at least once.
  • A dead-letter or redrive path is important for operational stability.

Course illustration
Course illustration

All Rights Reserved.