Introduction

When an object is created or deleted in S3, code running inside a Kubernetes cluster often needs to react. The reliable pattern is usually not “have S3 call a pod directly,” but to place a durable AWS service such as SQS or EventBridge between S3 and the cluster. That gives you retries, buffering, and a cleaner security model.

Prefer Queue-Based Delivery

S3 can publish event notifications to services such as SQS, SNS, Lambda, and EventBridge. Inside Kubernetes, SQS is often the easiest target because pods can poll it safely without exposing an inbound HTTP endpoint.

A common production flow is:

1. S3 emits an object event
2. S3 sends it to SQS
3. a deployment in the cluster polls SQS
4. the worker processes the event and deletes the message

This pattern is resilient because the queue absorbs spikes and lets workers recover after restarts.

Why Direct Webhooks Are Usually Weaker

You can route events through API Gateway or another public endpoint and forward them to the cluster, but that usually creates more operational work:

• you need ingress and authentication
• transient pod failures can cause delivery issues
• retries become your problem instead of the queue’s problem

A direct webhook is still reasonable when you need immediate push delivery, but for most internal processing pipelines SQS is simpler.

Example Worker Using SQS

A small Python worker can poll SQS and read the S3 event payload.

1import json
2import boto3
3
4sqs = boto3.client("sqs", region_name="us-east-1")
5QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/my-s3-events"
6
7while True:
8    response = sqs.receive_message(
9        QueueUrl=QUEUE_URL,
10        MaxNumberOfMessages=10,
11        WaitTimeSeconds=20,
12    )
13
14    for msg in response.get("Messages", []):
15        body = json.loads(msg["Body"])
16        record = body["Records"][0]
17        bucket = record["s3"]["bucket"]["name"]
18        key = record["s3"]["object"]["key"]
19
20        print(f"Process s3://{bucket}/{key}")
21
22        sqs.delete_message(
23            QueueUrl=QUEUE_URL,
24            ReceiptHandle=msg["ReceiptHandle"],
25        )

This is intentionally simple, but it demonstrates the important contract: process the event and delete the message only after success.

Give Pods AWS Access Correctly

In Kubernetes, the worker needs permission to read from SQS and often permission to fetch the referenced object from S3. On EKS, the cleanest approach is usually IAM Roles for Service Accounts, often called IRSA.

A service account can then be attached to the deployment:

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: s3-event-worker
5  annotations:
6    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3-event-worker
7---
8apiVersion: apps/v1
9kind: Deployment
10metadata:
11  name: s3-event-worker
12spec:
13  replicas: 2
14  selector:
15    matchLabels:
16      app: s3-event-worker
17  template:
18    metadata:
19      labels:
20        app: s3-event-worker
21    spec:
22      serviceAccountName: s3-event-worker
23      containers:
24        - name: worker
25          image: my-registry/s3-event-worker:latest

That avoids static AWS keys inside the cluster.

Think About Idempotency

S3 notifications are not a guarantee of exactly-once processing at the application level. Your consumer should be idempotent.

That means if the same event appears twice, the result should still be safe. For example:

• processing a file only if its output does not already exist
• storing processed object versions in a database
• using the object key plus event time as a deduplication key

Without idempotency, retries become bugs.

Alternative Architectures

SQS is a strong default, but it is not the only design:

• 'S3 -> Lambda -> internal API when preprocessing or filtering is easier in Lambda'
• 'S3 -> EventBridge -> multiple targets when several consumers need the same event stream'
• 'S3 -> SNS -> SQS when fan-out is needed with independent queues'

Choose based on delivery pattern, not on whichever AWS service you used last time.

Common Pitfalls

• Exposing a pod directly to S3 events when a queue would be safer and simpler.
• Giving pods static AWS credentials instead of using workload identity such as IRSA.
• Deleting SQS messages before processing succeeds.
• Assuming S3 event delivery is exactly once and skipping idempotency.
• Ignoring backpressure when a burst of uploads can overwhelm a small worker deployment.

Summary

• The usual Kubernetes pattern is S3 -> SQS -> worker pod.
• Queue-based delivery is easier to secure and more resilient than direct inbound webhooks.
• Pods should use workload identity, not embedded AWS keys.
• Consumers must be idempotent because retries and duplicates are normal distributed-system behavior.
• Pick EventBridge, SNS, or Lambda only when the event-routing requirements justify the extra complexity.