Introduction

AWS APIs often return partial results with continuation tokens. Boto3 paginators handle this pattern for you, but you still need to control page size, filtering, and retry behavior. Using paginators correctly prevents incomplete scans and memory-heavy code.

Basic Paginator Flow

Create a client, request a paginator for the API operation, and iterate over pages. Process each page incrementally rather than collecting everything at once.

1import boto3
2
3s3 = boto3.client("s3")
4
5
6def list_all_keys(bucket: str, prefix: str = ""):
7    paginator = s3.get_paginator("list_objects_v2")
8
9    for page in paginator.paginate(
10        Bucket=bucket,
11        Prefix=prefix,
12        PaginationConfig={"PageSize": 1000},
13    ):
14        for obj in page.get("Contents", []):
15            yield obj["Key"]
16
17
18if __name__ == "__main__":
19    for key in list_all_keys("my-example-bucket", "logs/"):
20        print(key)

This pattern is memory efficient because it streams keys page by page.

Control Limits and Resume Position

You can limit total results or resume from a saved token. This is useful for jobs that run in intervals and should continue where they stopped.

1import boto3
2
3client = boto3.client("dynamodb")
4
5
6def scan_table_incremental(table_name: str, start_token=None, max_items=2000):
7    paginator = client.get_paginator("scan")
8    iterator = paginator.paginate(
9        TableName=table_name,
10        PaginationConfig={
11            "StartingToken": start_token,
12            "MaxItems": max_items,
13            "PageSize": 100,
14        },
15    )
16
17    last_token = None
18    count = 0
19    for page in iterator:
20        items = page.get("Items", [])
21        count += len(items)
22        last_token = page.get("NextToken")
23        print(f"Processed {len(items)} items")
24
25    return count, last_token

Persisting resume tokens makes long-running data jobs far more resilient.

Build a Reusable Pagination Utility

A small utility wrapper reduces duplication across services and encourages consistent logging.

1from typing import Iterator, Dict, Any
2
3
4def paginate(client, operation: str, result_key: str, **kwargs) -> Iterator[Dict[str, Any]]:
5    paginator = client.get_paginator(operation)
6    for page in paginator.paginate(**kwargs):
7        for item in page.get(result_key, []):
8            yield item

Use this helper for S3 objects, DynamoDB scans, and many other AWS operations that follow the same pagination shape.

Add Robust Error Handling for Long Runs

Paginated operations can run for minutes or hours. Network hiccups and throttling are normal, so job code should include retries and clear progress logging. You do not want to restart from zero on a transient failure.

1import time
2import botocore
3
4
5def consume_with_retry(iterator_factory, max_attempts=5):
6    attempt = 1
7    while True:
8        try:
9            for item in iterator_factory():
10                yield item
11            return
12        except botocore.exceptions.ClientError as exc:
13            if attempt >= max_attempts:
14                raise
15
16            wait = 2 ** attempt
17            print(f"Paginator call failed: {exc}. Retrying in {wait}s")
18            time.sleep(wait)
19            attempt += 1
20
21
22# Example usage
23# items = consume_with_retry(lambda: paginate(client, "scan", "Items", TableName="users"))

Combining retries with continuation tokens gives you a durable pagination workflow suitable for production batch jobs.

Monitor Progress and Throughput

For large datasets, add periodic progress logs with page counts and item counts. This helps operators see that the job is healthy and estimate remaining time. It also makes troubleshooting easier when a run stalls or slows down.

Good pagination code is not only correct, it is observable. Add metrics early so batch behavior is transparent.

Validate Permissions Early

Paginator code can fail mid-run if IAM permissions are incomplete for certain resources. Run a small permission probe before full execution so failures happen quickly and clearly.

Short dry runs against a test prefix can reveal pagination mistakes before full-scale processing.

Common Pitfalls

• Calling a list API once and assuming all records were returned.
• Loading all pages into memory before processing results.
• Forgetting to handle empty page keys such as missing Contents in S3 responses.
• Ignoring retry and timeout settings for long paginated jobs.

Summary

• Use Boto3 paginators to iterate complete result sets safely.
• Process items incrementally to reduce memory pressure.
• Store continuation tokens for resumable batch jobs.
• Standardize paginator usage with a small shared helper.