TRIM_HORIZON and LATEST are two common shard iterator types used when working with Amazon Kinesis Data Streams. Understanding the distinction between these two iterator types is crucial for designing applications that consume data from Kinesis streams efficiently.

Introduction to Amazon Kinesis

Amazon Kinesis is a platform on AWS designed to handle real-time data streams. It provides a solution for collecting, processing, and analyzing streaming data in real-time. Among its offerings, Kinesis Data Streams allow applications to continuously capture, store, and process gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, and financial transactions.

Terminology

• Shard: A shard is the base throughput unit of an Amazon Kinesis data stream. Each shard can ingest up to 1 MB of data per second and emit up to 2 MB of data per second.
• Shard Iterator: A shard iterator is a pointer that enables you to read data from the shard in the Kinesis stream. It specifies the position in the shard from which to start reading data records.

The Kinesis API provides several types of shard iterators, with TRIM_HORIZON and LATEST being two of the most commonly used.

Understanding TRIM_HORIZON and LATEST

TRIM_HORIZON

When you use the TRIM_HORIZON iterator type, you instruct Kinesis to start reading records from the beginning of the stream. This means, regardless of when the data was ingested, your application will read all available records from the oldest to the newest.

Use Case

• Data Replay and Recovery: This iterator is ideal for replaying an entire data stream from the beginning, useful for system recovery, auditing, and for applications that process historical data.
• New Consumers: A fitting choice for a new consumer that needs to process all backlog data in the stream for historical analysis.

Example

• Real-Time Analytics: Use LATEST for applications focused on processing only new data records as they become available.
• Continuous Monitoring: Ideal for monitoring and alerting systems that need the most recent data inputs without the overhead of historical data.