Introduction

Writing Parquet to S3 from Firehose is not about uploading a .parquet file manually. The usual pattern is to send structured records into a delivery stream, enable Firehose data format conversion, and let Firehose buffer, transform, and write Parquet objects into S3.

The Core Requirements

For Firehose to convert records to Parquet, you usually need three things aligned:

• Incoming records in a structured format, often JSON.
• A schema Firehose can use for conversion.
• An S3 destination with data format conversion enabled.

If any of those pieces is missing, Firehose may still deliver data, but not as Parquet.

Why the Schema Matters

Parquet is a typed columnar format. Firehose cannot infer the full structure reliably from random incoming bytes, so it needs a schema definition. In AWS workflows this commonly means using a schema from the Glue Data Catalog.

Conceptually, the flow is:

1. Producers send JSON records.
2. Firehose buffers the records.
3. Firehose uses the schema to map fields and types.
4. Firehose writes Parquet files to S3.

That is why the setup is more than just choosing an S3 bucket and a file extension.

Example Configuration Shape

The exact infrastructure tool can vary, but the important part is the format-conversion section. A representative delivery-stream configuration looks like this:

1Resources:
2  FirehoseDeliveryStream:
3    Type: AWS::KinesisFirehose::DeliveryStream
4    Properties:
5      DeliveryStreamType: DirectPut
6      ExtendedS3DestinationConfiguration:
7        BucketARN: arn:aws:s3:::my-analytics-bucket
8        RoleARN: arn:aws:iam::123456789012:role/firehose-delivery-role
9        Prefix: parquet-output/
10        BufferingHints:
11          IntervalInSeconds: 300
12          SizeInMBs: 64
13        CompressionFormat: UNCOMPRESSED
14        DataFormatConversionConfiguration:
15          Enabled: true
16          InputFormatConfiguration:
17            Deserializer:
18              OpenXJsonSerDe: {}
19          OutputFormatConfiguration:
20            Serializer:
21              ParquetSerDe:
22                Compression: SNAPPY
23          SchemaConfiguration:
24            RoleARN: arn:aws:iam::123456789012:role/firehose-delivery-role
25            DatabaseName: analytics
26            TableName: events

The key idea is simple even if the cloud configuration looks long: JSON in, schema lookup, Parquet out.

Record Shape Still Has to Match the Schema

If your incoming JSON does not match the schema, conversion problems follow. That can mean missing fields, wrong data types, or records being routed to an error path depending on the delivery configuration.

For example, if the Glue schema expects:

• 'event_id as string'
• 'user_id as string'
• 'event_ts as timestamp'
• 'amount as double'

then the producers need to send records that actually fit that contract. Firehose is not a substitute for upstream schema discipline.

Producers Can Send Data Directly

If you are using a direct put delivery stream, a producer can send JSON records like this through the SDK:

1import boto3
2import json
3
4client = boto3.client("firehose", region_name="us-east-1")
5
6record = {
7    "event_id": "e-1001",
8    "user_id": "u-42",
9    "event_ts": "2026-03-07T12:00:00Z",
10    "amount": 19.99,
11}
12
13client.put_record(
14    DeliveryStreamName="my-firehose-stream",
15    Record={"Data": (json.dumps(record) + "\n").encode("utf-8")},
16)

This shows the producer side of the flow. Firehose still performs the buffering and S3 delivery.

Buffering Affects File Size and Latency

Firehose does not usually write one S3 object per event. It buffers records for a time interval or size threshold first. That is important because Parquet files that are too small are inefficient for analytics, while aggressive buffering increases delivery latency.

So there is a practical tradeoff:

• Larger buffers create fewer, larger Parquet files.
• Smaller buffers reduce latency but may create small-file problems.

Choose based on your analytics pattern, not by guesswork.

Common Pitfalls

• Assuming Firehose can convert arbitrary raw text to Parquet without a usable schema.
• Sending JSON that does not match the Glue schema.
• Ignoring buffering settings and then wondering why files are tiny or delayed.
• Treating file-format conversion as a replacement for upstream data validation.
• Forgetting to configure the correct IAM permissions for S3 and schema access.

Summary

• Firehose can write Parquet to S3 by using data format conversion.
• The normal setup is structured input plus a schema plus an S3 destination.
• Glue schema alignment is central because Parquet is typed.
• Producers still need to send well-formed records that match the schema.
• Buffering settings determine the latency versus file-size tradeoff of the delivered Parquet objects.