KSQL streams - Get data from Array of Struct
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Introduction
When a ksqlDB stream contains an ARRAY<STRUCT> field, the usual question is whether you need one specific element or one output row per element. Once that distinction is clear, the access pattern becomes much easier: index into the array for one element, or explode the array when each struct should be treated as its own record.
Start from the Declared Schema
Suppose the source stream contains an order with an array of item structs.
The items column is not a flat value. It is an array where each element is itself a struct. That means nested access usually happens in two stages: select an array element, then read a field from that struct.
Access One Known Element by Index
If you only care about one item position, index into the array and then project the desired field.
This pattern is useful when the schema guarantees that a certain position has a specific meaning. In most event streams, though, array positions are data-dependent, so direct indexing is more limited than it first appears.
Flatten the Array with EXPLODE
When you need to process each array element independently, flatten the array into one row per struct.
After that, the alias item behaves like a struct column whose fields can be projected normally.
This is the pattern to use when later steps need filtering, aggregation, or joins at the item level.
Build an Intermediate Stream for Clarity
Many nested queries become much easier to maintain if you split them into stages.
Now downstream queries can work with the flatter shape instead of repeating nested expressions everywhere.
This two-step approach is often easier to debug because you can inspect the intermediate stream on its own.
Filter and Aggregate After Flattening
Once the array elements have become rows, normal stream logic becomes easier.
Trying to aggregate directly inside a nested array shape is much harder to read and reason about. Flattening early is often the better modeling choice.
Debug Schema Problems First
Nested access failures are often schema problems rather than query problems. If a field cannot be read, verify:
- the stream schema matches the incoming payload
- the array really contains structs, not plain strings or maps
- the field names inside the struct are correct
- the chosen value format deserializes nested data as expected
If the schema and payload do not align, no amount of query tweaking will fix the result.
Common Pitfalls
A common mistake is trying to access items->name directly even though items is an array, not a struct. You need indexing or explosion first.
Another is overusing direct indexing in data where array length and order vary between records. That usually creates brittle queries.
Teams also tend to write one very dense query when a small intermediate stream would be much clearer and easier to debug.
Summary
- For
ARRAY<STRUCT>, decide whether you need one element or one row per element. - Use indexing when you truly want a specific position.
- Use
EXPLODEwhen each struct should become its own output row. - Create an intermediate flattened stream when downstream logic is easier on a row-wise shape.
- Check schema alignment first when nested field access fails.

