Introduction

Dremel's repetition and definition levels are the key idea that lets nested records be stored column by column without losing structural meaning. If you have ever wondered how a system can scan one nested field efficiently without fully flattening every row into a huge denormalized table, these two level values are the answer.

The point is not just compression. The levels encode whether a nested value exists and whether it begins a new repeated group, which lets a query engine reconstruct hierarchy while reading only the columns it needs.

Why Dremel Needs Extra Levels

Suppose a record contains optional fields and repeated lists. In a flat columnar store, a plain sequence of scalar values is not enough to recover which values belonged to which parent record and which optional fields were missing.

Dremel solves this by storing each leaf column together with:

• a definition level, which says how far down the schema path the value is actually defined,
• a repetition level, which says at what repeated nesting depth the current value continues the same repeated structure.

Without those extra integers, a column reader would know the values but not the nesting structure.

A Small Nested Example

Consider this schema idea:

• 'person has an optional name'
• 'person has a repeated phone'
• each phone has an optional number

Example data:

1[
2  {
3    "name": "Ana",
4    "phone": [
5      { "number": "111" },
6      { "number": "222" }
7    ]
8  },
9  {
10    "name": "Ben",
11    "phone": [
12      {}
13    ]
14  },
15  {
16    "name": "Cara"
17  }
18]

Now focus only on the leaf column phone.number. The values alone would be:

111, 222, null, null

That sequence by itself is ambiguous. The levels are what tell you which null means "phone entry exists but number is missing" and which null means "person had no phone entries at all."

Definition Level

The definition level counts how many optional or repeated nodes along the path are actually present for the current leaf.

For phone.number, the maximum definition level in this toy schema is reached when:

• the repeated phone entry exists,
• the optional number field exists.

If the phone entry exists but number is missing, the definition level is lower. If there is no phone entry at all for that person, it is lower still.

That is why definition level answers the question, "How much of the path to this leaf is defined?"

Repetition Level

The repetition level is about repeated structures. It tells the reader whether the current value starts a new parent record or continues an existing repeated group.

In the example above, the second phone number for Ana belongs to the same top-level person record as the first one. Its repetition level indicates that the repeated phone field is continuing rather than beginning a brand-new parent row.

That is why repetition level answers the question, "At what repeated depth am I still inside the same repeated collection?"

A Conceptual Encoding Sketch

For the phone.number leaf, a conceptual view might look like this:

1value   definition  repetition
2111     2           0
3222     2           1
4null    1           0
5null    0           0

Interpretation:

• '111 is fully defined and starts the first phone entry for Ana.'
• '222 is also fully defined, but repetition level 1 means it continues the repeated phone list for the same person.'
• the next null means Ben has a phone entry but the optional number field is missing.
• the final null means Cara has no phone entry at all.

The exact numeric levels depend on the schema path, but this pattern is the essence.

Why This Matters for Query Engines

Because each leaf column carries enough structural information, Dremel-style systems can scan nested data selectively. A query touching only one leaf column does not need to materialize the whole document structure upfront.

This makes nested analytics practical in columnar storage systems such as those inspired by Dremel. The engine gets the scan efficiency of columns while still being able to reconstruct repeated and optional records when necessary.

Common Pitfalls

• Thinking repetition level is just a row counter. It is specifically about repeated nesting depth.
• Treating definition level as a simple null flag. It distinguishes multiple kinds of absence along a schema path.
• Forgetting that the maximum definition level depends on the schema, not on the value alone.
• Looking only at the leaf values and assuming the original nested structure is obvious.
• Mixing up "missing repeated field" with "present repeated field containing a missing optional leaf."

Summary

• Dremel stores nested leaf columns together with repetition and definition levels.
• Definition level describes how much of the schema path to the leaf is actually present.
• Repetition level describes whether a repeated structure is continuing or restarting.
• These levels let a columnar engine reconstruct nested records without fully flattening them.
• The idea is central to efficient querying of nested data in Dremel-style systems.