How can a org.apache.kafka.connect.data.Decimal stored in an avro file be converted to a python type?
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Introduction
Kafka Connect's Decimal logical type is typically stored in Avro as a decimal logical type backed by bytes. In Python, the natural target type is decimal.Decimal, because it preserves exact fixed-scale values without the rounding errors that would come from converting straight to float.
What Is Actually Stored in Avro
When Kafka Connect writes a decimal to Avro, the logical meaning is decimal, but the physical representation is bytes plus schema metadata such as precision and scale.
A simplified Avro field schema looks like this:
The important part is scale. It tells you how many digits belong to the right of the decimal point after decoding the unscaled integer value.
Convert Bytes to Decimal
If your Avro reader gives you raw bytes for the field, convert them by:
- interpreting the bytes as a signed big-endian integer
- scaling that integer by the negative Avro
scale
This preserves the exact decimal value, which is why Decimal is the correct Python target type for finance or other precision-sensitive data.
Example with fastavro
Here is a practical pattern using fastavro to read records and then convert the raw Avro bytes when needed.
The exact schema lookup in real code depends on where the decimal field sits in the record, but the conversion logic stays the same.
Your Avro Library May Already Decode It
Before writing conversion code, inspect the value you actually get back. Some Avro libraries or configurations may already return a Python Decimal for Avro decimal logical types.
If the library already returns Decimal, you do not need to manually interpret the bytes. The manual conversion is mainly for cases where the reader exposes the underlying Avro bytes instead of the logical type.
Do Not Convert to float Unless Approximation Is Acceptable
It is technically possible to write:
but that throws away one of the main reasons the field was stored as a decimal logical type in the first place. Binary floating-point values cannot exactly represent many decimal fractions, so float is usually the wrong end type for money or precise measurements.
Validate Scale and Field Mapping Carefully
Conversion bugs often come from using the wrong schema field or wrong scale rather than from the byte-to-decimal math itself. Make sure you:
- read the correct field schema
- use the correct
scale - treat the bytes as signed and big-endian
If any of those assumptions is wrong, the number will decode incorrectly even though the code "looks right."
Common Pitfalls
The most common mistake is converting the bytes directly to float and losing exact precision. Another is forgetting that Avro decimal uses schema metadata, so the bytes alone are not enough without the field's scale. Developers also sometimes assume the Avro reader always returns raw bytes when some libraries can already decode the logical type for them. Finally, using unsigned interpretation instead of signed big-endian decoding produces incorrect negative values and large-number errors.
Summary
- Kafka Connect decimal data stored in Avro should usually become Python
decimal.Decimal. - Avro represents the value as signed bytes plus decimal metadata such as
scale. - Decode the bytes to an integer and apply the Avro scale to produce a
Decimal. - Check whether your Avro library already handles decimal logical types for you.
- Avoid
floatunless approximation is acceptable for the application.

