A Kafka record's value is a byte array, and the broker neither knows nor cares what those bytes mean. Schema Registry is the piece that gives them meaning, and it does it with a mechanism small enough to describe exactly: five extra bytes on the front of every record and an HTTP service that maps a number to a schema.

The five bytes

The whole wire format, and knowing it makes every registry problem debuggable.

A value serialised by KafkaAvroSerializer is a magic byte of 0x00, then a 4-byte big-endian schema id, then the Avro binary encoding of the payload. There is no schema in the record and no field names, which is the point: Avro's binary encoding writes field values positionally, so the schema is what makes the bytes parseable at all.

That is why a record read without the registry is unintelligible rather than merely awkward. It is also why the overhead is five bytes rather than the hundreds that embedding a schema would cost, and why a topic of small records carrying JSON with field names is several times larger than the same data in Avro.

Protobuf adds one thing: after the schema id comes a message index array, a varint-encoded path identifying which message type inside the schema file this record is, because a .proto file can define many. The common case of one top-level message is encoded as a single zero byte.

So decoding a record by hand is: read byte 0 and confirm it is zero, read bytes 1 to 4 as a big-endian integer, fetch that id from GET /schemas/ids/<id>, and decode the remainder against it. That sequence is worth being able to do, because it settles arguments about what a topic actually contains.

Subjects, which are the unit of compatibility

The concept most often confused with the topic.

A subject is a named sequence of schema versions, and compatibility is enforced per subject rather than per topic or per schema. The subject name is decided by a subject name strategy on the serialiser.

TopicNameStrategy, the default, names the subject <topic>-value for values and <topic>-key for keys. One topic therefore has two independent compatibility histories, and a topic carrying several unrelated record types under this strategy produces a subject whose versions alternate between them, which fails compatibility checks immediately and confusingly.

RecordNameStrategy names the subject after the record's fully qualified name, so one record type has one history regardless of which topics carry it, and a topic can carry many types. TopicRecordNameStrategy combines both, giving <topic>-<recordname>, which is the right answer for a topic deliberately carrying multiple event types with independent evolution.

The strategy is set with value.subject.name.strategy, and changing it on a live topic starts a new subject with no history, which means the new subject's first version passes any compatibility check regardless of what the old one said. That is a real way to accidentally break consumers while the registry reports everything as compatible.

Registration, and the setting that should usually be off

One default that is convenient in development and wrong in production.

auto.register.schemas defaults to true, so a producer whose local schema is not in the registry registers it, subject to the compatibility check. That means any producer with write access to the registry can add a schema version, and the first deployment of a new service defines the contract by accident rather than by review.

Setting it to false makes the producer look up its schema and fail if it is not already registered, which moves schema changes into whatever process registers them: a CI step, a Gradle or Maven plugin, or a reviewed pull request against a schema repository. That is the configuration a governed pipeline uses.

use.latest.version, default false, is its companion and is often misunderstood. With auto-registration off and this set to true, the producer serialises with the subject's latest registered version rather than its own compiled schema, which is how a producer can emit records conforming to a schema newer than the one it was built against. It is useful with schema references and is a trap otherwise, because the producer is now writing a shape its own code does not know.

latest.compatibility.strict, default true, controls whether that lookup also verifies the producer's schema is compatible with the latest, and turning it off removes the only guard on the combination above.

The client-side cache

Because the registry is on the data path and this is what keeps it off.

Both serialiser and deserialiser cache the schema-to-id mapping in memory, so steady-state serialisation makes no HTTP calls at all. The cache is bounded by max.schemas.per.subject, default 1,000, which is generous for any sane subject.

The consequence is that a registry outage does not immediately break a running application. Producers and consumers continue with cached schemas and fail only when they encounter an id they have not seen, which in practice means a registry outage breaks deployments and new schema versions rather than existing traffic.

It also means the first records after a restart carry a burst of registry calls, one per distinct schema id in flight. An application restarting across a hundred instances produces a hundred times that burst simultaneously, which is a real load pattern on a registry sized for steady state.

The deserialiser's cache is keyed on the id from the record, so it fetches exactly the writer's schema and no more. That is the mechanism that makes evolution work at all, and it is the subject of the next section.

What the HTTP API offers

Six calls, which are the whole operational surface.

GET /subjects lists subjects and GET /subjects/<s>/versions lists a subject's versions, which together are how you discover what a cluster's contracts actually are rather than what a repository claims.

GET /subjects/<s>/versions/<v> returns a specific version, with latest accepted as the version, and GET /schemas/ids/<id> returns a schema by id, which is the call for decoding a record you have in hand.

POST /compatibility/subjects/<s>/versions/latest checks a candidate schema without registering it, and it is the call that belongs in CI. Running it in a pull request turns a runtime serialisation failure into a build failure, which is the single highest-value integration between a schema repository and a registry.

PUT /config/<s> sets a subject's compatibility level and GET /config reads the global default. Per-subject configuration overriding a global default is normal and is also how a cluster ends up with subjects nobody realises are set to NONE.

Decoding a record by hand, as a worked example

Because doing it once makes the format permanent knowledge.

Suppose kafka-console-consumer.sh run without a deserialiser prints a value whose first bytes, in hexadecimal, are 00 00 00 01 2f. Byte 0 is 00, so this is registry-framed. Bytes 1 through 4 are 00 00 01 2f, which as a big-endian integer is 303. Everything from byte 5 onward is Avro binary.

curl -s localhost:8081/schemas/ids/303 returns a JSON object whose schema field holds the Avro schema as an escaped string. That schema is the writer's schema for this record, and GET /schemas/ids/303/versions names which subjects and versions share that id, which is how you learn which topic contract this record belongs to.

The useful property is that the id is global across the registry rather than per subject. Two subjects whose schemas are textually identical share one id, which occasionally confuses people reading the output of the versions call above and finding more subjects than they expected.

The reason this procedure matters is that it is the only way to answer "what is actually in this topic" without trusting an application's configuration. A team arguing about whether a producer deployed its new schema can settle it in three commands, and the answer is not subject to anyone's recollection of what was released.

The settings a producer and a consumer each need

Short lists, and they are asymmetric.

A producer needs schema.registry.url, key.serializer and value.serializer set to the registry serialisers, auto.register.schemas set to false in any governed environment, and value.subject.name.strategy if the topic is not single-type. Under security it also needs basic.auth.credentials.source and basic.auth.user.info, or the corresponding bearer-token settings.

A consumer needs schema.registry.url, the matching deserialisers, and specific.avro.reader if it wants generated classes rather than a GenericRecord. It needs no subject strategy at all, because it reads the schema by id rather than by subject, which is a detail that surprises people the first time they look for the setting and find it absent.

That asymmetry is the subject model showing through: subjects exist to organise compatibility checks, which happen at registration time on the producer side. A consumer never consults a subject during normal operation.

The one consumer-side registry setting worth being deliberate about is the registry URL list, which accepts several comma-separated entries and is how a consumer survives one registry instance being unreachable without a load balancer in front.