Practice Now
protobuf
Python
file-saving
data-serialization
programming-debugging

Trouble saving repeated protobuf object to file Python

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
PythonDebugging & Errors

Introduction

Protocol Buffers (protobuf) does not natively support writing multiple messages to a single file — there is no built-in delimiter between messages in the binary format. When you serialize multiple protobuf objects to a file, they are concatenated as raw bytes, and reading them back requires knowing where each message ends and the next begins. The standard solution is to prefix each message with its length (length-delimited format). The google.protobuf Python library does not provide this out of the box, so you must implement it manually.

The Problem

protobuf
1// person.proto
2syntax = "proto3";
3
4message Person {
5  string name = 1;
6  int32 age = 2;
7  repeated string hobbies = 3;
8}
python
1from person_pb2 import Person
2
3# Create multiple messages
4people = [
5    Person(name="Alice", age=30, hobbies=["reading", "hiking"]),
6    Person(name="Bob", age=25, hobbies=["gaming"]),
7    Person(name="Charlie", age=35, hobbies=["cooking", "cycling"]),
8]
9
10# WRONG — writing multiple messages back-to-back
11with open("people.bin", "wb") as f:
12    for person in people:
13        f.write(person.SerializeToString())
14
15# WRONG — reading back produces garbage
16with open("people.bin", "rb") as f:
17    data = f.read()
18    person = Person()
19    person.ParseFromString(data)
20    # Only parses the first message (or garbage) — no way to know boundaries

The binary data of multiple messages is concatenated without separators. ParseFromString cannot determine where one message ends and the next begins.

Prefix each message with its byte length using varint encoding:

python
1from google.protobuf.internal.encoder import _EncodeVarint
2from google.protobuf.internal.decoder import _DecodeVarint
3from person_pb2 import Person
4
5def write_delimited(file, message):
6    """Write a length-delimited protobuf message to a file."""
7    data = message.SerializeToString()
8    _EncodeVarint(file.write, len(data))  # Write length as varint
9    file.write(data)                       # Write message bytes
10
11def read_delimited(file, message_class):
12    """Read a length-delimited protobuf message from a file."""
13    buf = file.read(1)
14    if not buf:
15        return None  # EOF
16    # Decode varint (may need more bytes)
17    size, pos = _DecodeVarint(buf, 0)
18    while pos == 0:
19        buf += file.read(1)
20        size, pos = _DecodeVarint(buf, 0)
21    remaining = size - (len(buf) - pos)
22    data = buf[pos:] + file.read(remaining)
23    message = message_class()
24    message.ParseFromString(data)
25    return message
26
27# Write multiple messages
28people = [
29    Person(name="Alice", age=30, hobbies=["reading", "hiking"]),
30    Person(name="Bob", age=25, hobbies=["gaming"]),
31    Person(name="Charlie", age=35, hobbies=["cooking", "cycling"]),
32]
33
34with open("people.bin", "wb") as f:
35    for person in people:
36        write_delimited(f, person)
37
38# Read them back
39with open("people.bin", "rb") as f:
40    while True:
41        person = read_delimited(f, Person)
42        if person is None:
43            break
44        print(f"{person.name}, {person.age}, {list(person.hobbies)}")
45# Alice, 30, ['reading', 'hiking']
46# Bob, 25, ['gaming']
47# Charlie, 35, ['cooking', 'cycling']

Solution 2: Fixed-Size Length Prefix

Simpler but less space-efficient for small messages:

python
1import struct
2from person_pb2 import Person
3
4def write_with_length(file, message):
5    data = message.SerializeToString()
6    file.write(struct.pack('>I', len(data)))  # 4-byte big-endian length
7    file.write(data)
8
9def read_with_length(file, message_class):
10    length_bytes = file.read(4)
11    if len(length_bytes) < 4:
12        return None
13    length = struct.unpack('>I', length_bytes)[0]
14    data = file.read(length)
15    message = message_class()
16    message.ParseFromString(data)
17    return message
18
19# Write
20with open("people.bin", "wb") as f:
21    for person in people:
22        write_with_length(f, person)
23
24# Read
25with open("people.bin", "rb") as f:
26    while True:
27        person = read_with_length(f, Person)
28        if person is None:
29            break
30        print(person.name)

Solution 3: Wrapper Message with Repeated Field

Instead of writing multiple messages, wrap them in a container message:

protobuf
1// people.proto
2syntax = "proto3";
3
4message Person {
5  string name = 1;
6  int32 age = 2;
7  repeated string hobbies = 3;
8}
9
10message PeopleList {
11  repeated Person people = 1;
12}
python
1from people_pb2 import Person, PeopleList
2
3# Create a container
4people_list = PeopleList()
5people_list.people.add(name="Alice", age=30)
6people_list.people.add(name="Bob", age=25)
7people_list.people.add(name="Charlie", age=35)
8
9# Single write — no delimiter needed
10with open("people.bin", "wb") as f:
11    f.write(people_list.SerializeToString())
12
13# Single read
14with open("people.bin", "rb") as f:
15    loaded = PeopleList()
16    loaded.ParseFromString(f.read())
17    for person in loaded.people:
18        print(f"{person.name}, {person.age}")

This is the simplest approach but requires all data to fit in memory at once.

Working with Repeated Fields

python
1from person_pb2 import Person
2
3person = Person(name="Alice", age=30)
4
5# Add to repeated field
6person.hobbies.append("reading")
7person.hobbies.append("hiking")
8person.hobbies.extend(["cooking", "cycling"])
9
10# Cannot assign directly — this DOES NOT work:
11# person.hobbies = ["reading", "hiking"]  # AttributeError
12
13# Must use slice assignment or extend on empty
14del person.hobbies[:]  # Clear
15person.hobbies.extend(["reading", "hiking"])
16
17# Iterate
18for hobby in person.hobbies:
19    print(hobby)
20
21# Check length
22print(len(person.hobbies))  # 2

Solution 4: JSON Lines (One JSON per Line)

For human-readable storage:

python
1from google.protobuf.json_format import MessageToJson, Parse
2from person_pb2 import Person
3
4# Write as JSON Lines
5with open("people.jsonl", "w") as f:
6    for person in people:
7        json_str = MessageToJson(person, preserving_proto_field_name=True)
8        f.write(json_str.replace("\n", " ") + "\n")  # One line per message
9
10# Read back
11with open("people.jsonl", "r") as f:
12    for line in f:
13        person = Parse(line, Person())
14        print(person.name)

Common Pitfalls

  • Writing multiple messages without length delimiters: Raw SerializeToString() output has no message boundary markers. Concatenating multiple messages and then reading produces corrupt data. Always use length-delimited format or a wrapper message.
  • Using MergeFromString instead of ParseFromString: MergeFromString appends to existing data in the message instead of replacing it. If you reuse a message object in a read loop without clearing it, fields accumulate. Use ParseFromString (which clears first) or call message.Clear() before MergeFromString.
  • Assigning to repeated fields directly: person.hobbies = ["a", "b"] raises AttributeError in protobuf. Repeated fields are special containers that must be modified with .append(), .extend(), or slice assignment. Use del person.hobbies[:] to clear and .extend() to set new values.
  • Forgetting to compile .proto files: The Python classes (person_pb2.py) are generated by protoc --python_out=. person.proto. If you modify the .proto file, you must regenerate the Python module. Importing a stale _pb2.py file causes field mismatches.
  • Large files with the wrapper approach: Putting all messages in a PeopleList container requires loading everything into memory at once. For millions of records, use length-delimited streaming instead.

Summary

  • Protobuf has no built-in multi-message file format — you must delimit messages yourself
  • Use varint length prefix (_EncodeVarint/_DecodeVarint) for standard length-delimited format
  • Alternatively, use struct.pack('>I', length) for fixed 4-byte length prefixes
  • Wrap multiple messages in a container message (repeated field) for simple cases
  • Use .append() and .extend() for repeated fields — direct assignment does not work
  • Regenerate _pb2.py files with protoc after modifying .proto definitions
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.