Simple Python Challenge Fastest Bitwise XOR on Data Buffers

Python

Bitwise Operations

XOR

Data Buffers

Programming Challenge

Simple Python Challenge Fastest Bitwise XOR on Data Buffers

Data Structures & Algorithms practice on Codemia

Step through 300+ coding interview problems with animated visualisers that show the data structure changing as the code runs. The Blind 75 are free.

Practice algorithms

Python

Introduction

XORing two data buffers byte-by-byte in pure Python is slow because of per-byte loop overhead. The fastest approaches convert buffers to Python int (which supports arbitrary-width XOR natively), use NumPy's vectorized operations, or leverage bytes with int.from_bytes/int.to_bytes. For large buffers, NumPy is 100-1000x faster than a Python for-loop.

The Naive Approach (Slow)

python

1def xor_naive(buf1, buf2):
2    return bytes(a ^ b for a, b in zip(buf1, buf2))
3
4a = b'\x01\x02\x03\x04' * 10000
5b = b'\xff\xfe\xfd\xfc' * 10000
6
7result = xor_naive(a, b)
8# Works but very slow for large buffers

This creates a generator that XORs one byte at a time. Python's per-iteration overhead makes this O(n) with a large constant factor.

Method 1: int.from_bytes XOR (Fast, No Dependencies)

Convert both buffers to a single large integer, XOR them, and convert back:

python

1def xor_int(buf1, buf2):
2    int1 = int.from_bytes(buf1, byteorder='big')
3    int2 = int.from_bytes(buf2, byteorder='big')
4    xored = int1 ^ int2
5    return xored.to_bytes(len(buf1), byteorder='big')
6
7a = b'\x01\x02\x03\x04' * 10000
8b = b'\xff\xfe\xfd\xfc' * 10000
9
10result = xor_int(a, b)

Python's arbitrary-precision integer XOR is implemented in C and operates on machine words (8 bytes at a time), making it much faster than byte-by-byte iteration.

Method 2: NumPy (Fastest for Large Buffers)

python

1import numpy as np
2
3def xor_numpy(buf1, buf2):
4    arr1 = np.frombuffer(buf1, dtype=np.uint8)
5    arr2 = np.frombuffer(buf2, dtype=np.uint8)
6    return (arr1 ^ arr2).tobytes()
7
8a = b'\x01\x02\x03\x04' * 10000
9b = b'\xff\xfe\xfd\xfc' * 10000
10
11result = xor_numpy(a, b)

NumPy uses SIMD (Single Instruction Multiple Data) instructions to XOR 16-64 bytes per CPU instruction. This is the fastest option for buffers over a few kilobytes.

Method 3: bytearray with memoryview

python

1def xor_bytearray(buf1, buf2):
2    result = bytearray(len(buf1))
3    for i in range(len(buf1)):
4        result[i] = buf1[i] ^ buf2[i]
5    return bytes(result)

Slightly faster than the generator approach because bytearray avoids creating intermediate tuples, but still limited by Python's loop overhead.

Method 4: Using struct for Word-Aligned XOR

python

1import struct
2
3def xor_struct(buf1, buf2):
4    n = len(buf1)
5    # Process 8 bytes at a time
6    result = bytearray(n)
7    chunks = n // 8
8    remainder = n % 8
9
10    fmt = f'{chunks}Q'
11    ints1 = struct.unpack(fmt, buf1[:chunks * 8])
12    ints2 = struct.unpack(fmt, buf2[:chunks * 8])
13    xored = struct.pack(fmt, *(a ^ b for a, b in zip(ints1, ints2)))
14    result[:chunks * 8] = xored
15
16    # Handle remaining bytes
17    for i in range(chunks * 8, n):
18        result[i] = buf1[i] ^ buf2[i]
19
20    return bytes(result)

This processes 8 bytes per iteration by unpacking to 64-bit unsigned integers. Faster than byte-by-byte but slower than int.from_bytes or NumPy.

Method 5: ctypes / cffi for C-Level Speed

python

1import ctypes
2
3def xor_ctypes(buf1, buf2):
4    n = len(buf1)
5    result = bytearray(n)
6
7    # Use memmove and XOR via ctypes
8    src1 = (ctypes.c_uint8 * n).from_buffer_copy(buf1)
9    src2 = (ctypes.c_uint8 * n).from_buffer_copy(buf2)
10    dst = (ctypes.c_uint8 * n).from_buffer(result)
11
12    for i in range(n):
13        dst[i] = src1[i] ^ src2[i]
14
15    return bytes(result)

Still limited by Python loop overhead. For true C-level performance, write a C extension or use Cython:

cython

1# xor_fast.pyx
2def xor_cython(bytes buf1, bytes buf2):
3    cdef int n = len(buf1)
4    cdef bytearray result = bytearray(n)
5    cdef int i
6    for i in range(n):
7        result[i] = buf1[i] ^ buf2[i]
8    return bytes(result)

Benchmarks

python

1import timeit
2
3size = 100_000  # 100 KB buffers
4a = bytes(range(256)) * (size // 256)
5b = bytes(range(255, -1, -1)) * (size // 256)
6
7print("Naive:      ", timeit.timeit(lambda: xor_naive(a, b), number=100))
8print("int:        ", timeit.timeit(lambda: xor_int(a, b), number=100))
9print("numpy:      ", timeit.timeit(lambda: xor_numpy(a, b), number=100))
10print("struct:     ", timeit.timeit(lambda: xor_struct(a, b), number=100))

Typical results for 100 KB buffers (100 iterations):

Method	Time (s)	Relative Speed
Naive (generator)	3.2	1x
bytearray loop	2.5	1.3x
struct (8B chunks)	0.8	4x
int.from_bytes	0.05	64x
NumPy	0.02	160x

Practical Use Cases

python

1import numpy as np
2
3# XOR encryption (one-time pad)
4def xor_encrypt(plaintext, key):
5    p = np.frombuffer(plaintext, dtype=np.uint8)
6    k = np.frombuffer(key, dtype=np.uint8)
7    return (p ^ k).tobytes()
8
9message = b"Hello, World!"
10key = bytes([0x42] * len(message))
11encrypted = xor_encrypt(message, key)
12decrypted = xor_encrypt(encrypted, key)
13print(decrypted)  # b'Hello, World!'
14
15# Detecting differences between two binary files
16def diff_buffers(buf1, buf2):
17    arr = np.frombuffer(xor_numpy(buf1, buf2), dtype=np.uint8)
18    changed_positions = np.nonzero(arr)[0]
19    return changed_positions

Common Pitfalls

Different buffer lengths: XOR requires equal-length inputs. zip() silently truncates to the shorter buffer. Always check lengths first or pad the shorter buffer.
int.from_bytes with empty buffers: int.from_bytes(b'', 'big') returns 0, and (0).to_bytes(0, 'big') returns b''. This is correct but can be confusing.
NumPy overhead for small buffers: NumPy has fixed overhead for array creation. For buffers under 100 bytes, the naive approach or int.from_bytes may be faster.
Mutability: bytes is immutable. If you need to XOR in-place, use bytearray or NumPy arrays with np.bitwise_xor(arr1, arr2, out=arr1).
Memory usage: int.from_bytes creates a Python int that uses roughly the same memory as the buffer. NumPy arrays also copy by default. For multi-gigabyte buffers, use np.frombuffer (zero-copy view) and process in chunks.

Summary

Naive byte-by-byte XOR in Python is 100-1000x slower than optimized approaches
int.from_bytes + ^ + to_bytes is the fastest pure-Python method (no dependencies)
NumPy is the fastest overall — uses SIMD instructions for vectorized XOR
For small buffers (under 100 bytes), the overhead of NumPy or int conversion may exceed the loop savings
Use struct.unpack for moderate speedup without external dependencies
For production crypto or network code, use libraries like cryptography that implement XOR in C

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Basis Lab

No course covers this one yet

Basis Lab writes one for you from a sentence about what you want to be able to do, then teaches it and checks you understood. Your first course is free.

Build my course

Data Structures & Algorithms practice on Codemia

Step through 300+ coding interview problems with animated visualisers that show the data structure changing as the code runs. The Blind 75 are free.

Practice algorithms

Simple Python Challenge Fastest Bitwise XOR on Data Buffers

Data Structures & Algorithms practice on Codemia

Introduction

The Naive Approach (Slow)

Method 1: int.from_bytes XOR (Fast, No Dependencies)

Method 2: NumPy (Fastest for Large Buffers)

Method 3: bytearray with memoryview

Method 4: Using struct for Word-Aligned XOR

Method 5: ctypes / cffi for C-Level Speed

Benchmarks

Practical Use Cases

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

No course covers this one yet

Data Structures & Algorithms practice on Codemia