Optimizing several million char to string conversions
Data Structures & Algorithms practice on Codemia
Step through 300 algorithm problems with animated visualisers that show the data structure changing as the code runs.
Introduction
Converting millions of char* C-strings to std::string in C++ is expensive because each conversion allocates heap memory, copies characters, and may trigger the allocator frequently. The main optimizations are: pre-allocating with reserve(), using std::string_view to avoid copies entirely, employing a memory pool or arena allocator, enabling Small String Optimization (SSO), and batching conversions to reduce allocator pressure. For read-only access, std::string_view eliminates the conversion cost completely.
Baseline: Naive Conversion
Each std::string construction from char* calls strlen() to find the length, allocates heap memory (if above SSO threshold), and copies the characters.
Optimization 1: Reserve Vector Capacity
reserve() pre-allocates the vector's internal buffer. emplace_back constructs the string directly in the vector, avoiding a temporary std::string and move.
Optimization 2: Use string_view (Zero-Copy)
std::string_view stores a pointer and length without owning the data. It is the fastest option when you only need read access and the original char* data outlives the string_view.
Optimization 3: Provide Length to Avoid strlen
If you already know the string lengths (e.g., from a database driver or parser), pass them to the std::string constructor to avoid the O(n) strlen() call per string.
Optimization 4: Arena Allocator
std::pmr::monotonic_buffer_resource allocates from a contiguous block without per-string malloc/free overhead. Deallocation is instant — the entire arena is freed at once.
Optimization 5: Leverage Small String Optimization (SSO)
For strings below the SSO threshold, std::string stores characters in the object itself. If most of your data consists of short strings (names, IDs, codes), SSO eliminates heap allocation automatically.
Optimization 6: Parallel Conversion
C++17 parallel algorithms distribute the conversion across multiple threads. Pre-size the output vector to avoid race conditions on push_back.
Benchmarking
Common Pitfalls
- Using
std::string_viewwhen the sourcechar*is freed:string_viewdoes not own the data. If the original buffer is freed or overwritten, thestring_viewbecomes a dangling reference. Only use it when the source data outlives all views. - Not reserving vector capacity: Without
reserve(), the vector doubles its allocation multiple times as it grows, copying all existing strings each time. For 10 million strings, this causes ~23 reallocations and copies. - Constructing
std::stringfromchar*without known length: The default constructor callsstrlen()which scans the entire string. If you already know the length from parsing or a database API, pass it explicitly:std::string(ptr, len). - Allocating each string individually in a hot loop: Millions of small
malloc/newcalls fragment the heap and stress the allocator. Use an arena allocator (pmr::monotonic_buffer_resource) to allocate from a single contiguous block. - Ignoring SSO for short strings: If your strings are under ~15 characters, they are already stored inline without heap allocation. Optimizing further (arena, pool) adds complexity without meaningful speedup for short string workloads.
Summary
- Use
std::string_viewfor zero-copy read-only access when the source data remains valid - Pre-allocate with
reserve()and useemplace_backto avoid vector reallocations and temporaries - Pass known string lengths to the
std::string(ptr, len)constructor to skipstrlen() - Use
std::pmr::monotonic_buffer_resourcefor arena allocation when converting millions of strings - Strings under ~15 characters benefit from SSO — no heap allocation needed
- Use C++17 parallel algorithms (
std::execution::par) for multi-threaded conversion on large datasets
Related reading
- Optimizing shuffle buffer size in tensorflow dataset api
- Optimizing subgraph of large graph - slower than optimizing subgraph by itself
- Optimizing this C algorithm
- Optimizing very often used anagram function
- Order Statistic Tree in C
- Pass multiple arguments into stdthread
- Optimum path in a graph to maximize a value
- Ordered starting and waiting for containers

DSA Fundamentals
Master algorithmic patterns and data structures through hands-on LeetCode-style problems - from arrays and hashing to dynamic programming and advanced graphs.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
Data Structures & Algorithms practice on Codemia
Step through 300 algorithm problems with animated visualisers that show the data structure changing as the code runs.