python - how to append numpy array to a pandas dataframe
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Appending a NumPy array to a pandas DataFrame usually means converting the array into a DataFrame first and then combining the two objects. In modern pandas, the old DataFrame.append() method is no longer the recommended answer, so the normal tool is pd.concat().
Row Append Versus Column Append
Before writing code, decide what "append" means:
- add new rows to the bottom of the DataFrame
- add new columns to the right side of the DataFrame
The shape requirements are different for each case.
Appending Rows
If the NumPy array represents extra rows, it must have the same number of columns as the target DataFrame. Convert the array into a DataFrame with matching column names, then concatenate along axis 0.
Using ignore_index=True gives the combined DataFrame a clean sequential index.
Appending Columns
If the array contains new columns, the number of rows must match the existing DataFrame length. In that case, concatenate along axis 1.
Now the array becomes extra columns instead of extra rows.
Why append() Is Not the Modern Answer
Older code often uses:
That style is outdated. pd.concat() is the modern, explicit, and supported way to combine DataFrames in current pandas versions.
It also scales better when you combine many pieces at once.
Build the Intermediate DataFrame Carefully
The conversion step matters because a NumPy array does not carry pandas column labels. If you skip the column names, pandas may create default numeric labels such as 0, 1, and 2, which often do not match the target DataFrame.
A good pattern is:
- inspect the shape of the array
- decide whether it represents rows or columns
- build a DataFrame with explicit labels
- concatenate with the correct axis
That prevents most alignment bugs.
Dtype Considerations
A NumPy array has one dtype for the whole array, while a pandas DataFrame can have different dtypes per column. Mixed data often becomes dtype=object in NumPy.
For example:
That is fine, but you may want to cast numeric columns afterward:
Being explicit keeps downstream code predictable.
Performance Tip for Loops
If you are repeatedly appending arrays in a loop, do not concatenate on every iteration. That causes repeated DataFrame allocations.
A better approach is:
- collect temporary DataFrames in a list
- call
pd.concat()once at the end
This pattern is much more efficient for larger workloads.
Common Pitfalls
The biggest mistake is using the removed DataFrame.append() pattern in modern pandas code. Prefer pd.concat().
Another mistake is mismatching shapes. Row appends require matching column counts, while column appends require matching row counts.
People also forget to assign column names when converting the NumPy array to a DataFrame, which leads to surprising alignment results.
Finally, mixed-type arrays often become object dtype. If a column should be numeric, cast it explicitly after concatenation.
Summary
- Convert the NumPy array into a DataFrame before combining it with pandas data.
- Use
pd.concat()instead of the oldappend()method. - Concatenate with
axis=0for rows andaxis=1for columns. - Match column labels and shapes carefully.
- If you combine many pieces, collect them first and concatenate once for better performance.
Related reading
- Python - machine learning
- Python Convert timedelta to int in a dataframe
- python dataframe pandas drop column using int
- Python float argument must be a string or a number, not ''pandas._libs.interval.Interval''
- Python - How to intuit word from abbreviated text using NLP?
- Python - Is a dictionary slow to find frequency of each character?
- Python Graph Library
- Python Implementation of OPTICS Clustering Algorithm
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.