How can I pivot a dataframe?
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Pivoting a pandas DataFrame means reshaping long-form rows into a wider table where one column becomes the new columns axis. The right tool is usually pivot when every index and column combination is unique, or pivot_table when duplicates exist and you need aggregation.
Use pivot For Clean One-To-One Reshaping
Suppose you start with long data:
You can pivot it like this:
Result:
- each unique
datebecomes a row - each
metricbecomes a column - '
valuefills the cells'
This is the cleanest API when there is exactly one value for each row-column combination.
Use pivot_table When Duplicates Exist
If the same index and column pair appears more than once, pivot raises an error because it does not know which value to keep.
That is when pivot_table is the right tool:
Now duplicates are aggregated instead of causing failure.
Common aggregation functions include:
- '
"mean"' - '
"sum"' - '
"count"' - '
max' - '
min'
That makes pivot_table the more flexible tool for real-world messy data.
Reset The Index If You Want A Flat DataFrame
After pivoting, the former index often becomes the actual DataFrame index. If you want it back as a regular column, call reset_index.
This is useful before exporting, merging, or serializing the result.
Multiple Value Columns And MultiIndex Output
Pandas can also pivot multiple value columns, which often creates a MultiIndex on the columns.
The output is correct, but the column structure becomes more complex. If needed, you can flatten it afterward.
Know The Reverse Operation
Pivoting is often only half the story. To move wide data back into long form, use melt.
Understanding both pivot and melt makes reshaping much easier because you can move between long and wide forms deliberately.
A Good Rule Of Thumb
Use:
- '
pivotwhen the data is already unique per index-column pair' - '
pivot_tablewhen duplicates exist or aggregation is needed' - '
meltwhen you need to go back to long form'
That simple rule covers most pandas reshaping tasks.
Common Pitfalls
The biggest mistake is using pivot on data that contains duplicate combinations. Pandas will reject that because the result would be ambiguous.
Another mistake is forgetting that the chosen index becomes the DataFrame index after the pivot. If later code expects a normal column, use reset_index.
People also get confused by MultiIndex columns after pivoting multiple value fields. The data is valid, but you may need extra cleanup if you want simple flat column names.
Finally, do not guess which reshaping function to use. Decide first whether you are doing pure rearrangement or rearrangement plus aggregation.
Summary
- '
pivotreshapes long data into wide form when each index-column pair is unique.' - '
pivot_tablehandles duplicates by aggregating values.' - '
reset_indexflattens the result when you want former index values back as columns.' - Pivoting multiple value columns can create MultiIndex columns.
- '
meltis the reverse-style operation when you need to return to long form.'
Related reading
- How can I print all eli5.explain_weights results without ellipsis?
- How can i programmatically generate descriptors for an arbitrary data set?
- How can I remove duplicate rows?
- How can I remove or omit data using map method for tf.data.Dataset objects?
- How can I print bold text in Python?
- How can I print multiple things fixed text and/or variable values on the same line, all at once?
- How can I remove the top and right axis?
- How can I return pivot table output in MySQL?
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.