Convert Python dict into a dataframe
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Converting a Python dictionary into a pandas DataFrame is straightforward once you know what shape the dictionary represents. The main question is whether the keys represent columns, rows, or records, because pandas supports each of those patterns with slightly different constructors.
Dictionary of lists becomes columns
The most common case is a dictionary where each key is a column name and each value is a list of column values.
This is the simplest constructor path. pandas assumes each list is one column and aligns rows by position.
That also means the lists should be the same length. If they are not, construction will fail because pandas cannot form a rectangular table from columns of incompatible sizes.
List of dictionaries becomes rows
Sometimes the source data looks like records rather than columns.
Here each dictionary becomes one row. This is often the right format when data arrives from JSON APIs or row-oriented processing.
If some dictionaries are missing keys, pandas fills those cells with NaN. That behavior is useful, but it also means inconsistent input can silently introduce missing data if you are not watching for it.
Use from_dict when orientation matters
DataFrame.from_dict is useful when the dictionary should be interpreted differently, especially when keys represent rows instead of columns.
With orient="index", the outer keys become row labels. Without that argument, pandas would try to treat the outer keys as columns instead.
This is one of the most common points of confusion. The data is still a dictionary, but the intended table shape is different.
Choose the constructor by data shape
A practical rule is:
- dictionary of lists:
pd.DataFrame(data) - list of dictionaries:
pd.DataFrame(records) - nested dictionary with row keys:
pd.DataFrame.from_dict(data, orient="index")
Once you think in terms of table shape rather than only Python type, the constructor choice becomes much easier.
Set the index deliberately when needed
Sometimes the dictionary already contains a natural identifier such as a user id or date key. In that case, building the DataFrame is only the first step. You may also want to move one field into the index so later joins, lookups, or time-series operations behave the way you expect. That is another reason to inspect the resulting frame rather than assuming the default integer index is always the right shape.
Common Pitfalls
- Forgetting that a dictionary of lists requires all lists to have compatible lengths.
- Treating a list of dictionaries like a dictionary of columns and getting the wrong shape.
- Ignoring
orient="index"when the outer keys should become rows. - Being surprised by
NaNvalues when some records omit keys. - Converting data successfully but not checking whether the resulting columns and index actually match the intended layout.
Summary
- pandas can build a
DataFramefrom several dictionary-shaped inputs. - The right constructor depends on whether the data is column-oriented, row-oriented, or index-oriented.
- '
pd.DataFrame(...)handles common column and record cases well.' - '
from_dict(..., orient="index")is useful when dictionary keys should become rows.' - Think about the table shape first, then choose the constructor.
Related reading
- Convert row to column header for Pandas DataFrame
- Convert unix time to readable date in pandas dataframe
- Converting a Pandas GroupBy multiindex output from Series back to DataFrame
- Converting a tf.Tensor to numpy array in tf.data.Dataset.map graph mode in TF 2.0
- Convert python opencv mat image to tensorflow image data
- Convert Python program to C/C code?
- Converting from Pandas dataframe to TensorFlow tensor object
- converting list of header and row lists into pandas DataFrame
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.