Return multiple columns from pandas apply
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Returning multiple columns from DataFrame.apply() is a common pattern when one row contains enough information to derive several output fields. The key is to return a shape pandas can expand reliably, usually a named Series or a tuple that is explicitly expanded into separate columns.
Return a Named Series
The clearest pattern is to return a Series with named entries:
This is explicit, readable, and self-documenting.
Tuple Expansion for Simpler Cases
If the logic is simple and you already know the target column order, a tuple can be concise:
This works well, but it is less self-documenting than returning a named Series.
Do Not Forget axis=1
This is one of the easiest mistakes:
axis=1 means apply the function row by row. Without it, pandas applies by column instead, and your function may receive an entirely different shape than expected.
Validate Output Shape When the Function Evolves
If the transformation function changes over time, it helps to validate the returned structure:
This catches accidental schema drift early instead of silently corrupting downstream logic.
Prefer Vectorized Operations When Possible
apply() is flexible, but it is not always the fastest choice. If the logic is simple arithmetic across columns, vectorized expressions are usually better:
Use row-wise apply() when:
- the logic is genuinely row-dependent
- the transformation is awkward to express vectorially
- clarity matters more than raw speed for the specific workload
Assigning the Result Cleanly
If you know the output columns ahead of time, assign them explicitly:
This is often cleaner than creating intermediate unnamed structures and trying to merge them later.
Keep Transformation Functions Small
When the row function gets too large, it becomes harder to reason about both the logic and the return shape. Small transformation functions with clear output fields are easier to test and safer to reuse.
If the logic becomes large enough to feel like a mini-parser, that is usually a signal to move it out into a more explicit transformation layer rather than hiding everything inside one apply() call.
Common Pitfalls
- Forgetting
axis=1and accidentally applying by column. - Returning plain lists with inconsistent lengths.
- Using row-wise
apply()for logic that would be cleaner and faster as vectorized expressions. - Assigning expanded results without stable column names.
- Letting the output shape drift over time without validation.
Summary
- Return a named
Serieswhen you want the clearest multi-column result. - Use tuple expansion for concise simple cases.
- Always remember
axis=1for row-wise logic. - Prefer vectorized expressions when the calculation is straightforward.
- Keep the output shape explicit so downstream code stays predictable.
Related reading
- Reverse colormap in matplotlib
- Right function for normalizing input of sklearn SVM
- Right order of doing feature selection, PCA and normalization?
- Right way to reverse a pandas DataFrame?
- Returning mutiple values in the input function for tf.py_func
- Returning the product of a list
- ROC curve for binary classification in python
- roc curve with sklearn python
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseNo course covers this one yet
Basis Lab writes one for you from a sentence about what you want to be able to do, then teaches it and checks you understood. Your first course is free.
Build my courseML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.