Applying function with multiple arguments to create a new pandas column
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
When working with Pandas DataFrames, you often need to compute a new column based on values from multiple existing columns. While simple arithmetic can be done directly with column operations, more complex logic requires passing multiple arguments into a function. Pandas provides several approaches for this, each with different trade-offs in readability, flexibility, and performance. This article walks through four techniques so you can pick the right one for your situation.
Using apply with a Lambda Function
The most straightforward approach is calling apply() on a single column and passing additional arguments through a lambda. This works well when your function takes one "main" column value plus some extra parameters.
Output:
The axis=1 parameter tells Pandas to pass each row (not each column) to the lambda. Inside the lambda, row is a Series with the column names as its index, so you access values with row['column_name'].
You can also use the args parameter to pass extra fixed values:
Here, apply calls compute_discounted(price_value, 0.15) for each element in the price column. The args tuple supplies the second argument.
Using apply with axis=1 and a Named Function
For more complex row-wise logic, define a standalone function that receives the entire row. This is cleaner than cramming everything into a lambda.
Output:
This pattern is ideal when the logic involves conditionals, lookups, or anything that does not reduce to a simple arithmetic expression. The function signature takes a single row parameter, and you access columns by name inside the function body.
Using np.vectorize
When apply feels slow on large DataFrames, np.vectorize offers a middle ground. It wraps a Python function so you can call it with entire columns at once, even though it still loops internally.
The key advantage is readability: the call site looks like a vectorized operation even though the underlying execution is element-wise. Performance is modestly better than apply with axis=1 because it avoids constructing a Series for each row, but it is not as fast as true NumPy vectorization.
Use np.vectorize when your function is simple, takes multiple column inputs, and you want clean syntax without rewriting the logic as a pure NumPy expression.
Using df.assign with Vectorized Operations
The fastest approach is to skip apply entirely and use native Pandas or NumPy vectorized operations. The assign method lets you create new columns in a chainable, expressive way.
Output:
This is the recommended approach whenever the computation can be expressed as arithmetic on columns. It runs at C speed through NumPy's underlying implementation and can be orders of magnitude faster than apply on DataFrames with hundreds of thousands of rows.
You can also use np.where for conditional logic:
Common Pitfalls
- Forgetting axis=1 when using apply row-wise. The default
axis=0applies the function column-wise, which produces confusing errors or wrong results when you intended row-wise operation. - Using apply for simple arithmetic that could be vectorized. Writing
df.apply(lambda row: row['a'] + row['b'], axis=1)is roughly 100 times slower thandf['a'] + df['b']on large DataFrames. Always prefer vectorized operations when possible. - Passing mutable default arguments to the function. If your function has a default argument like
def f(row, cache={}), the cache persists across calls and can cause subtle bugs. - Confusing args and kwargs in apply. The
argsparameter expects a tuple for positional arguments. A common mistake is writingargs=(0.15)instead ofargs=(0.15,), which passes a float rather than a single-element tuple. - Assuming np.vectorize gives true vectorized performance. Despite the name,
np.vectorizeis a convenience wrapper, not a performance tool. For genuine speedups, rewrite the logic using NumPy array operations or use libraries like Numba.
Summary
- Use
df.apply(func, axis=1)with a lambda or named function when you need row-wise access to multiple columns and the logic is too complex for vectorization. - Use the
argsparameter onSeries.apply()to pass additional fixed arguments to your function. - Use
np.vectorizefor cleaner syntax when applying a simple multi-argument function across columns, but do not expect major speed gains. - Use
df.assignwith vectorized column operations for the best performance on large DataFrames. - Always benchmark with
%timeitbefore committing to an approach, especially when working with DataFrames that have more than 10,000 rows.
Related reading
- Applying PCA to one sample
- Appropriate Deep Learning Structure for multi-class classification
- Are there any examples of anomaly detection algorithms implemented with TensorFlow?
- .arff files with scikit-learn?
- Approve a CSR in Kuberentes Using the Python client
- APScheduler. What is the best practices to shutdown schedulers with any non memory storage?
- ARIMA Forecast Cannot cast ufunc subtract output from dtype'float64' to dtype'int64' with casting rule 'same_kind
- Assign a name to a tensor?
System design interview problems on this topic
- Design a Web Crawler: solution and walkthrough
- Design a Task Scheduler: solution and walkthrough
- Design a URL Shortener (TinyURL): solution and walkthrough
- Design an API Rate Limiter: solution and walkthrough
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseNo course covers this one yet
Basis Lab writes one for you from a sentence about what you want to be able to do, then teaches it and checks you understood. Your first course is free.
Build my courseML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.