Dropping infinite values from dataframes in pandas?
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Infinite values (np.inf and -np.inf) in pandas DataFrames arise from operations like division by zero, logarithm of zero, or overflow in floating-point arithmetic. These values break statistical computations (mean(), std()), machine learning model training, and plotting. The standard approach is df.replace([np.inf, -np.inf], np.nan).dropna() — replace infinities with NaN, then use pandas' built-in NaN handling. Alternatively, use pd.options.mode.use_inf_as_na = True to treat infinities as NaN globally.
Identifying Infinite Values
np.isinf() detects both positive and negative infinity. Use np.isposinf() and np.isneginf() to check for each separately.
Method 1: Replace and Drop
This is the most common approach. replace() converts infinities to NaN, and dropna() removes rows containing NaN.
Method 2: Replace with a Specific Value
Method 3: Global Setting
Setting use_inf_as_na = True makes all pandas NaN-handling functions (like dropna(), fillna(), isna()) automatically treat infinities as missing values.
Method 4: Boolean Masking
Method 5: clip() to Cap Extreme Values
clip() replaces values outside the specified range, effectively capping infinities to a large but finite number.
Preventing Infinite Values
Common Pitfalls
isna()does not detect infinity by default:pd.isna(np.inf)returnsFalse. Infinities are not NaN. You must usenp.isinf()or setpd.options.mode.use_inf_as_na = Trueto catch them.dropna()alone does not remove infinities:df.dropna()only drops NaN values. You needdf.replace([np.inf, -np.inf], np.nan).dropna()to also remove infinite values.- Silent propagation in calculations:
np.mean([1, 2, np.inf])returnsinf, not an error. Infinite values silently corrupt aggregate statistics without warning. - Mixed types hide infinities: Columns with mixed types (object dtype) are not checked by
np.isinf(). Ensure numeric columns are typed correctly withdf.astype(float)before checking. use_inf_as_nais deprecated in recent pandas: In pandas 2.1+,use_inf_as_nais deprecated. Use explicitreplace([np.inf, -np.inf], np.nan)instead for future compatibility.
Summary
- Use
df.replace([np.inf, -np.inf], np.nan).dropna()to remove rows with infinities - Use
np.isinf(df)to detect infinite values (notpd.isna()) - Replace infinities with specific values using
replace()orclip() pd.options.mode.use_inf_as_na = Truetreats infinities as NaN globally (deprecated in pandas 2.1+)- Prevent infinities with safe division (
np.where(b != 0, a/b, fill)) and clamped log (np.log(x.clip(lower=epsilon))) - Always check for infinities as part of data validation before analysis or model training
Related reading
- Dummy variables when not all categories are present
- Dump a NumPy array into a csv file
- Dump a NumPy array into a csv file
- Duplicating training examples to handle class imbalance in a pandas data frame
- Duplicate log output when using Python logging module
- Dynamic instantiation from string name of a class in dynamically imported module?
- DynamicFrame vs DataFrame
- Effective queries in machine learning
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.