Practice Now
Polars
DataFrames
Scikit-learn
Machine Learning
Python

How to use polars dataframes with scikit-learn?

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Machine LearningPython

Introduction

Scikit-learn expects NumPy arrays or pandas DataFrames as input. Polars DataFrames are not directly compatible, but converting between them is straightforward. You can call .to_numpy() on a Polars DataFrame to get a NumPy array, .to_pandas() to get a pandas DataFrame, or (since scikit-learn 1.4+) use Polars directly with the set_output API. This article covers each approach with practical machine learning examples.

Setup

bash
pip install polars scikit-learn
python
1import polars as pl
2import numpy as np
3from sklearn.model_selection import train_test_split
4from sklearn.ensemble import RandomForestClassifier
5from sklearn.metrics import accuracy_score

Method 1: Convert to NumPy (.to_numpy())

The most common and reliable approach:

python
1# Create a Polars DataFrame
2df = pl.DataFrame({
3    "feature_1": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],
4    "feature_2": [5.0, 4.0, 3.0, 2.0, 1.0, 6.0, 7.0, 8.0],
5    "label": [0, 0, 0, 0, 1, 1, 1, 1],
6})
7
8# Extract features and labels as NumPy arrays
9X = df.select(["feature_1", "feature_2"]).to_numpy()
10y = df["label"].to_numpy()
11
12# Standard scikit-learn workflow
13X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)
14model = RandomForestClassifier(n_estimators=100)
15model.fit(X_train, y_train)
16predictions = model.predict(X_test)
17print(f"Accuracy: {accuracy_score(y_test, predictions)}")

.to_numpy() creates a contiguous NumPy array, which is what scikit-learn uses internally. This is zero-overhead for numeric data with no null values.

Method 2: Convert to Pandas (.to_pandas())

Useful when scikit-learn utilities expect DataFrame features (column names, dtypes):

python
1# Convert to pandas DataFrame
2pdf = df.to_pandas()
3
4X = pdf[["feature_1", "feature_2"]]
5y = pdf["label"]
6
7model = RandomForestClassifier()
8model.fit(X, y)
9
10# Feature importance with column names
11for name, importance in zip(X.columns, model.feature_importances_):
12    print(f"{name}: {importance:.3f}")

The conversion has overhead — Polars creates a new pandas DataFrame in memory. For large datasets, prefer .to_numpy().

Method 3: set_output API (scikit-learn 1.4+)

Scikit-learn 1.4+ supports Polars DataFrames directly through the set_output API:

python
1from sklearn.preprocessing import StandardScaler
2from sklearn.pipeline import Pipeline
3
4# Configure scikit-learn to output Polars DataFrames
5import sklearn
6sklearn.set_config(transform_output="polars")
7
8# Now transformers return Polars DataFrames
9scaler = StandardScaler()
10X_pl = df.select(["feature_1", "feature_2"])
11X_scaled = scaler.fit_transform(X_pl)
12print(type(X_scaled))  # <class 'polars.dataframe.frame.DataFrame'>
13
14# Or per-transformer
15scaler = StandardScaler().set_output(transform="polars")

This avoids manual conversion in preprocessing pipelines. Note that .fit() and .predict() still work with Polars since scikit-learn calls NumPy internally.

Full Pipeline Example

python
1import polars as pl
2from sklearn.pipeline import Pipeline
3from sklearn.preprocessing import StandardScaler, OneHotEncoder
4from sklearn.compose import ColumnTransformer
5from sklearn.linear_model import LogisticRegression
6from sklearn.model_selection import cross_val_score
7
8# Polars DataFrame with mixed types
9df = pl.DataFrame({
10    "age": [25, 30, 35, 40, 45, 50, 55, 60],
11    "income": [30000, 50000, 70000, 90000, 40000, 60000, 80000, 100000],
12    "city": ["NY", "LA", "NY", "SF", "LA", "SF", "NY", "LA"],
13    "purchased": [0, 1, 1, 1, 0, 1, 1, 1],
14})
15
16# Separate features and target
17feature_cols = ["age", "income", "city"]
18X = df.select(feature_cols).to_pandas()  # ColumnTransformer needs pandas
19y = df["purchased"].to_numpy()
20
21# Preprocessing pipeline
22preprocessor = ColumnTransformer([
23    ("num", StandardScaler(), ["age", "income"]),
24    ("cat", OneHotEncoder(drop="first"), ["city"]),
25])
26
27pipeline = Pipeline([
28    ("preprocess", preprocessor),
29    ("classifier", LogisticRegression()),
30])
31
32# Cross-validation
33scores = cross_val_score(pipeline, X, y, cv=3)
34print(f"CV Accuracy: {scores.mean():.3f}")

Handling Null Values

Polars uses null for missing values. NumPy converts these to NaN, which scikit-learn estimators do not accept by default:

python
1df = pl.DataFrame({
2    "feature": [1.0, None, 3.0, None, 5.0],
3    "label": [0, 0, 1, 1, 1],
4})
5
6# .to_numpy() converts None to NaN
7X = df.select("feature").to_numpy()
8# array([[ 1.], [nan], [ 3.], [nan], [ 5.]])
9
10# Fix: impute missing values
11from sklearn.impute import SimpleImputer
12
13imputer = SimpleImputer(strategy="mean")
14X_imputed = imputer.fit_transform(X)
15
16# Or drop nulls in Polars first
17df_clean = df.drop_nulls()
18X = df_clean.select("feature").to_numpy()

Performance: Polars vs Pandas for ML Prep

python
1import polars as pl
2import pandas as pd
3import time
4
5# Generate large dataset
6n = 1_000_000
7df_polars = pl.DataFrame({
8    f"f{i}": np.random.randn(n) for i in range(50)
9})
10
11# Polars to NumPy — fast (near zero-copy for contiguous data)
12start = time.time()
13X = df_polars.to_numpy()
14print(f"Polars to NumPy: {time.time() - start:.3f}s")
15
16# Polars to Pandas — slower (copies data)
17start = time.time()
18pdf = df_polars.to_pandas()
19print(f"Polars to Pandas: {time.time() - start:.3f}s")

For numeric-only DataFrames, .to_numpy() is significantly faster than .to_pandas() because it avoids constructing a pandas DataFrame.

Common Pitfalls

  • Passing Polars DataFrames directly to .fit(): Most scikit-learn estimators do not accept Polars DataFrames before version 1.4. You get TypeError: no valid checking.... Convert with .to_numpy() or .to_pandas() first.
  • Null values becoming NaN: Polars null becomes NumPy NaN, which causes ValueError in most scikit-learn estimators. Impute or drop nulls before conversion.
  • Losing column names with .to_numpy(): NumPy arrays have no column names. If you need feature names (for ColumnTransformer or feature importance), use .to_pandas() or pass column names manually.
  • Categorical columns in .to_numpy(): String and categorical columns cannot be converted to a numeric NumPy array. Encode them first with Polars (df.with_columns(pl.col("city").cast(pl.Categorical))) or use OneHotEncoder in a pipeline.
  • Memory duplication: Both .to_numpy() and .to_pandas() copy data. For very large datasets, this doubles memory usage. Consider processing in chunks or using Polars' lazy API to reduce the DataFrame before conversion.

Summary

  • Use .to_numpy() for the fastest conversion when column names are not needed
  • Use .to_pandas() when scikit-learn utilities need column names (ColumnTransformer, feature importance)
  • Use set_output(transform="polars") (scikit-learn 1.4+) to keep Polars DataFrames through pipelines
  • Handle null values before conversion — impute with SimpleImputer or drop with .drop_nulls()
  • Encode categorical columns before calling .to_numpy() — NumPy arrays are numeric only
  • For large datasets, .to_numpy() is faster and uses less memory than .to_pandas()
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.