Introduction

Fitting models on large tabular data with pandas and scikit-learn often fails because everything is loaded into memory at once. The fix is to stream data in chunks, reduce memory footprint early, and use estimators that support incremental updates. With this pattern, you can train useful models on datasets much larger than system memory.

Prepare Data in Chunks

Pandas can read large CSV files in chunks. You can downcast numeric columns and convert repeated strings to categorical values to reduce memory pressure before model training starts.

1import pandas as pd
2
3csv_path = "train.csv"
4chunk_iter = pd.read_csv(csv_path, chunksize=50_000)
5
6for i, chunk in enumerate(chunk_iter, start=1):
7    for col in chunk.select_dtypes(include=["int64"]).columns:
8        chunk[col] = pd.to_numeric(chunk[col], downcast="integer")
9    for col in chunk.select_dtypes(include=["float64"]).columns:
10        chunk[col] = pd.to_numeric(chunk[col], downcast="float")
11
12    mem_mb = chunk.memory_usage(deep=True).sum() / (1024 * 1024)
13    print(f"chunk {i} memory: {mem_mb:.2f} MB")

Chunked loading prevents immediate out of memory failures and gives you a place to apply light cleaning.

Train Incrementally with partial_fit

Many scikit-learn estimators support incremental learning through partial_fit. This is ideal for chunk workflows.

1import numpy as np
2import pandas as pd
3from sklearn.linear_model import SGDClassifier
4from sklearn.preprocessing import StandardScaler
5
6TARGET = "label"
7FEATURES = ["x1", "x2", "x3", "x4"]
8CLASSES = np.array([0, 1])
9
10scaler = StandardScaler(with_mean=False)
11clf = SGDClassifier(loss="log_loss", random_state=42)
12
13chunk_iter = pd.read_csv("train.csv", usecols=FEATURES + [TARGET], chunksize=20_000)
14
15for idx, chunk in enumerate(chunk_iter):
16    X = chunk[FEATURES].to_numpy()
17    y = chunk[TARGET].to_numpy()
18
19    scaler.partial_fit(X)
20    X_scaled = scaler.transform(X)
21
22    if idx == 0:
23        clf.partial_fit(X_scaled, y, classes=CLASSES)
24    else:
25        clf.partial_fit(X_scaled, y)
26
27print("incremental training completed")

This code is runnable and scales well for many row counts as long as feature width is manageable.

Evaluate and Predict in Streaming Mode

Evaluation can also be chunked so memory usage remains stable. Avoid concatenating all chunks unless you truly need a full in-memory frame.

1import pandas as pd
2from sklearn.metrics import accuracy_score
3
4scores = []
5for chunk in pd.read_csv("valid.csv", usecols=FEATURES + [TARGET], chunksize=20_000):
6    X = chunk[FEATURES].to_numpy()
7    y = chunk[TARGET].to_numpy()
8    X_scaled = scaler.transform(X)
9    preds = clf.predict(X_scaled)
10    scores.append(accuracy_score(y, preds))
11
12print(f"mean chunk accuracy: {sum(scores)/len(scores):.4f}")

For deployment, persist both transformer and model. If the scaling state and model weights drift apart, predictions will degrade quickly.

Practical Performance Tips

Use parquet when possible, since CSV parsing is expensive. If you must use CSV, specify explicit dtypes in read_csv. Remove unused columns early, and avoid object dtypes for numeric-like values. If training still saturates memory, reduce batch size and monitor process RSS over time.

You can also combine feature hashing with linear models for high-cardinality text or IDs. This avoids huge one-hot matrices and keeps updates fast.

Monitoring Resource Usage During Training

Large-data training should be observable. Track chunk processing time, rows per second, and peak memory per stage so regressions are easy to spot after code changes. Emit metrics after each chunk and include model checkpoint timing in logs. If throughput suddenly drops, investigate I O bottlenecks, compression overhead, or datatype conversion hotspots before changing estimator settings. This performance discipline usually delivers larger gains than random hyperparameter tuning.

Common Pitfalls

• Loading the full dataset into one DataFrame before dropping unused columns.
• Fitting scalers on the full file while the model trains chunk by chunk.
• Forgetting to pass classes on the first partial_fit call for classifiers.
• Mixing different preprocessing logic between train and validation streams.
• Measuring only training throughput while ignoring prediction latency and artifact size.

Summary

• Use chunked pandas reads to control memory usage.
• Downcast dtypes and prune columns early in the pipeline.
• Prefer incremental estimators with partial_fit for large datasets.
• Stream evaluation and prediction to keep resource use predictable.
• Persist preprocessing state together with the model for stable inference.