Introduction

Reading a large CSV with pandas is mostly a memory-management problem, not a syntax problem. The practical solution is usually to avoid loading the whole file at once: read in chunks, keep only required columns, and push aggregation into the streaming loop instead of building one huge DataFrame first.

Start By Reducing What You Read

The easiest win is to read fewer columns and declare types where possible:

1import pandas as pd
2
3df = pd.read_csv(
4    "transactions.csv",
5    usecols=["user_id", "country", "amount"],
6    dtype={
7        "user_id": "int64",
8        "country": "string",
9        "amount": "float64",
10    },
11)
12
13print(df.head())

If the file still fits in memory after narrowing the schema, this may be enough.

The point is that large-file performance is often improved more by reading less data than by tweaking parser internals.

Use chunksize For Streaming Reads

When the file is too large to fit comfortably in memory, use chunked iteration:

1import pandas as pd
2
3total = 0.0
4
5for chunk in pd.read_csv(
6    "transactions.csv",
7    usecols=["amount"],
8    chunksize=100_000,
9):
10    total += chunk["amount"].fillna(0).sum()
11
12print(total)

This keeps only one chunk in memory at a time.

For many workloads, chunking is the main pandas pattern you need to know.

Aggregate While Reading

A common anti-pattern is loading the whole file and only then grouping or summing it. If your goal is a summary, aggregate during the read:

1import pandas as pd
2from collections import defaultdict
3
4country_sum = defaultdict(float)
5
6for chunk in pd.read_csv(
7    "transactions.csv",
8    usecols=["country", "amount"],
9    chunksize=100_000,
10):
11    grouped = chunk.groupby("country", dropna=False)["amount"].sum()
12    for country, value in grouped.items():
13        country_sum[str(country)] += float(value)
14
15print(country_sum)

That avoids building a huge intermediate DataFrame only to reduce it immediately afterward.

Inspect A Small Sample First

Before running the full job, inspect a sample:

1import pandas as pd
2
3sample = pd.read_csv("transactions.csv", nrows=1000)
4print(sample.head())
5print(sample.dtypes)

This catches delimiter, quoting, encoding, and schema surprises early. For large files, failing after two minutes of parsing is much worse than failing after reading the first thousand rows.

Be Deliberate About Bad Rows

Large CSV exports are often messy. Decide whether to fail fast or skip malformed lines:

1import pandas as pd
2
3for chunk in pd.read_csv(
4    "transactions.csv",
5    chunksize=50_000,
6    on_bad_lines="skip",
7):
8    print(len(chunk))

Skipping bad lines may be acceptable for exploratory work, but for production pipelines you often want to log or isolate those rows rather than silently discarding them.

Know When Pandas Is Not The Best Tool

Pandas handles many large-file workflows well, especially with chunking. But if the work is heavily SQL-like, larger than single-machine comfort, or needs more parallelism, tools such as DuckDB, Polars, or Spark may be a better fit.

That does not mean pandas failed. It just means the workload outgrew the "one process, one file reader, one chunk loop" model.

It is also worth measuring with a realistic sample. Sometimes a file that looks intimidating still fits comfortably once unused columns are removed and dtypes are fixed, which lets you keep a simpler full-DataFrame approach.

Common Pitfalls

One common mistake is reading the full file with default settings and only then asking why memory exploded.

Another issue is letting pandas infer every dtype on a huge file even though the schema is already known.

A third problem is loading columns that are never used in the downstream calculation.

Finally, people often treat chunking as only a fallback for huge files, when it is often the cleanest design for any streaming summary task.

Summary

• Read fewer columns with usecols before trying lower-level optimizations.
• Use chunksize when the full CSV should not live in memory at once.
• Aggregate while streaming instead of after a full load when possible.
• Sample the file first to discover schema and formatting problems early.
• If the workload outgrows pandas, choose a tool designed for larger-scale tabular processing.