pandas
data analysis
data manipulation
counting occurrences
Python

What is the most efficient way of counting occurrences in pandas?

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Introduction

In pandas, the most efficient way to count occurrences usually depends on what exactly you are counting. For a single Series, value_counts() is the standard fast path. For grouped tables, groupby().size() or crosstab() is often the better fit.

Count Values in One Column with value_counts

For a single column, value_counts() is usually the first choice because it is concise and optimized for frequency counting.

python
1import pandas as pd
2
3s = pd.Series(["a", "b", "a", "c", "b", "a"])
4counts = s.value_counts()
5print(counts)

This returns the frequency of each unique value, sorted by count descending by default.

For a DataFrame column, the pattern is the same:

python
counts = df["category"].value_counts()

That is typically the fastest and clearest answer when the task is simply, "How many times does each value occur in this column?"

Normalize or Keep Missing Values When Needed

value_counts() also supports useful options.

python
counts = df["category"].value_counts(dropna=False)
proportions = df["category"].value_counts(normalize=True)

These let you include missing values or compute proportions without writing extra aggregation code.

Count by Multiple Columns with groupby().size()

When the frequency depends on combinations of columns, use groupby().size().

python
1import pandas as pd
2
3df = pd.DataFrame({
4    "city": ["Toronto", "Toronto", "Montreal", "Toronto"],
5    "status": ["open", "closed", "open", "open"]
6})
7
8counts = df.groupby(["city", "status"]).size().reset_index(name="count")
9print(counts)

This is the right pattern when you want counts per key combination rather than per individual column value.

Use crosstab for Frequency Tables

If you want a matrix of counts between two categorical variables, pd.crosstab is often more readable than a grouped aggregation.

python
table = pd.crosstab(df["city"], df["status"])
print(table)

This is especially useful in exploratory data analysis and reporting.

Performance Guidance

In real workloads, efficiency depends less on tiny syntax differences and more on whether you are using vectorized pandas operations instead of Python loops. Avoid manually iterating over rows to count values.

For example, this is usually the wrong direction:

python
counts = {}
for value in df["category"]:
    counts[value] = counts.get(value, 0) + 1

It works, but pandas already provides faster and clearer columnar operations.

If the column is categorical and reused heavily, converting it to the category dtype can also reduce memory pressure and improve some group operations on repeated labels.

python
df["category"] = df["category"].astype("category")
counts = df["category"].value_counts()

That optimization is especially relevant when the column has many repeated labels and the dataset is large enough for dtype choice to matter.

Common Pitfalls

  • Writing Python loops over DataFrame rows is almost always slower and less idiomatic than value_counts() or groupby().size(). Use pandas-native operations first.
  • Using value_counts() when you really need counts of multiple-column combinations gives the wrong result shape. Switch to groupby().size() for compound keys.
  • Forgetting dropna=False can hide missing-value counts that are analytically important. Decide explicitly how nulls should be treated.
  • Sorting assumptions can create confusion because value_counts() sorts by frequency by default. Use .sort_index() if label order matters more than count order.
  • Treating all counting tasks as identical misses better tools such as crosstab for two-dimensional frequency tables. Pick the method that matches the output you need.

Summary

  • For one Series or one DataFrame column, value_counts() is usually the most efficient and direct option.
  • For counts across multiple grouping keys, use groupby().size().
  • For matrix-style frequency tables, use pd.crosstab().
  • Avoid row-by-row Python loops for counting in pandas.
  • Efficiency in pandas comes mostly from staying in vectorized, built-in operations.

Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

All Rights Reserved.