pandas resample documentation
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
pandas.resample is the time-series equivalent of groupby: it groups rows into time buckets and then lets you aggregate or fill them. It is most useful when your data has a datetime index or when you point resample at a datetime column with on=. Once that requirement is satisfied, you can downsample, upsample, and control how the time bins are labeled.
Start with a Datetime Index
The most common pattern is a Series or DataFrame indexed by timestamps.
"1H" is the resampling rule, meaning one-hour buckets. Because we call .sum(), all rows inside each hour are aggregated together.
Downsampling Means Combining Smaller Intervals
Downsampling reduces frequency, such as minute data into hourly data or daily data into monthly data. The aggregation step is required because each output bucket represents multiple input rows.
Common aggregations include:
- '
.sum()for totals.' - '
.mean()for averages.' - '
.max()and.min()for extremes.' - '
.agg()for multiple calculations at once.'
This is especially useful for monitoring, financial data, and sensor streams where raw timestamps are too fine-grained for the question you want to answer.
Upsampling Creates Empty Time Slots
Upsampling moves to a higher frequency, such as hourly data to 15-minute intervals. That creates new timestamps where no original data exists, so the immediate result contains missing values.
Once the empty rows exist, you can decide how to fill them:
Use forward fill only when carrying the last known value forward is logically correct. For state measurements such as temperature or account balance, that may be reasonable. For event counts or transactions, it is often wrong.
Use on= When the Datetime Is a Column
You do not have to move the datetime into the index. If the timestamps live in a column, pass that column name to on=.
This is convenient when you want to preserve an existing index or avoid reshaping the frame.
label and closed Control Bucket Semantics
Time bins have edges, so it matters which side is closed and which timestamp labels the result. For example, a one-hour bucket can be labeled at the start or the end of the interval.
These options matter most when you compare pandas results against database reports, dashboards, or business definitions that specify how interval boundaries should behave.
Resampling Is Only as Good as the Time Data
If the index is not actually datetime-like, resample will fail. If the timestamps have mixed time zones or unexpected gaps, the result may be technically correct but analytically misleading. Always validate the timestamp column before trusting the aggregate.
Common Pitfalls
- Calling
resampleon an index that is not datetime-like. - Forgetting that downsampling requires an aggregation step.
- Upsampling and then filling missing values with a method that changes the meaning of the data.
- Ignoring
labelandclosedwhen interval boundaries matter. - Assuming
resamplesorts and cleans bad timestamps for you.
Summary
- '
resamplegroups time-series data into new frequency buckets.' - Use a datetime index or pass a datetime column with
on=. - Downsampling combines multiple rows and requires aggregation.
- Upsampling creates empty time slots that you may need to fill explicitly.
- Pay attention to interval labeling and boundary rules when accuracy matters.
Related reading
- Pandas Setting no. of max rows
- Pandas sum DataFrame rows for given columns
- pandas three-way joining multiple dataframes on columns
- pandas unique values multiple columns
- pandas.factorize on an entire data frame
- pandas.parser.CParserError Error tokenizing data
- Parallel processes in distributed tensorflow
- Parsing one terabyte of text and efficiently counting the number of occurrences of each word
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.