Introduction

Pandas groupby splits a DataFrame into groups based on one or more columns, then applies an aggregation function to each group. The default aggregations — sum, mean, count — collapse each group into a single scalar. But sometimes you need to collect all values in a group into a list rather than reducing them. This is useful for building lookup tables, generating comma-separated summaries, or feeding grouped data into machine learning pipelines.

Basic: Collect Column Values into Lists

Use .apply(list) or .agg(list) on a grouped column:

1import pandas as pd
2
3df = pd.DataFrame({
4    'category': ['A', 'A', 'B', 'B', 'C'],
5    'value': [1, 2, 3, 4, 5]
6})
7
8# Group by 'category', collect 'value' into lists
9result = df.groupby('category')['value'].apply(list).reset_index()
10print(result)
11#   category  value
12# 0        A  [1, 2]
13# 1        B  [3, 4]
14# 2        C    [5]
15
16# Equivalent using .agg(list)
17result2 = df.groupby('category')['value'].agg(list).reset_index()

Both .apply(list) and .agg(list) produce the same result. .agg(list) is slightly faster because it avoids the overhead of apply.

Multiple Columns into Lists

To collect multiple columns into lists simultaneously, use .agg(list) on the entire grouped DataFrame:

1df = pd.DataFrame({
2    'category': ['A', 'A', 'B', 'B', 'C'],
3    'value': [1, 2, 3, 4, 5],
4    'name': ['x', 'y', 'z', 'w', 'v']
5})
6
7result = df.groupby('category').agg(list).reset_index()
8print(result)
9#   category  value    name
10# 0        A  [1, 2]  [x, y]
11# 1        B  [3, 4]  [z, w]
12# 2        C    [5]     [v]

Mixed Aggregations: Lists and Scalars

Use .agg() with a dictionary to apply different functions to different columns:

1df = pd.DataFrame({
2    'category': ['A', 'A', 'B', 'B', 'C'],
3    'value': [10, 20, 30, 40, 50],
4    'name': ['x', 'y', 'z', 'w', 'v']
5})
6
7result = df.groupby('category').agg(
8    values=('value', list),
9    total=('value', 'sum'),
10    names=('name', list),
11    count=('name', 'count')
12).reset_index()
13
14print(result)
15#   category    values  total   names  count
16# 0        A  [10, 20]     30  [x, y]      2
17# 1        B  [30, 40]     70  [z, w]      2
18# 2        C      [50]     50     [v]      1

This named aggregation syntax (pandas 0.25+) is the cleanest way to mix list collection with scalar aggregations.

Collect Unique Values or Sorted Lists

1# Unique values only (no duplicates)
2df.groupby('category')['value'].apply(lambda x: list(set(x)))
3
4# Sorted lists
5df.groupby('category')['value'].apply(lambda x: sorted(x.tolist()))
6
7# Unique sorted values using a set
8df.groupby('category')['value'].apply(lambda x: sorted(set(x)))

Convert Lists to Strings

A common follow-up is joining list values into comma-separated strings:

1df.groupby('category')['name'].apply(', '.join).reset_index()
2#   category  name
3# 0        A  x, y
4# 1        B  z, w
5# 2        C     v
6
7# For non-string columns, convert first
8df.groupby('category')['value'].apply(lambda x: ', '.join(str(v) for v in x))

Group by Multiple Columns

1df = pd.DataFrame({
2    'dept': ['Sales', 'Sales', 'Sales', 'Eng', 'Eng'],
3    'region': ['East', 'East', 'West', 'East', 'West'],
4    'employee': ['Alice', 'Bob', 'Carol', 'Dave', 'Eve']
5})
6
7result = df.groupby(['dept', 'region'])['employee'].agg(list).reset_index()
8print(result)
9#     dept region     employee
10# 0    Eng   East       [Dave]
11# 1    Eng   West        [Eve]
12# 2  Sales   East  [Alice, Bob]
13# 3  Sales   West      [Carol]

Performance: apply(list) vs agg(list)

For large DataFrames, .agg(list) is generally faster than .apply(list) because agg is optimized internally:

1import numpy as np
2
3# Large DataFrame
4df_large = pd.DataFrame({
5    'key': np.random.choice(list('ABCDEFGHIJ'), size=1_000_000),
6    'val': np.random.randn(1_000_000)
7})
8
9# .agg(list) — typically 10-30% faster
10%timeit df_large.groupby('key')['val'].agg(list)
11
12# .apply(list) — slower due to apply overhead
13%timeit df_large.groupby('key')['val'].apply(list)

For very large datasets where you do not need actual Python lists, consider using tuple instead of list (tuples are slightly more memory-efficient) or restructuring your workflow to avoid collecting into lists altogether.

Common Pitfalls

• Lists in DataFrame cells are hard to query: Once values are collected into lists, filtering and joining become awkward. Consider whether a MultiIndex or a separate lookup table is a better design.
• .apply(list) returns a Series, not a DataFrame: If you need a DataFrame, chain .reset_index() or use .agg(list) on the full group.
• Memory with large groups: Collecting millions of values into lists can consume significant memory. If each group is very large, consider sampling or using generators instead.
• Column name conflicts with named agg: In named aggregation (agg(col_name=('col', func))), the output column name cannot match an existing groupby key name. Rename if needed.
• Non-hashable groupby keys: If the groupby column itself contains lists or dicts, groupby will fail. Convert to tuples or strings first.

Summary

• Use df.groupby('col')['val'].agg(list) to collect values into lists (fastest)
• Use .apply(list) as an equivalent alternative with slightly more overhead
• Use named aggregation agg(name=('col', list)) to mix list collection with scalar aggregations like sum or count
• Chain .reset_index() to convert the grouped result back into a flat DataFrame
• For string concatenation, use .apply(', '.join) instead of collecting into lists