Introduction
NLTK and pandas are complementary tools for text processing in Python. NLTK handles natural language operations — tokenization, stemming, lemmatization, stopword removal, POS tagging, and sentiment analysis. pandas handles the data structure — loading text datasets, applying NLP functions across columns, filtering, grouping, and exporting results. The typical workflow is: load text data into a DataFrame with pandas, apply NLTK transformations using .apply() or .map(), and analyze the results with pandas aggregation.
Setup
1import nltk
2import pandas as pd
3
4# Download required NLTK data (first time only)
5nltk.download('punkt')
6nltk.download('punkt_tab')
7nltk.download('stopwords')
8nltk.download('wordnet')
9nltk.download('averaged_perceptron_tagger')
10nltk.download('vader_lexicon')
Tokenization
1from nltk.tokenize import word_tokenize, sent_tokenize
2
3text = "NLTK is a powerful library. It handles text processing well!"
4
5# Word tokenization
6words = word_tokenize(text)
7# ['NLTK', 'is', 'a', 'powerful', 'library', '.', 'It', 'handles', 'text', 'processing', 'well', '!']
8
9# Sentence tokenization
10sentences = sent_tokenize(text)
11# ['NLTK is a powerful library.', 'It handles text processing well!']
12
13# Apply to a DataFrame column
14df = pd.DataFrame({"text": [
15 "Python is great for NLP.",
16 "pandas handles data well.",
17 "NLTK provides many tools."
18]})
19
20df["tokens"] = df["text"].apply(word_tokenize)
21df["word_count"] = df["tokens"].apply(len)
Stopword Removal
1from nltk.corpus import stopwords
2from nltk.tokenize import word_tokenize
3
4stop_words = set(stopwords.words("english"))
5
6def remove_stopwords(text):
7 tokens = word_tokenize(text.lower())
8 return [w for w in tokens if w.isalpha() and w not in stop_words]
9
10df["filtered"] = df["text"].apply(remove_stopwords)
11# ['python', 'great', 'nlp']
12# ['pandas', 'handles', 'data', 'well']
13# ['nltk', 'provides', 'many', 'tools']
Stemming and Lemmatization
1from nltk.stem import PorterStemmer, WordNetLemmatizer
2from nltk.tokenize import word_tokenize
3
4stemmer = PorterStemmer()
5lemmatizer = WordNetLemmatizer()
6
7words = ["running", "flies", "better", "studies", "easily"]
8
9# Stemming — chops word endings (faster, less accurate)
10stemmed = [stemmer.stem(w) for w in words]
11# ['run', 'fli', 'better', 'studi', 'easili']
12
13# Lemmatization — returns dictionary form (slower, more accurate)
14lemmatized = [lemmatizer.lemmatize(w, pos="v") for w in words]
15# ['run', 'fly', 'better', 'study', 'easily']
16
17# Apply to DataFrame
18df["lemmatized"] = df["filtered"].apply(
19 lambda tokens: [lemmatizer.lemmatize(t) for t in tokens]
20)
Sentiment Analysis with VADER
1from nltk.sentiment.vader import SentimentIntensityAnalyzer
2
3sid = SentimentIntensityAnalyzer()
4
5reviews = pd.DataFrame({"review": [
6 "This product is amazing! I love it.",
7 "Terrible quality. Would not recommend.",
8 "It's okay, nothing special.",
9 "Best purchase I've ever made!",
10 "Completely broken on arrival. Waste of money."
11]})
12
13reviews["sentiment"] = reviews["review"].apply(
14 lambda x: sid.polarity_scores(x)["compound"]
15)
16reviews["label"] = reviews["sentiment"].apply(
17 lambda x: "positive" if x >= 0.05 else ("negative" if x <= -0.05 else "neutral")
18)
19
20print(reviews[["review", "sentiment", "label"]])
Full Text Processing Pipeline
1import pandas as pd
2from nltk.tokenize import word_tokenize
3from nltk.corpus import stopwords
4from nltk.stem import WordNetLemmatizer
5from collections import Counter
6
7stop_words = set(stopwords.words("english"))
8lemmatizer = WordNetLemmatizer()
9
10def preprocess(text):
11 tokens = word_tokenize(text.lower())
12 tokens = [lemmatizer.lemmatize(t) for t in tokens if t.isalpha() and t not in stop_words]
13 return tokens
14
15# Load data
16df = pd.read_csv("articles.csv") # columns: title, body, category
17
18# Apply preprocessing
19df["tokens"] = df["body"].apply(preprocess)
20df["token_count"] = df["tokens"].apply(len)
21
22# Word frequency per category
23for category in df["category"].unique():
24 all_tokens = df[df["category"] == category]["tokens"].explode()
25 freq = Counter(all_tokens).most_common(10)
26 print(f"\n{category}: {freq}")
N-grams and Frequency Analysis
1from nltk import ngrams, FreqDist
2from nltk.tokenize import word_tokenize
3
4text = "the quick brown fox jumps over the lazy dog the quick brown fox"
5tokens = word_tokenize(text.lower())
6
7# Bigrams
8bigrams = list(ngrams(tokens, 2))
9bigram_freq = FreqDist(bigrams)
10print(bigram_freq.most_common(3))
11# [('the', 'quick'), ('quick', 'brown'), ('brown', 'fox')] — each count 2
12
13# Apply to DataFrame
14df["bigrams"] = df["tokens"].apply(lambda t: list(ngrams(t, 2)))
TF-IDF with pandas
1from sklearn.feature_extraction.text import TfidfVectorizer
2
3corpus = df["body"].tolist()
4
5vectorizer = TfidfVectorizer(max_features=100, stop_words="english")
6tfidf_matrix = vectorizer.fit_transform(corpus)
7
8# Convert to DataFrame for analysis
9tfidf_df = pd.DataFrame(
10 tfidf_matrix.toarray(),
11 columns=vectorizer.get_feature_names_out()
12)
13print(tfidf_df.head())
Common Pitfalls
Not downloading NLTK data before using it: Functions like word_tokenize and stopwords.words() require downloaded data files. Call nltk.download('punkt') and nltk.download('stopwords') before use, or you get LookupError.
Applying NLTK functions to NaN values in a DataFrame: If a text column contains NaN, word_tokenize(NaN) raises TypeError. Filter or fill NaN values first: df["text"].fillna("").apply(word_tokenize).
Using stemming when lemmatization is more appropriate: Stemming produces non-words ("studies" becomes "studi"). For tasks where readable output matters (search, display), use lemmatization. For bag-of-words classification where exact form does not matter, stemming is faster.
Not specifying the POS tag for lemmatization: WordNetLemmatizer.lemmatize("better") returns "better" by default (assumes noun). Pass pos="a" (adjective) to get "good". For best results, POS-tag words first with nltk.pos_tag() and map tags to WordNet POS.
Loading an entire large CSV into memory before processing: For large datasets, use pd.read_csv(chunksize=10000) to process in chunks. Applying NLTK to millions of rows at once can exhaust memory.
Summary
Use NLTK for tokenization, stopword removal, stemming/lemmatization, and sentiment analysis
Use pandas to load, structure, and aggregate text data with .apply() and .explode()
Preprocess text by lowercasing, tokenizing, removing stopwords, and lemmatizing
Use VADER (SentimentIntensityAnalyzer) for quick sentiment scoring without training data
Combine with scikit-learn's TfidfVectorizer for feature extraction in ML pipelines
.png&w=3840&q=75)