Practice Now
Python
Text Processing
NLTK
pandas
Natural Language Processing

Python text processing NLTK and pandas

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Natural Language ProcessingData SciencePython

Introduction

NLTK and pandas are complementary tools for text processing in Python. NLTK handles natural language operations — tokenization, stemming, lemmatization, stopword removal, POS tagging, and sentiment analysis. pandas handles the data structure — loading text datasets, applying NLP functions across columns, filtering, grouping, and exporting results. The typical workflow is: load text data into a DataFrame with pandas, apply NLTK transformations using .apply() or .map(), and analyze the results with pandas aggregation.

Setup

python
1import nltk
2import pandas as pd
3
4# Download required NLTK data (first time only)
5nltk.download('punkt')
6nltk.download('punkt_tab')
7nltk.download('stopwords')
8nltk.download('wordnet')
9nltk.download('averaged_perceptron_tagger')
10nltk.download('vader_lexicon')

Tokenization

python
1from nltk.tokenize import word_tokenize, sent_tokenize
2
3text = "NLTK is a powerful library. It handles text processing well!"
4
5# Word tokenization
6words = word_tokenize(text)
7# ['NLTK', 'is', 'a', 'powerful', 'library', '.', 'It', 'handles', 'text', 'processing', 'well', '!']
8
9# Sentence tokenization
10sentences = sent_tokenize(text)
11# ['NLTK is a powerful library.', 'It handles text processing well!']
12
13# Apply to a DataFrame column
14df = pd.DataFrame({"text": [
15    "Python is great for NLP.",
16    "pandas handles data well.",
17    "NLTK provides many tools."
18]})
19
20df["tokens"] = df["text"].apply(word_tokenize)
21df["word_count"] = df["tokens"].apply(len)

Stopword Removal

python
1from nltk.corpus import stopwords
2from nltk.tokenize import word_tokenize
3
4stop_words = set(stopwords.words("english"))
5
6def remove_stopwords(text):
7    tokens = word_tokenize(text.lower())
8    return [w for w in tokens if w.isalpha() and w not in stop_words]
9
10df["filtered"] = df["text"].apply(remove_stopwords)
11# ['python', 'great', 'nlp']
12# ['pandas', 'handles', 'data', 'well']
13# ['nltk', 'provides', 'many', 'tools']

Stemming and Lemmatization

python
1from nltk.stem import PorterStemmer, WordNetLemmatizer
2from nltk.tokenize import word_tokenize
3
4stemmer = PorterStemmer()
5lemmatizer = WordNetLemmatizer()
6
7words = ["running", "flies", "better", "studies", "easily"]
8
9# Stemming — chops word endings (faster, less accurate)
10stemmed = [stemmer.stem(w) for w in words]
11# ['run', 'fli', 'better', 'studi', 'easili']
12
13# Lemmatization — returns dictionary form (slower, more accurate)
14lemmatized = [lemmatizer.lemmatize(w, pos="v") for w in words]
15# ['run', 'fly', 'better', 'study', 'easily']
16
17# Apply to DataFrame
18df["lemmatized"] = df["filtered"].apply(
19    lambda tokens: [lemmatizer.lemmatize(t) for t in tokens]
20)

Sentiment Analysis with VADER

python
1from nltk.sentiment.vader import SentimentIntensityAnalyzer
2
3sid = SentimentIntensityAnalyzer()
4
5reviews = pd.DataFrame({"review": [
6    "This product is amazing! I love it.",
7    "Terrible quality. Would not recommend.",
8    "It's okay, nothing special.",
9    "Best purchase I've ever made!",
10    "Completely broken on arrival. Waste of money."
11]})
12
13reviews["sentiment"] = reviews["review"].apply(
14    lambda x: sid.polarity_scores(x)["compound"]
15)
16reviews["label"] = reviews["sentiment"].apply(
17    lambda x: "positive" if x >= 0.05 else ("negative" if x <= -0.05 else "neutral")
18)
19
20print(reviews[["review", "sentiment", "label"]])

Full Text Processing Pipeline

python
1import pandas as pd
2from nltk.tokenize import word_tokenize
3from nltk.corpus import stopwords
4from nltk.stem import WordNetLemmatizer
5from collections import Counter
6
7stop_words = set(stopwords.words("english"))
8lemmatizer = WordNetLemmatizer()
9
10def preprocess(text):
11    tokens = word_tokenize(text.lower())
12    tokens = [lemmatizer.lemmatize(t) for t in tokens if t.isalpha() and t not in stop_words]
13    return tokens
14
15# Load data
16df = pd.read_csv("articles.csv")  # columns: title, body, category
17
18# Apply preprocessing
19df["tokens"] = df["body"].apply(preprocess)
20df["token_count"] = df["tokens"].apply(len)
21
22# Word frequency per category
23for category in df["category"].unique():
24    all_tokens = df[df["category"] == category]["tokens"].explode()
25    freq = Counter(all_tokens).most_common(10)
26    print(f"\n{category}: {freq}")

N-grams and Frequency Analysis

python
1from nltk import ngrams, FreqDist
2from nltk.tokenize import word_tokenize
3
4text = "the quick brown fox jumps over the lazy dog the quick brown fox"
5tokens = word_tokenize(text.lower())
6
7# Bigrams
8bigrams = list(ngrams(tokens, 2))
9bigram_freq = FreqDist(bigrams)
10print(bigram_freq.most_common(3))
11# [('the', 'quick'), ('quick', 'brown'), ('brown', 'fox')] — each count 2
12
13# Apply to DataFrame
14df["bigrams"] = df["tokens"].apply(lambda t: list(ngrams(t, 2)))

TF-IDF with pandas

python
1from sklearn.feature_extraction.text import TfidfVectorizer
2
3corpus = df["body"].tolist()
4
5vectorizer = TfidfVectorizer(max_features=100, stop_words="english")
6tfidf_matrix = vectorizer.fit_transform(corpus)
7
8# Convert to DataFrame for analysis
9tfidf_df = pd.DataFrame(
10    tfidf_matrix.toarray(),
11    columns=vectorizer.get_feature_names_out()
12)
13print(tfidf_df.head())

Common Pitfalls

  • Not downloading NLTK data before using it: Functions like word_tokenize and stopwords.words() require downloaded data files. Call nltk.download('punkt') and nltk.download('stopwords') before use, or you get LookupError.
  • Applying NLTK functions to NaN values in a DataFrame: If a text column contains NaN, word_tokenize(NaN) raises TypeError. Filter or fill NaN values first: df["text"].fillna("").apply(word_tokenize).
  • Using stemming when lemmatization is more appropriate: Stemming produces non-words ("studies" becomes "studi"). For tasks where readable output matters (search, display), use lemmatization. For bag-of-words classification where exact form does not matter, stemming is faster.
  • Not specifying the POS tag for lemmatization: WordNetLemmatizer.lemmatize("better") returns "better" by default (assumes noun). Pass pos="a" (adjective) to get "good". For best results, POS-tag words first with nltk.pos_tag() and map tags to WordNet POS.
  • Loading an entire large CSV into memory before processing: For large datasets, use pd.read_csv(chunksize=10000) to process in chunks. Applying NLTK to millions of rows at once can exhaust memory.

Summary

  • Use NLTK for tokenization, stopword removal, stemming/lemmatization, and sentiment analysis
  • Use pandas to load, structure, and aggregate text data with .apply() and .explode()
  • Preprocess text by lowercasing, tokenizing, removing stopwords, and lemmatizing
  • Use VADER (SentimentIntensityAnalyzer) for quick sentiment scoring without training data
  • Combine with scikit-learn's TfidfVectorizer for feature extraction in ML pipelines
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Basis Lab
No course covers this one yet

Basis Lab writes one for you from a sentence about what you want to be able to do, then teaches it and checks you understood. Your first course is free.

Build my course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.