Introduction

The important part of a DocumentTermMatrix workflow is not creating the training matrix once. It is making sure future test documents are transformed into the exact same feature space so the trained model sees the same columns in the same meaning.

The fix is to learn the vocabulary on the training corpus, save that vocabulary, and force new documents to use it. If you build a fresh matrix on the test set without that constraint, the resulting columns can drift and the model input becomes invalid.

Build the Training Matrix First

In R, the tm package creates a document-term matrix from a corpus plus preprocessing rules. Here is a small training example:

1library(tm)
2
3train_docs <- c(
4  "fresh apple juice",
5  "banana smoothie recipe",
6  "apple banana snack"
7)
8
9train_corpus <- VCorpus(VectorSource(train_docs))
10
11train_dtm <- DocumentTermMatrix(
12  train_corpus,
13  control = list(
14    tolower = TRUE,
15    removePunctuation = TRUE,
16    removeNumbers = TRUE,
17    wordLengths = c(1, Inf)
18  )
19)
20
21inspect(train_dtm)

At this stage, the training matrix defines the vocabulary and column order that the downstream model understands.

Reuse the Training Vocabulary for Test Data

When new documents arrive, the test matrix should be projected onto the training vocabulary instead of inventing its own set of terms.

1test_docs <- c(
2  "apple dessert",
3  "fresh banana"
4)
5
6test_corpus <- VCorpus(VectorSource(test_docs))
7
8test_dtm <- DocumentTermMatrix(
9  test_corpus,
10  control = list(
11    dictionary = Terms(train_dtm),
12    tolower = TRUE,
13    removePunctuation = TRUE,
14    removeNumbers = TRUE,
15    wordLengths = c(1, Inf)
16  )
17)
18
19inspect(test_dtm)

The key line is dictionary = Terms(train_dtm). That freezes the feature space. Words that were never seen during training are ignored, which is normally the correct behavior for scoring.

Keep Preprocessing Identical

Using the same dictionary is only half the solution. The cleaning pipeline must also stay identical between training and test data. If the training data lowercases text, removes punctuation, strips stop words, or stems tokens, the test data must go through the same sequence.

A reusable cleaning function helps:

1clean_corpus <- function(corpus) {
2  corpus <- tm_map(corpus, content_transformer(tolower))
3  corpus <- tm_map(corpus, removePunctuation)
4  corpus <- tm_map(corpus, removeNumbers)
5  corpus <- tm_map(corpus, stripWhitespace)
6  corpus
7}
8
9train_corpus <- clean_corpus(VCorpus(VectorSource(train_docs)))
10test_corpus <- clean_corpus(VCorpus(VectorSource(test_docs)))

If you apply different cleaning to test data, the dictionary still exists, but matching tokens may no longer line up correctly.

Save the Vocabulary for Later Scoring

In a real pipeline, do not rely on the training session still being in memory. Save the learned terms so they can be reused later.

1training_terms <- Terms(train_dtm)
2saveRDS(training_terms, "training_terms.rds")
3
4loaded_terms <- readRDS("training_terms.rds")
5
6test_dtm <- DocumentTermMatrix(
7  test_corpus,
8  control = list(dictionary = loaded_terms)
9)

This is the basic "fit once, transform many times" pattern. The training corpus fits the vocabulary. Future corpora transform into that frozen representation.

Think About Weighting Separately

If your model uses TF-IDF or another weighting scheme, the vocabulary and the weighting are related but not identical concerns. First get the matrix shape right, then apply the expected weighting consistently.

weighted_train <- weightTfIdf(train_dtm)
weighted_test <- weightTfIdf(test_dtm)

Consistency matters more than the specific weighting choice. A model trained on raw counts should not be scored on TF-IDF values, and vice versa.

Common Pitfalls

• Creating a brand-new test matrix with a brand-new vocabulary.
• Applying different preprocessing steps to training and test corpora.
• Forgetting to save the learned term list for later scoring.
• Ignoring column meaning and focusing only on matrix dimensions.
• Mixing weighted and unweighted matrices across training and prediction.

Summary

• The training corpus should define the vocabulary.
• Use dictionary = Terms(train_dtm) when creating the test DocumentTermMatrix.
• Keep text preprocessing identical across training and test data.
• Save the learned terms so future datasets can be transformed consistently.
• Treat the workflow as fit on training data, then transform every new dataset the same way.