Introduction

Predicting whether a page or article will perform well is a useful ranking problem, but only if "good content" is defined in measurable business terms. With scikit-learn, you can build a model that combines text features and metadata to estimate whether new content is likely to drive engagement, conversions, or retention.

Define The Label Before Writing Any Model Code

The first technical step is not model selection. It is deciding what good means. A practical label might be:

• '1 if an article exceeds a threshold for conversions, time on page, or return visits.'
• '0 otherwise.'

That label should be created from data collected after publication, while the features should come only from information available before or at publish time. This separation prevents target leakage.

For example, you might start with a table like this:

1import pandas as pd
2
3df = pd.DataFrame(
4    [
5        {
6            "title": "How to cache API responses",
7            "body": "Caching reduces latency and load on backend services.",
8            "category": "backend",
9            "author": "alice",
10            "word_count": 680,
11            "published_hour": 9,
12            "good": 1,
13        },
14        {
15            "title": "Weekly engineering update",
16            "body": "Internal process changes and team announcements.",
17            "category": "company",
18            "author": "bob",
19            "word_count": 420,
20            "published_hour": 17,
21            "good": 0,
22        },
23    ]
24)

Build A Mixed Text And Metadata Pipeline

Scikit-learn works well when you keep preprocessing and modeling inside one pipeline. For content prediction, a strong baseline is:

• 'TfidfVectorizer for title and body text.'
• 'OneHotEncoder for categorical metadata.'
• A simple classifier such as logistic regression.

1import pandas as pd
2from sklearn.compose import ColumnTransformer
3from sklearn.feature_extraction.text import TfidfVectorizer
4from sklearn.linear_model import LogisticRegression
5from sklearn.model_selection import train_test_split
6from sklearn.pipeline import Pipeline
7from sklearn.preprocessing import OneHotEncoder
8
9X = df[["body", "category", "author", "word_count", "published_hour"]]
10y = df["good"]
11
12preprocess = ColumnTransformer(
13    transformers=[
14        ("body", TfidfVectorizer(max_features=5000, ngram_range=(1, 2)), "body"),
15        ("cat", OneHotEncoder(handle_unknown="ignore"), ["category", "author"]),
16        ("num", "passthrough", ["word_count", "published_hour"]),
17    ]
18)
19
20model = Pipeline(
21    steps=[
22        ("preprocess", preprocess),
23        ("clf", LogisticRegression(max_iter=1000)),
24    ]
25)
26
27X_train, X_test, y_train, y_test = train_test_split(
28    X, y, test_size=0.2, random_state=42, stratify=y
29)
30
31model.fit(X_train, y_train)
32print(model.score(X_test, y_test))

This is not the final model, but it is a credible baseline that is easy to inspect and improve.

Predict Probabilities, Not Just Classes

Editorial teams usually benefit more from confidence scores than from yes-or-no labels:

1new_articles = pd.DataFrame(
2    [
3        {
4            "body": "A tutorial on HTTP caching and stale-while-revalidate.",
5            "category": "backend",
6            "author": "alice",
7            "word_count": 900,
8            "published_hour": 10,
9        }
10    ]
11)
12
13proba = model.predict_proba(new_articles)[:, 1]
14print(proba[0])

A probability can feed content ranking, human review queues, or A/B testing decisions. It also makes threshold tuning easier when the cost of false positives and false negatives is asymmetric.

Evaluate The Right Metrics

Accuracy alone is usually not enough. If only 10 percent of articles count as good, a naive model that predicts 0 for everything can still look accurate.

Use metrics aligned with the business goal:

• Precision if editorial review time is limited and false positives are expensive.
• Recall if you want to catch as many high-potential articles as possible.
• ROC AUC or PR AUC when class balance is skewed.

Scikit-learn makes this easy:

1from sklearn.metrics import classification_report, roc_auc_score
2
3pred = model.predict(X_test)
4prob = model.predict_proba(X_test)[:, 1]
5
6print(classification_report(y_test, pred))
7print("roc_auc:", roc_auc_score(y_test, prob))

Feature Ideas That Usually Help

Useful features often include readability scores, article age category, presence of code blocks, internal-link count, topic tags, and source channel. If you have historical ranking data, headline length and semantic embeddings may also help.

Start simple before reaching for deep learning. In many editorial systems, clean metadata and a strong baseline model outperform a more complex model trained on noisy labels.

Common Pitfalls

The biggest pitfall is label leakage. If you use page views, shares, or bounce rate as input features while also using them to define good, the model will appear excellent and be useless for pre-publication prediction.

Another common issue is training on too little data. Text models with many sparse features need enough examples to generalize. If your dataset is small, reduce feature complexity and focus on interpretable baselines.

Developers also often ignore drift. Content that performed well last year may not match current audience interests, search behavior, or platform changes. Retrain and re-evaluate regularly.

Finally, remember that "good content" is partly a product decision, not just a machine-learning task. The model only reflects the target you defined.

Summary

• Define good with a measurable label before building the model.
• Use only features available at prediction time to avoid leakage.
• A scikit-learn pipeline with TF-IDF, metadata, and logistic regression is a strong baseline.
• Predict probabilities so teams can rank and review content more effectively.
• Evaluate with metrics that match the editorial or business goal, not accuracy alone.