Introduction

If you want large cross-validation scores from LinearRegression in scikit-learn, the answer is not to “game” cross-validation but to make the model actually fit the data well. High scores come from the right problem setup, the right evaluation metric, and features that are close enough to a linear relationship for ordinary least squares to work.

Know What Score You Are Looking At

With regression, scikit-learn’s default score is usually R^2. That matters because “large” does not mean the same thing as it would for an accuracy percentage.

For R^2:

• '1.0 is perfect'
• values near 0 mean the model is barely better than predicting the mean
• negative values mean it performs worse than that baseline

So the first task is making sure you are interpreting the cross-validation output correctly.

1import numpy as np
2from sklearn.datasets import make_regression
3from sklearn.linear_model import LinearRegression
4from sklearn.model_selection import cross_val_score
5
6X, y = make_regression(
7    n_samples=300,
8    n_features=5,
9    noise=10.0,
10    random_state=42,
11)
12
13model = LinearRegression()
14scores = cross_val_score(model, X, y, cv=5, scoring="r2")
15
16print(scores)
17print("Mean R^2:", np.mean(scores))

If your expectation of “large” is unrealistic for the noise in the data, the model may be working fine already.

Linear Regression Can Only Learn Linear Structure

A LinearRegression model assumes the target can be explained as a linear combination of the features. If the real relationship is nonlinear, no amount of cross-validation tweaking will produce consistently high scores.

This is why good cross-validation scores often come from:

• strong feature engineering
• removing obviously irrelevant features
• transforming variables so the relationship becomes more linear

For example, polynomial features can help if the raw relationship is curved:

1import numpy as np
2from sklearn.pipeline import make_pipeline
3from sklearn.preprocessing import PolynomialFeatures
4from sklearn.linear_model import LinearRegression
5from sklearn.model_selection import cross_val_score
6
7X = np.random.rand(300, 1)
8y = 4 * X[:, 0] ** 2 + 2 * X[:, 0] + np.random.randn(300) * 0.1
9
10model = make_pipeline(
11    PolynomialFeatures(degree=2, include_bias=False),
12    LinearRegression(),
13)
14
15scores = cross_val_score(model, X, y, cv=5, scoring="r2")
16print("Mean R^2:", scores.mean())

The estimator is still linear in the transformed features, which is why the score improves.

Use a Pipeline to Avoid Leakage

If preprocessing is involved, put it inside a pipeline so each fold learns transformations only from its own training split.

1from sklearn.pipeline import make_pipeline
2from sklearn.preprocessing import StandardScaler
3from sklearn.linear_model import LinearRegression
4from sklearn.model_selection import cross_val_score
5
6pipeline = make_pipeline(
7    StandardScaler(),
8    LinearRegression(),
9)
10
11scores = cross_val_score(pipeline, X, y, cv=5, scoring="r2")
12print(scores.mean())

For ordinary least squares, scaling often does not change the predictive fit much, but pipelines still matter whenever preprocessing is part of the workflow. More importantly, the pipeline keeps evaluation honest.

Pick the Right Cross-Validation Strategy

Not every dataset should use the same splitter. If the rows are time-ordered, random k-fold can give misleadingly optimistic scores. If repeated measurements from the same entity appear in multiple folds, leakage can happen even if the model code itself is correct.

Large, trustworthy cross-validation scores come from the right evaluation design, not just a good algorithm.

Use:

• standard k-fold for ordinary iid data
• grouped splits when samples from the same group must stay together
• time-series splits for temporal data

If the split is wrong, the score is not informative even when it looks impressive.

Improve the Data Before You Blame the Model

Linear regression is sensitive to data quality issues such as:

• strong outliers
• missing-value handling mistakes
• duplicate or mismatched rows
• target leakage

Sometimes the path to better cross-validation scores is not a fancier model. It is fixing the dataset.

A quick residual plot or scatter plot often teaches more than repeated CV runs do.

Regularization May Generalize Better

If the plain linear model is unstable or suffers from multicollinearity, regularized models such as Ridge or Lasso can generalize better and therefore improve cross-validation scores.

1from sklearn.linear_model import Ridge
2from sklearn.model_selection import cross_val_score
3
4ridge = Ridge(alpha=1.0)
5scores = cross_val_score(ridge, X, y, cv=5, scoring="r2")
6print(scores.mean())

This is especially helpful when features are correlated or when the model needs a bit more bias in exchange for less variance.

Common Pitfalls

• Chasing a “large” score without checking which metric is actually being reported.
• Expecting ordinary linear regression to perform well on a strongly nonlinear relationship.
• Performing preprocessing outside a pipeline and leaking information across folds.
• Using the wrong cross-validation splitter for grouped or time-dependent data.
• Treating noisy or low-signal data as if it should naturally produce near-perfect R^2.

Summary

• High cross-validation scores come from good data, good splits, and a relationship the model can actually learn.
• For regression, understand whether you are looking at R^2 or another metric before interpreting the result.
• Use feature engineering or transformed features when the raw relationship is not linear enough.
• Put preprocessing inside a pipeline so cross-validation stays honest.
• If plain linear regression underperforms, regularized linear models may generalize better.