Introduction

Hierarchical clustering is useful when you want to group variables that behave similarly without choosing the number of groups in advance. When the input is a correlation matrix, the key step is converting correlation into a distance measure that SciPy can cluster correctly.

Correlation Is Similarity, Not Distance

A correlation coefficient tells you how strongly two variables move together. Clustering algorithms in scipy.cluster.hierarchy expect distances, where smaller values mean more similar observations.

A common conversion is distance = 1 - correlation. That works well when strong positive correlation should mean closeness. If negative correlation should also count as strong similarity, use distance = 1 - abs(correlation) instead.

The choice depends on the question:

• use 1 - correlation when positive and negative relationships should stay separate
• use 1 - abs(correlation) when both directions represent strong association

Build The Correlation Matrix

Assume the columns of a DataFrame are the variables you want to cluster.

1import pandas as pd
2
3frame = pd.DataFrame(
4    {
5        "a": [1, 2, 3, 4, 5],
6        "b": [2, 4, 6, 8, 10],
7        "c": [5, 4, 3, 2, 1],
8        "d": [1, 1, 2, 2, 3],
9    }
10)
11
12corr = frame.corr()
13print(corr)

This gives you a square matrix where rows and columns represent the same variable set. SciPy does not want that full square matrix in linkage. It wants either raw observations or a condensed distance vector.

Convert To A Condensed Distance Vector

The usual workflow is:

1. compute the correlation matrix
2. transform it into a distance matrix
3. convert that square matrix into condensed form with squareform
4. call linkage

1import numpy as np
2from scipy.cluster.hierarchy import linkage, dendrogram
3from scipy.spatial.distance import squareform
4import matplotlib.pyplot as plt
5
6corr = frame.corr()
7distance = 1 - corr
8
9# Numerical cleanup for the diagonal
10np.fill_diagonal(distance.values, 0.0)
11
12condensed = squareform(distance.values, checks=False)
13Z = linkage(condensed, method="average")
14
15plt.figure(figsize=(8, 4))
16dendrogram(Z, labels=corr.columns.tolist())
17plt.tight_layout()
18plt.show()

The call to squareform is important. Passing the square distance matrix directly to linkage often leads to wrong results because SciPy may interpret it as observations rather than pairwise distances.

Choosing The Linkage Method

The method argument changes how cluster distances are updated:

• 'single uses the closest pair between clusters'
• 'complete uses the farthest pair'
• 'average uses the mean pairwise distance'
• 'ward is popular, but it assumes Euclidean geometry and is usually not the right default for correlation-derived distances'

For correlation clustering, average is a sensible starting point because it is stable and easy to explain.

Cluster Rows Instead Of Columns

Sometimes you want to cluster observations, not variables. In that case, correlate rows by transposing first.

1row_corr = frame.T.corr()
2row_distance = 1 - row_corr
3np.fill_diagonal(row_distance.values, 0.0)
4row_condensed = squareform(row_distance.values, checks=False)
5row_Z = linkage(row_condensed, method="complete")
6print(row_Z[:3])

This pattern is useful in gene expression, user behavior analysis, and any setting where each row is an entity with multiple measured features.

Interpret The Dendrogram Carefully

A dendrogram shows merge order and merge distance. It does not automatically tell you the correct number of clusters. You still need domain judgment or a cutoff rule.

If two variables merge very low in the tree, they are strongly related under your chosen distance definition. If a variable joins the tree much higher up, it behaves differently from the rest.

Common Pitfalls

The most common mistake is feeding linkage a correlation matrix directly. A correlation matrix is neither raw observations nor a valid condensed distance vector, so the result is easy to misread.

Another mistake is forgetting that negative correlation may be either similarity or dissimilarity depending on the problem. The difference between 1 - correlation and 1 - abs(correlation) changes the cluster structure substantially.

A final issue is using ward with a non-Euclidean distance transformation. That combination is mathematically inconsistent for many correlation-based workflows and can produce misleading structure.

Summary

• Compute a correlation matrix first, then convert it to distance.
• Use squareform before passing pairwise distances to linkage.
• Choose between 1 - correlation and 1 - abs(correlation) based on what similarity means in your problem.
• 'average linkage is usually a safer default than ward for correlation clustering.'
• Read the dendrogram as a hierarchy, not as an automatic cluster count.