Introduction

Cosine similarity measures how aligned two vectors are, independent of their magnitude. For tensors, the calculation depends on what you mean by “between two tensors”: you may want a single similarity score for the entire tensors, or one score per row, channel, token, or batch element.

The math is the same in every case. Compute the dot product along the dimension of interest and divide by the product of the corresponding norms.

The Formula

For two non-zero vectors a and b, cosine similarity is:

cos(a, b) = dot(a, b) / (||a|| * ||b||)

The result is:

• '1 if the vectors point in the same direction.'
• '0 if they are orthogonal.'
• '-1 if they point in opposite directions.'

If your inputs are higher-rank tensors, first decide which axis represents the vector dimension.

Whole-Tensor Similarity

If you want one score for the entire tensors, flatten both tensors into one-dimensional vectors first.

1import torch
2
3x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
4y = torch.tensor([[2.0, 4.0], [6.0, 8.0]])
5
6x_flat = x.reshape(-1)
7y_flat = y.reshape(-1)
8
9similarity = torch.dot(x_flat, y_flat) / (torch.norm(x_flat) * torch.norm(y_flat))
10print(float(similarity))

This returns 1.0 because y is a scalar multiple of x, so they point in the same direction after flattening.

Row-Wise or Batch-Wise Similarity

In machine-learning code, you often have tensors shaped like batch_size x embedding_dim. In that case you usually want one cosine similarity value per row, not one score for the entire matrix.

PyTorch provides this directly.

1import torch
2import torch.nn.functional as F
3
4x = torch.tensor([[1.0, 0.0, 1.0],
5                  [1.0, 1.0, 0.0]])
6y = torch.tensor([[1.0, 1.0, 0.0],
7                  [1.0, 0.0, 1.0]])
8
9scores = F.cosine_similarity(x, y, dim=1)
10print(scores)

Here dim=1 means each row is treated as a vector and the output contains one similarity score per row.

That “choose the dimension” step is the part people most often get wrong. The formula is easy; selecting the correct axis is the real design decision.

Manual Computation With Stability Protection

When implementing the formula yourself, protect against zero vectors to avoid division by zero.

1import torch
2
3
4def cosine_similarity_safe(a, b, dim=-1, eps=1e-8):
5    dot = torch.sum(a * b, dim=dim)
6    norm_a = torch.norm(a, dim=dim)
7    norm_b = torch.norm(b, dim=dim)
8    return dot / (norm_a * norm_b).clamp_min(eps)
9
10
11x = torch.tensor([[1.0, 2.0], [0.0, 0.0]])
12y = torch.tensor([[2.0, 1.0], [1.0, 1.0]])
13print(cosine_similarity_safe(x, y, dim=1))

Using a small eps is common in neural-network code because real data can include all-zero embeddings or masked values.

Interpreting the Result

Cosine similarity ignores scale. That is useful for embeddings, text vectors, and normalized feature spaces where direction matters more than raw magnitude.

It is less useful when vector length itself carries important information. Two tensors can have cosine similarity 1 even if one is ten times larger than the other.

Also remember that if you flatten structured tensors, you are discarding spatial or channel semantics. For an image tensor, a single whole-tensor score may be mathematically valid but conceptually meaningless depending on the task.

Common Pitfalls

A common mistake is comparing tensors of incompatible shapes and assuming broadcasting produced the intended result. Always confirm that the compared dimension matches your conceptual vector dimension.

Another issue is forgetting to handle zero vectors. The raw formula divides by the norm, so all-zero inputs must be handled explicitly or stabilized with an epsilon.

Developers also often flatten tensors too early. If the data is batched, flattening everything together destroys the per-example structure and produces one aggregate score instead of a score per sample.

Finally, do not confuse cosine similarity with Euclidean distance. They answer different questions and often produce different rankings.

Summary

• Cosine similarity is the normalized dot product of two vectors.
• For whole-tensor similarity, flatten both tensors first.
• For batched data, compute along the feature dimension rather than flattening everything.
• Use framework helpers such as F.cosine_similarity when available.
• Protect against zero vectors and choose the comparison axis deliberately.