Inner product geometry

Deep Breakdown

What it tells me (Situation Awareness): The formula $\cos\theta = \frac{\langle x, y\rangle}{\|x\|\|y\|}$ connects the inner product (dot product) of two vectors to the geometric angle $\theta$ between them. It is the bridge between algebra and geometry in inner product spaces. When $\theta = 0°$: vectors are parallel, $\cos\theta = 1$, inner product is maximized. When $\theta = 90°$: vectors are orthogonal, $\cos\theta = 0$, inner product is zero. When $\theta = 180°$: vectors are antiparallel, $\cos\theta = -1$. In DA/DS this is the foundational formula behind cosine similarity, which is the standard similarity measure for text embeddings, recommendation systems, and any scenario where direction matters more than magnitude.

Flowchart

Real-world use

Where it is used

Cosine similarity in NLP/recommendation (TF-IDF vectors, word embeddings, BERT embeddings), measuring feature alignment in PCA (eigenvectors are orthogonal by construction), validating that model residuals are orthogonal to fitted values (OLS guarantee), and computing angles between gradient vectors to detect conflicting gradient directions in multi-task learning.

Problem it solves

The raw inner product $\langle x, y\rangle$ conflates direction with magnitude: two short vectors with small angle can give a smaller dot product than two long vectors with a large angle. Dividing by $\|x\|\|y\|$ normalizes the result to $[-1, 1]$, giving a pure measure of directional alignment independent of vector length.

How it solves the problem

Rearranged: $\langle x, y\rangle = \|x\|\|y\|\cos\theta$. The inner product is $\|x\|\|y\|$ (maximum possible value, when $\theta=0$) scaled by $\cos\theta \in [-1,1]$. Dividing by $\|x\|\|y\|$ strips the magnitude, leaving only $\cos\theta$ — the pure directional signal. This is why normalizing embeddings to unit length before taking dot products gives cosine similarity.

Example in a business

A DS builds a document recommendation engine. Each document is a TF-IDF vector. To find documents similar to a query, they compute $\cos\theta = \frac{\langle q, d_i\rangle}{\|q\|\|d_i\|}$ for each document $d_i$. Documents with $\cos\theta > 0.85$ are returned as recommendations. Long and short documents are treated equally because the formula normalizes by length.

The formula, piece by piece

• $\langle x, y\rangle$: the inner product (dot product) of $x$ and $y$; for $\mathbb{R}^n$: $\sum_{i=1}^n x_i y_i$
• $\|x\|$: the norm (length) of $x$; equals $\sqrt{\langle x, x\rangle}$
• $\|y\|$: the norm (length) of $y$
• $\|x\|\|y\|$: the product of both lengths; the maximum possible value of $|\langle x,y\rangle|$ (by Cauchy-Schwarz)
• $\cos\theta$: the cosine of the angle between $x$ and $y$; always in $[-1, 1]$
• $\theta \in [0°, 180°]$: the angle between the two vectors (always taken as the smaller angle)
• The formula: $\cos\theta = \frac{\langle x, y\rangle}{\|x\|\|y\|}$; equivalently, $\theta = \arccos\!\left(\frac{\langle x,y\rangle}{\|x\|\|y\|}\right)$

The formula in words

"The cosine of the angle between two vectors equals their dot product divided by the product of their lengths. Scale both vectors to unit length first, then their dot product IS the cosine of their angle." Tiny example: $x = (1, 0)$, $y = (1, 1)$. $\langle x, y\rangle = 1\cdot1 + 0\cdot1 = 1$. $\|x\| = 1$, $\|y\| = \sqrt{2}$. $\cos\theta = \frac{1}{1 \cdot \sqrt{2}} = \frac{1}{\sqrt{2}} \approx 0.707$. $\theta = \arccos(0.707) = 45°$. $\checkmark$

Critical Concepts

• Cauchy-Schwarz bound: $|\langle x,y\rangle| \leq \|x\|\|y\|$, which guarantees $\cos\theta \in [-1, 1]$.
• Cosine similarity vs. cosine distance: cosine similarity = $\cos\theta \in [-1, 1]$; cosine distance = $1 - \cos\theta \in [0, 2]$.
• Unit vectors: if $\|x\| = \|y\| = 1$, then $\cos\theta = \langle x, y\rangle$ directly. This is why embedding normalization simplifies similarity search to dot products.
• Symmetry: $\cos\theta(x, y) = \cos\theta(y, x)$ because both $\langle x, y\rangle$ and $\|x\|\|y\|$ are symmetric.
• Scale invariance: $\cos\theta(\alpha x, \beta y) = \cos\theta(x, y)$ for $\alpha, \beta > 0$. The angle is unchanged by positive scaling.
• OLS residuals: the OLS condition $X^T(y - \hat{y}) = 0$ is exactly saying the residual vector is orthogonal to every column of $X$, i.e., $\cos\theta = 0$.

Step-by-step calculation (tiny example)

Step 1: Choose two vectors and compute their dot product $x = (3, 4)$, $y = (4, -3)$. $\langle x, y\rangle = 3\cdot4 + 4\cdot(-3) = 12 - 12 = 0$.

Step 2: Compute the norms $\|x\| = \sqrt{9+16} = 5$. $\|y\| = \sqrt{16+9} = 5$.

Step 3: Compute cosine and interpret $\cos\theta = \frac{0}{5 \cdot 5} = 0$, so $\theta = 90°$. These two vectors are orthogonal, which we can verify geometrically: $(3,4)$ and $(4,-3)$ are perpendicular in the plane.

Code examples

Beginner example (Python)

import numpy as np   # -- Cosine of angle between two vectors ---
x = np.array([3.0, 4.0])
y = np.array([4.0, -3.0])
dot_xy = np.dot(x, y)
norm_x = np.linalg.norm(x)
norm_y = np.linalg.norm(y)
cos_theta = dot_xy / (norm_x * norm_y)
theta_deg = np.degrees(np.arccos(np.clip(cos_theta, -1.0, 1.0)))
print(f"x = {x}, y = {y}")
print(f"<x,y> = {dot_xy:.4f}")
print(f"||x|| = {norm_x:.4f}, ||y|| = {norm_y:.4f}")
print(f"cos(theta) = {cos_theta:.4f}")
print(f"theta = {theta_deg:.2f} degrees")   # Verify: unit vectors
x_unit = x / norm_x
y_unit = y / norm_y
print(f"\nUnit vectors: <x_hat, y_hat> = {np.dot(x_unit, y_unit):.4f} (= cos_theta directly)")

Beginner example (R)

# -- Cosine of angle between two vectors ---
x <- c(3, 4)
y <- c(4, -3)
dot_xy <- sum(x * y)
norm_x <- sqrt(sum(x^2))
norm_y <- sqrt(sum(y^2))
cos_theta <- dot_xy / (norm_x * norm_y)
theta_deg <- acos(pmax(pmin(cos_theta, 1), -1)) * 180 / pi
cat(sprintf("<x,y> = %.4f\n", dot_xy))
cat(sprintf("||x|| = %.4f, ||y|| = %.4f\n", norm_x, norm_y))
cat(sprintf("cos(theta) = %.4f\n", cos_theta))
cat(sprintf("theta = %.2f degrees\n", theta_deg))

Professional example (Python)

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
np.random.seed(42)   # -- Business scenario: document similarity via cosine angle ---
# A content recommendation engine uses cosine similarity between TF-IDF vectors.
documents = [ "machine learning model training optimization", "deep learning neural networks gradient descent", "data analysis business intelligence dashboard", "SQL database query optimization performance", "python pandas dataframe aggregation statistics", "machine learning feature engineering selection",
]
query = "machine learning training gradient optimization"
vectorizer = TfidfVectorizer()
doc_matrix = vectorizer.fit_transform(documents).toarray()
query_vec = vectorizer.transform([query]).toarray()[0]   # Compute cosine similarity for each document
results = []
for i, doc_vec in enumerate(doc_matrix):
    dot_product = np.dot(query_vec, doc_vec)
norm_q = np.linalg.norm(query_vec)
norm_d = np.linalg.norm(doc_vec)
denom = norm_q * norm_d
cos_sim = dot_product / denom if denom > 0 else 0.0
angle_deg = np.degrees(np.arccos(np.clip(cos_sim, -1, 1)))
results.append({ "document": documents[i], "cos_similarity": round(cos_sim, 4), "angle_degrees": round(angle_deg, 2), })
results_df = pd.DataFrame(results).sort_values("cos_similarity", ascending=False)
print(f"Query: '{query}'\n")
print(results_df.to_string(index=False))
print("\nHigher cosine similarity = smaller angle = more aligned direction.")

Source: MML1 p. 80

#linear-algebra#spatial-distance-norms-and-analytic-geometry#angles-and-orthogonality