Projection (Encoding)

We are working on the eigenvector equation for Principal Component Analysis, PCA, defined by

where $S$ is the sample covariance matrix, $v$ is an eigenvector, and $\lambda$ is the corresponding eigenvalue. In PCA, the eigenvectors of $S$ are the principal directions, or principal axes, and the eigenvalues give the variance explained along those directions. This is the central linear-algebra equation behind PCA, covariance-based dimensionality reduction, and many factor-style analyses.

Deep Breakdown

The eigenvector equation is the mathematical heart of PCA. After we center the data and compute the sample covariance matrix, PCA asks a very specific question: > In which direction does the data vary the most? That direction is the first principal component. Then PCA asks: > What is the next direction of greatest variance, subject to being orthogonal to the first? That becomes the second principal component, and so on. The answer to these questions comes from the eigenvector equation:

where:

• $S$ is the sample covariance matrix,
• $v$ is a direction vector,
• $\lambda$ is a scalar.

This equation says that when the covariance matrix acts on the vector $v$, the result points in exactly the same direction as $v$, only rescaled by $\lambda$. That is what makes $v$ an eigenvector.

Why this matters in PCA

The covariance matrix describes how the centered features vary together. If we take a unit direction vector $v$ and project the data onto that direction, the variance of the projected data is:

PCA chooses the direction $v$ that maximizes this variance, under the constraint that:

The optimization solution leads directly to:

So PCA is not using eigenvectors by accident. The eigenvectors are exactly the solutions to the variance-maximization problem.

What the eigenvalue means

Each eigenvector has a corresponding eigenvalue:

The eigenvalue $\lambda$ tells us how much variance exists in that principal direction.

So:

• larger $\lambda$ means more variance explained,
• smaller $\lambda$ means less variance explained.

This is why PCA orders components by decreasing eigenvalue.

If:

then:

• $v_1$ is the first principal axis,
• $v_2$ is the second principal axis,
• and so on.

Geometric meaning

The covariance matrix stretches space differently in different directions. Eigenvectors identify the special directions that do not rotate under this covariance transformation. They only get scaled. In a PCA plot, these directions are the main axes of the data cloud. That is why people call them principal axes.

Orthogonality Because the covariance matrix $S$ is symmetric

its eigenvectors can be chosen to be orthogonal. That is perfect for PCA, because the principal components are supposed to represent independent directions of variation in Euclidean geometric terms.

So the PCA directions are:

• orthogonal,
• ordered by variance,
• extracted from the covariance matrix.

Connection to projection

If we project the centered data onto the first eigenvector $v_1$, we get the first principal component scores. If the centered data matrix is $X_c$, then the score vector is:

That is the data expressed along the first principal axis. So the eigenvector equation gives the directions, and projection onto those directions gives the transformed lower-dimensional coordinates.

DA/DS relevance

In DA/DS work, this matters because PCA is widely used for:

• dimensionality reduction,
• visualization,
• denoising,
• collinearity compression,
• feature extraction,
• preprocessing before modeling.

Understanding the eigenvector equation makes PCA much less mysterious. It shows that PCA is really a structured search for directions of maximal variance inside the covariance geometry of the data.

Flowchart

Real-world use

Where it is used

The eigenvector equation is used whenever PCA is performed from the covariance matrix, including dimensionality reduction, exploratory data analysis, feature compression, and factor-style interpretation.

Problem it solves

It solves the problem of identifying the directions in feature space along which the data vary the most. Without this, dimensionality reduction would have no principled way to choose which directions to keep.

How it solves the problem

It finds the eigenvectors of the sample covariance matrix. These eigenvectors are the principal axes, and their eigenvalues quantify the variance along each axis. The analyst then keeps the leading directions with the largest eigenvalues.

Example in a business

Suppose a business analyst has many customer variables such as spend, purchase frequency, average basket size, return rate, and discount sensitivity. These variables are correlated. PCA solves the eigenvector equation of their covariance matrix and identifies a few dominant latent directions, such as “overall value” or “promotion sensitivity,” which can then be used in segmentation or dashboard simplification.

The formula, piece by piece

Canonical eigenvector equation:

Ordered eigenvalue convention in PCA:

Variance of projection onto direction $v$:

Unit-norm constraint for PCA direction:

Principal component score vector for direction $v_k$:

Explained variance ratio for component $k$:

The formula in words

Take the covariance matrix $S$ and a direction vector $v$. If multiplying $S$ by $v$ gives back the same direction scaled by some number $\lambda$, then $v$ is an eigenvector and $\lambda$ is its eigenvalue.

In PCA, that means:

• $v$ is a principal axis,
• $\lambda$ is the variance along that axis.

So the eigenvector equation identifies both the important directions and how important they are.

Critical Concepts

• Eigenvectors of the covariance matrix are principal directions. This is the core PCA fact.
• Eigenvalues measure explained variance. Bigger eigenvalues mean more important components.
• PCA orders components by decreasing eigenvalue. The first principal component explains the most variance.
• Covariance symmetry matters. Because $S$ is symmetric, PCA directions can be chosen orthogonal.
• Projection comes after direction selection. Solve for $v_k$, then compute scores as $X_c v_k$.
• PCA is a variance-maximization method. The eigenvector equation is the solution to that constrained optimization problem.
• Centered data are required. Without centering, the covariance-based interpretation breaks.

Step-by-step calculation (tiny example)

Suppose the sample covariance matrix is:

We want to solve:

Let:

Step 1, write the matrix equation

This gives:

So the coordinate equations are:

Step 2, find one eigenvalue and eigenvector Take

Then the equations become:

So one eigenvector is:

Step 3, find the second eigenvalue and eigenvector Take

Then:

So another eigenvector is:

Step 4, interpret The first principal axis is

with variance:

The second principal axis is:

with variance:

Tiny plain-language reading

The data vary twice as much in the first coordinate direction as in the second, so the first axis becomes the first principal component direction.

Code examples

Beginner friendly Example [Python])

import numpy as np # Eigenvectors and eigenvalues of a covariance matrix
S = np.array([ [2, 0], [0, 1]
], dtype=float) eigenvalues, eigenvectors = np.linalg.eigh(S) # sort from largest to smallest
order = np.argsort(eigenvalues)[::-1]
eigenvalues = eigenvalues[order]
eigenvectors = eigenvectors[:, order] print("Eigenvalues:")
print(eigenvalues)
print("Eigenvectors:")
print(eigenvectors)

Explanation :

• np.linalg.eigh is used because the covariance matrix is symmetric.
• The eigenvectors are the principal directions.
• The eigenvalues tell how much variance each direction explains.

Beginner friendly Example [R])

# Eigenvectors and eigenvalues of a covariance matrix
S <- matrix( c( 2, 0, 0, 1 ), nrow = 2, byrow = TRUE
) eigen_result <- eigen(S) cat("Eigenvalues:\n")
print(eigen_result$values)
cat("Eigenvectors:\n")
print(eigen_result$vectors)

Explanation :

• eigen(S) computes eigenvalues and eigenvectors.
• For PCA, the leading eigenvector is the first principal axis.

Final Professional Example [Python])

from __future__ import annotations from dataclasses import dataclass
from typing import Sequence import numpy as np RANDOM_SEED = 42 @dataclass(frozen=True)
class PcaCovarianceExample: data_matrix: Sequence[Sequence[float]] feature_names: Sequence[str] def validate(self) -> None: if len(self.data_matrix) < 2: raise ValueError("At least two observations are required.") if len(self.feature_names) == 0: raise ValueError("At least one feature name is required.") feature_count = len(self.feature_names) for row in self.data_matrix: if len(row) != feature_count: raise ValueError("Each row must match the number of feature names.") def center_columns(X: np.ndarray) -> np.ndarray: return X - np.mean(X, axis=0) def sample_covariance_matrix(X_centered: np.ndarray) -> np.ndarray: observation_count = X_centered.shape[0] return (X_centered.T @ X_centered) / (observation_count - 1) def summarize_pca_directions(example: PcaCovarianceExample) -> None: example.validate() X = np.array(example.data_matrix, dtype=float) X_centered = center_columns(X) S = sample_covariance_matrix(X_centered) eigenvalues, eigenvectors = np.linalg.eigh(S) order = np.argsort(eigenvalues)[::-1] eigenvalues = eigenvalues[order] eigenvectors = eigenvectors[:, order] explained_variance_ratio = eigenvalues / np.sum(eigenvalues) print("PCA eigenvector-equation summary") print("-" * 72) print("Feature names:") print(list(example.feature_names)) print("-" * 72) print("Sample covariance matrix S:") print(S) print("-" * 72) print("Eigenvalues:") print(eigenvalues) print("Explained variance ratios:") print(explained_variance_ratio) print("-" * 72) print("Eigenvectors, principal axes:") print(eigenvectors) print("-" * 72) print("First principal component scores:") print(X_centered @ eigenvectors[:, 0]) def main() -> None: example = PcaCovarianceExample( feature_names=["annual_spend", "purchase_frequency", "basket_size"], data_matrix=[ [1200.0, 12.0, 100.0], [1500.0, 15.0, 110.0], [1800.0, 18.0, 130.0], [1600.0, 14.0, 115.0], ], ) summarize_pca_directions(example) if __name__ == "__main__": main()

What this professional example teaches:

1. PCA starts from the covariance matrix of centered data.
2. Solving the eigenvector equation gives principal directions and explained variances.
3. Ordering eigenvalues from largest to smallest defines the PCA ranking.

Projecting centered data onto leading eigenvectors gives principal component scores.

Self-check: mathematical verification

Re-derivation from the variance-maximization problem PCA chooses a unit direction vector $v$ that maximizes projected variance

Use a Lagrange multiplier $\lambda$:

Differentiate with respect to $v$:

Set the gradient equal to zero:

Divide by 2:

This matches the canonical eigenvector equation exactly.

Eigenvalue-as-variance check If $v$ is a unit eigenvector satisfying

then projected variance is:

Substitute the eigenvector equation:

Because $|v|_2=1$, we have:

So:

This confirms that the eigenvalue equals the variance along that principal direction.

Orthogonality check The sample covariance matrix $S$ is symmetric

For symmetric matrices, eigenvectors corresponding to distinct eigenvalues are orthogonal. That matches the PCA requirement that principal axes be orthogonal.

Non-negativity check

A sample covariance matrix is positive semidefinite, so for any vector $a$:

Since eigenvalues are projected variances, they must satisfy:

This confirms the nonnegative variance interpretation.

Projection-score check Once a principal axis $v_k$ is found, projecting centered data onto it gives

This is dimensionally and geometrically correct, because it produces the coordinate of each observation along that axis.

Verbal-to-algebra consistency

The verbal explanation says eigenvectors are the principal directions and eigenvalues are the variance along those directions. The derivation confirms both claims exactly.

Structural verification

• Opening restatement paragraph is present.
• All required sections are present and non-empty.
• Block equations use `

`.

• Symbol-table entries use math mode such as $S$, $v$, $\lambda$, $X_c$, and $z_k$.
• Mermaid uses flowchart TD and node text in ["..."].
• Three code blocks are present: Python beginner, R beginner, Python professional.
• Links to sample covariance, projection, PCA, and factor analysis are explicitly included.
• No placeholder text remains.

Common Pitfalls and Edge Cases

• Skipping centering before PCA: Without centered data, the covariance interpretation of principal directions becomes distorted.
• Confusing eigenvectors with projected scores: Eigenvectors are directions; scores are data coordinates after projection.
• Ignoring eigenvalue ordering: PCA requires ordering from largest variance to smallest.
• Forgetting the unit-norm convention: Eigenvectors are usually normalized for consistent interpretation.
• Mixing covariance PCA with correlation PCA: The choice matters when features are on very different scales.
• Assuming all eigenvalues will be strictly positive: Some can be zero if the data are rank-deficient.
• Treating PCA directions as causal factors: They are variance directions, not causal explanations.
• Ignoring sign indeterminacy: If $v$ is an eigenvector, then $-v$ is also an eigenvector of the same component.

References and Standard Results

• Eigenvector equation, $Sv=\lambda v$
• Variance-maximization characterization of PCA
• Sample covariance matrix as the PCA input
• Eigenvalues as explained variances
• Eigenvectors as principal axes
• Orthogonality of eigenvectors for symmetric covariance matrices
• Score projection formula, $z_k=X_c v_k$
• Explained variance ratio, $\lambda_k / \sum_j \lambda_j$

#machine-learning#machine-learning-algorithms#dimensionality-reduction-principal-component-analysis-pca