Residual Error (Truncated Eigenvalues)

We are working on PCA residual error under the standard covariance-eigendecomposition setup, where the eigenvalues satisfy $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d \ge 0$, and we keep only the first $k$ principal components. The canonical residual variance is

which measures the variance not captured by the retained $k$-dimensional representation.

Deep Breakdown

Residual error in PCA, using truncated eigenvalues, is the amount of variance left unexplained after reducing a $d$-dimensional dataset to its first $k$ principal components. The inputs are the ordered PCA eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_d$, and the chosen number of retained components $k$.

The output is a single non-negative scalar:

This quantity matters because PCA is fundamentally a compression method. When we compress data from $d$ dimensions down to $k$ dimensions, we deliberately discard some directions of variation. The sum of the discarded eigenvalues tells us exactly how much total variance was lost. In DA/DS work, this is central when deciding how many components to keep. Too few components, and the compressed representation may miss important structure. Too many, and the reduction may not simplify the model enough. Residual error provides a direct tradeoff measure between compression and fidelity. This object is tightly linked to three core PCA ideas:

1. Reconstruction: lower residual variance usually means better reconstruction from the retained components.
2. Low-rank approximation: PCA gives the best rank-$k$ approximation in squared-error sense.
3. Explained variance ratio: if the retained variance is $\sum_{j=1}^{k}\lambda_j$, then the lost fraction is residual variance divided by total variance.

So, conceptually:

• total variance = what exists in the original centered data,
• retained variance = what the first $k$ PCs keep,
• residual variance = what the truncation throws away. ---

Flowchart

#machine-learning#machine-learning-algorithms#dimensionality-reduction-principal-component-analysis-pca