M-Step (Covariance Update)

We are working on the covariance update in the M-step of the EM algorithm for Gaussian Mixture Models. The canonical formula is

where $N_k = \sum_{i=1}^{n} r_{ik}$ is the effective component size and $r_{ik}$ are the responsibilities. This update gives the soft, responsibility-weighted covariance matrix for component $k$ under a standard full-covariance GMM.

Deep Breakdown

In a Gaussian Mixture Model, each component $k$ has its own mean vector $\mu_k$, covariance matrix $\Sigma_k$, and mixture weight $\pi_k$. During the EM algorithm, the E-step computes the responsibilities $r_{ik}$, which are soft assignments telling us how much observation $x_i$ belongs to component $k$. Once we have those soft assignments, the M-step updates the parameters of each Gaussian component. The covariance update is the part that determines the shape, spread, and orientation of each cluster.

The update is:

Inputs:

• observations $x_1, \dots, x_n$,
• responsibilities $r_{ik}$,
• updated mean $\mu_k^{\text{new}}$,
• effective component size $N_k$.

Output:

• updated covariance matrix $\Sigma_k^{\text{new}}$ for component $k$.

Why it matters in DA/DS:

• it determines cluster shape, not just cluster center,
• it lets one mixture component stretch more in one direction than another,
• it controls elliptical geometry in multivariate data,
• it affects responsibilities in the next E-step,
• it is essential for anomaly scoring, segmentation, and density estimation.

In a one-dimensional Gaussian mixture, covariance reduces to variance, which tells us how wide the component is.

In multiple dimensions, covariance does more:

• diagonal terms measure variance in each feature,
• off-diagonal terms measure how features move together inside the component,
• eigenstructure of $\Sigma_k$ determines the principal axes of the cluster ellipse or ellipsoid.

This update is the soft-clustering analogue of a sample covariance matrix. In an ordinary sample covariance, every point counts equally. In the GMM M-step, every point is weighted by its responsibility for component $k$. So points that strongly belong to component $k$ contribute more, and points that weakly belong contribute less.

This object links directly to:

• Effective Component Size, because $N_k$ normalizes the weighted sum,
• M-Step (Means Update), because the covariance is centered around $\mu_k^{\text{new}}$,
• M-Step (Weights Update), because the same responsibilities govern all M-step parameter updates,
• cluster geometry, because the covariance matrix defines each Gaussian component’s shape.

Flowchart

#machine-learning#machine-learning-algorithms#expectation-maximisation-em-for-gmms