Posterior Predictive Distribution · Rafael Braga-Kribitz

Contents

01Deep Breakdown
02Flowchart

p(y_{\text{new}} \mid x_{\text{new}}, D) = \mathcal{N}\left(x_{\text{new}}^\top m_N,\; \sigma^2 + x_{\text{new}}^\top S_N x_{\text{new}}\right)

(1)

We are working on the posterior predictive distribution in Bayesian Linear Regression, with canonical formula $p(y_{\text{new}} \mid x_{\text{new}}, D) = \mathcal{N}\left(x_{\text{new}}^\top m_N,\; \sigma^2 + x_{\text{new}}^\top S_N x_{\text{new}}\right)$ , where $m_N$ is the posterior mean of the regression weights, $S_N$ is the posterior covariance, and $\sigma^2$ is the observation noise variance. The domain is Gaussian Bayesian linear regression with known noise variance, Gaussian prior, Gaussian posterior over weights, and a new feature vector $x_{\text{new}}$ . This predictive distribution includes both irreducible observation noise and parameter uncertainty.

Deep Breakdown

In Bayesian Linear Regression, we do not produce only a single predicted number for a new input. Instead, we produce a full probability distribution for the future response. That distribution is called the posterior predictive distribution. Suppose we have already observed a training dataset:

D = \{X, y\}

(2)

and from this dataset we obtained the posterior over regression weights:

w \mid D \sim \mathcal{N}(m_N, S_N)

(3)

Now for a new feature vector $x_{\text{new}}$ , the data-generating model says:

y_{\text{new}} = x_{\text{new}}^\top w + \varepsilon_{\text{new}}, \qquad \varepsilon_{\text{new}} \sim \mathcal{N}(0, \sigma^2)

(4)

Because $w$ is uncertain and the new observation has noise, the predicted response is also uncertain. The predictive distribution is found by integrating over the posterior uncertainty in $w$ :

p(y_{\text{new}} \mid x_{\text{new}}, D) = \int p(y_{\text{new}} \mid x_{\text{new}}, w)\, p(w \mid D)\, dw

(5)

In Gaussian Bayesian linear regression, this integral has a closed form:

p(y_{\text{new}} \mid x_{\text{new}}, D) = \mathcal{N}\left( x_{\text{new}}^\top m_N,\; \sigma^2 + x_{\text{new}}^\top S_N x_{\text{new}} \right)

(6)

This formula has two parts:

the predictive mean $x_{\text{new}}^\top m_N$ , which is the average predicted value,
the predictive variance $\sigma^2 + x_{\text{new}}^\top S_N x_{\text{new}}$ , which combines noise and parameter uncertainty.

Why this matters in DA/DS:

Forecast intervals: gives uncertainty bands around predictions, not only point estimates.
Decision quality: helps distinguish “high expected value, low confidence” from “high expected value, high confidence.”
Risk-aware modeling: important in finance, marketing, operations, supply chain, and demand forecasting.
Out-of-sample caution: predictions far from the training region often have larger uncertainty.
Transparent uncertainty decomposition: one part is data noise, another part is uncertainty about coefficients.

A useful mental model is this:

if you knew the regression weights exactly, uncertainty would come only from $\sigma^2$ ,
but because you do not know the weights exactly, there is an extra uncertainty term $x_{\text{new}}^\top S_N x_{\text{new}}$ .

So Bayesian prediction is not just “a line with noise.” It is “a line with uncertainty about the line itself.”

Flowchart

#machine-learning #machine-learning-algorithms #bayesian-linear-regression-blr