Independence Condition

Deep Breakdown

What it tells me (Situation Awareness): Two random variables X and Y are statistically independent if and only if their joint distribution factors into the product of the marginals for all (x, y). This is the mathematical definition of independence: knowing the value of X provides no information about Y. In DA/DS this is a fundamental assumption in Naive Bayes, factored probabilistic graphical models, many ML regularisation arguments, and the justification for using chi-square tests to detect dependence.

Flowchart

Real-world use

Where it is used

Testing whether two features are independent before applying Naive Bayes, validating independence assumptions in a causal model, or checking that residuals from a regression are uncorrelated with covariates.

Problem it solves

If X and Y are independent, you can model them separately and multiply their distributions together. This drastically reduces the number of parameters needed. If they are not independent, you must use the full joint distribution.

How it solves the problem

You check whether the joint PMF/PDF factorises. For discrete data, compare every cell of the joint table with the product of the corresponding row and column marginals. Any deviation implies dependence.

Example in a business

A data scientist applies Naive Bayes for email spam classification. The key assumption is that each word feature is conditionally independent of every other word given the class label. The independence condition justifies multiplying individual word likelihoods: $P(\text{spam}|x_1,...,x_n) \propto P(\text{spam}) \prod_i P(x_i|\text{spam})$.

The formula, piece by piece

Full formula: $X \perp Y \iff f(x,y) = f_X(x) \cdot f_Y(y)$ for all $(x,y)$

The formula in words

"X and Y are independent if and only if the joint density or mass function factors into the product of the two marginals at every point." One failing pair is sufficient to establish dependence.

Critical Concepts

• Necessary and sufficient: factorisation must hold everywhere, not just on average.
• Zero covariance $\ne$ independence: uncorrelated Gaussians are independent, but uncorrelated non-Gaussian variables may still be dependent.
• Conditional independence: $X \perp Y \mid Z$ means factorisation holds given Z; different from marginal independence.
• Chi-square test: empirically tests independence in contingency tables; $H_0$: independence.
• Naive Bayes: explicitly assumes feature conditional independence to make inference tractable.

Step-by-step calculation (tiny example)

Joint PMF for (Device: 0=mobile, 1=desktop) × (Purchase: 0=no, 1=yes):

Step 1: Check independence for (X=0, Y=0) $f_X(0) \cdot f_Y(0) = 0.50 \times 0.70 = 0.35 = f(0,0)$

Step 2: Check for (X=0, Y=1) $0.50 \times 0.30 = 0.15 = f(0,1)$

Step 3: Check all remaining cells Both hold. Conclusion: X and Y are independent.

Step 4: Contrast with dependent case If $f(0,0) = 0.40 \ne 0.50 \times 0.70 = 0.35$, independence fails.

Code examples

Beginner example (Python)

import numpy as np

# Joint PMF: independent case
joint_pmf = np.array([
    [0.35, 0.15],
    [0.35, 0.15],
])
marginal_x = joint_pmf.sum(axis=1)   # row sums
marginal_y = joint_pmf.sum(axis=0)   # col sums

# Expected under independence: outer product of marginals
expected_if_independent = np.outer(marginal_x, marginal_y)
print("Joint PMF:"); print(joint_pmf)
print("\nExpected under independence:"); print(expected_if_independent.round(4))
is_independent = np.allclose(joint_pmf, expected_if_independent, atol=1e-6)
print(f"\nAre X and Y independent? {is_independent}")

Beginner example (R)

joint_pmf <- matrix(c(0.35, 0.15, 0.35, 0.15), nrow = 2, byrow = TRUE)
marginal_x <- rowSums(joint_pmf)
marginal_y <- colSums(joint_pmf)
expected_independent <- outer(marginal_x, marginal_y)
is_independent <- all(abs(joint_pmf - expected_independent) < 1e-6)
cat("Are X and Y independent?", is_independent, "\n")

Professional example (Python)

import numpy as np
import pandas as pd
from scipy.stats import chi2_contingency


def test_independence(
    df: pd.DataFrame, col_x: str, col_y: str, alpha: float = 0.05
) -> dict[str, float | bool | str]:
    # Chi-square test for independence between two categorical variables.
    # Returns test statistic, p-value, degrees of freedom, and interpretation.
    contingency_table = pd.crosstab(df[col_x], df[col_y])
    chi2_stat, p_value, dof, expected = chi2_contingency(contingency_table)
    max_deviation = np.max(np.abs(contingency_table.values - expected) / expected)
    return {
        "chi2_statistic": round(float(chi2_stat), 4),
        "p_value": round(float(p_value), 6),
        "degrees_of_freedom": int(dof),
        "reject_independence": p_value < alpha,
        "max_relative_deviation": round(float(max_deviation), 4),
        "conclusion": (
            f"DEPENDENT (p={p_value:.4f} < {alpha})"
            if p_value < alpha
            else f"Cannot reject independence (p={p_value:.4f} >= {alpha})"
        ),
    }


if __name__ == "__main__":
    rng = np.random.default_rng(seed=42)
    n = 5_000
    device_choices = rng.choice(["mobile", "desktop"], size=n, p=[0.60, 0.40])
    # Desktop users purchase more: dependence induced
    purchase_probs = np.where(device_choices == "desktop", 0.45, 0.20)
    purchase_choices = (rng.random(n) < purchase_probs).astype(str)
    purchase_choices = np.where(purchase_choices == "True", "yes", "no")
    sample_df = pd.DataFrame({"device": device_choices, "purchase": purchase_choices})
    result = test_independence(sample_df, col_x="device", col_y="purchase")
    for key, val in result.items():
        print(f" {key}: {val}")

Source: AIO p. 86`

#probability-statistics#multivariable-statistics-and-information#joint-and-marginal-distributions