Absolute homogeneity

Deep Breakdown

What it tells me (Situation Awareness): Absolute homogeneity is one of the three defining axioms of a norm: $\|\lambda x\| = |\lambda| \|x\|$. It says that scaling a vector by a scalar $\lambda$ scales its norm by $|\lambda|$ — the absolute value, not $\lambda$ itself. This is the property that ensures a norm is consistent with scalar multiplication. If you double a vector, its length doubles ($\lambda = 2$). If you negate a vector ($\lambda = -1$), its length is unchanged (because $|-1| = 1$). This axiom distinguishes a norm from an arbitrary function on vectors.

Flowchart

Real-world use

Where it is used

Verifying that a function qualifies as a valid norm (must satisfy all three axioms: positive definiteness, absolute homogeneity, triangle inequality). This comes up when: defining custom distance measures for ML, checking that a regularization penalty behaves correctly, understanding why $\|\alpha \nabla L\| = \alpha \|\nabla L\|$ when $\alpha > 0$ (gradient norm scales linearly with learning rate), and in theoretical analysis of optimization algorithms.

Problem it solves

Without homogeneity, a "norm" could behave erratically under scaling: doubling a vector might triple its "length," or reversing a vector might change its "length." Absolute homogeneity enforces that scaling and norms interact consistently: stretching a vector stretches its norm proportionally, and direction (sign) does not affect length.

How it solves the problem

For the $L_2$ norm: $\|\lambda x\|_2 = \sqrt{\sum_i (\lambda x_i)^2} = \sqrt{\lambda^2 \sum_i x_i^2} = |\lambda| \sqrt{\sum_i x_i^2} = |\lambda| \|x\|_2$. The $\lambda^2$ under the square root becomes $|\lambda|$ after the root, giving the absolute value. The same derivation works for any $L_p$ norm.

Example in a business

A DS applies gradient descent with learning rate $\alpha = 0.01$. The gradient vector $\nabla L$ has norm $\|\nabla L\|_2 = 1.5$. The update step $\alpha \nabla L$ has norm $\|\alpha \nabla L\|_2 = |\alpha| \|\nabla L\|_2 = 0.01 \times 1.5 = 0.015$ — the step size is exactly $\alpha$ times the gradient magnitude. This is absolute homogeneity: the norm of the scaled vector is $\alpha$ times the norm of the original.

The formula, piece by piece

• $\|\cdot\|$: any valid norm (must satisfy all three norm axioms)
• $\lambda$: any scalar (real or complex)
• $\lambda x$: the vector $x$ scaled by $\lambda$; every component multiplied by $\lambda$
• $|\lambda|$: the absolute value of $\lambda$; ensures the right-hand side is non-negative
• $\|x\|$: the norm of the original unscaled vector
• The property: $\|\lambda x\| = |\lambda| \cdot \|x\|$

The formula in words

"Scaling a vector by a factor $\lambda$ scales its norm by $|\lambda|$. The absolute value ensures that reversing direction ($\lambda < 0$) does not change the length." Tiny examples with $L_2$ norm:

• $x = (3,4)$, $\|x\|_2 = 5$. $\lambda = 2$: $\|2x\|_2 = \|(6,8)\|_2 = 10 = 2 \cdot 5$. $\checkmark$
• $\lambda = -3$: $\|{-3x}\|_2 = \|(-9,-12)\|_2 = \sqrt{81+144} = 15 = |-3| \cdot 5$. $\checkmark$

Critical Concepts

• All $L_p$ norms satisfy this: $\|\lambda x\|_p = |\lambda| \|x\|_p$ for all $p \geq 1$.
• Part of the three norm axioms: a function $\|\cdot\|$ is a norm if and only if it satisfies: (1) positive definiteness, (2) absolute homogeneity, (3) triangle inequality.
• Consequence for unit vectors: $\hat{x} = x/\|x\|$ has $\|\hat{x}\| = \|x/\|x\|\| = (1/\|x\|)\|x\| = 1$. Normalization works because of absolute homogeneity.
• Consequences for optimization: the gradient of $\|x\|_2^2 = x^T x$ is $2x$ (linear in $x$). This is because $(||x||_2^2)$ itself is quadratic, and homogeneity ensures it behaves predictably.
• Not a norm axiom violation: $\|{-x}\| = \|x\|$ is a consequence of absolute homogeneity with $\lambda = -1$.

Reversing a vector does not change its length.

Step-by-step calculation (tiny example)

Step 1: Choose a vector, compute its norm $x = (1, -2, 2)$. $\|x\|_2 = \sqrt{1+4+4} = 3$.

Step 2: Scale by a negative scalar $\lambda = -4$. $\lambda x = (-4, 8, -8)$.

Step 3: Verify the property $\|\lambda x\|_2 = \sqrt{16+64+64} = \sqrt{144} = 12$. $|\lambda| \|x\|_2 = |-4| \cdot 3 = 4 \cdot 3 = 12$. $\checkmark$

Code examples

Beginner example (Python)

import numpy as np

# -- Absolute homogeneity: verify for L2, L1, Linf norms ---
x = np.array([1.0, -2.0, 2.0])
for lam in [2.0, -3.0, 0.5, -1.0, 0.0]:
    scaled_x = lam * x
    for p, name in [(2, "L2"), (1, "L1"), (np.inf, "Linf")]:
        lhs = np.linalg.norm(scaled_x, ord=p)
        rhs = abs(lam) * np.linalg.norm(x, ord=p)
        ok = np.isclose(lhs, rhs)
        print(f"lambda={lam:4g} {name}: ||lam*x|| = {lhs:.4f}, |lam|*||x|| = {rhs:.4f}, match={ok}")
    print()

Beginner example (R)

# -- Absolute homogeneity for L2 norm ---
x <- c(1, -2, 2)
for (lam in c(2, -3, 0.5, -1, 0)) {
  scaled_x <- lam * x
  lhs <- sqrt(sum(scaled_x^2))
  rhs <- abs(lam) * sqrt(sum(x^2))
  cat(sprintf("lambda=%4g : ||lam*x|| = %.4f, |lam|*||x|| = %.4f, match=%s\n", lam, lhs, rhs, isTRUE(all.equal(lhs, rhs))))
}

Professional example (Python)

import numpy as np
import pandas as pd
np.random.seed(42)

# -- Business scenario: gradient norm vs. learning rate ---
# Demonstrates absolute homogeneity: ||alpha * grad||_2 = alpha * ||grad||_2
# Shows how step size scales exactly linearly with learning rate.
N_PARAMS = 10
n_trials = 8

# Simulate gradient vectors of varying magnitudes
gradients = [np.random.randn(N_PARAMS) * scale for scale in np.linspace(0.1, 5.0, n_trials)]
learning_rates = [0.001, 0.01, 0.1, 1.0]
results = []
for lr in learning_rates:
    for i, grad in enumerate(gradients):
        step = lr * grad   # alpha * gradient
        step_norm = np.linalg.norm(step)   # ||alpha * grad||
        predicted = abs(lr) * np.linalg.norm(grad)   # |alpha| * ||grad||
        results.append({
            "learning_rate": lr,
            "grad_trial": i,
            "grad_norm": round(np.linalg.norm(grad), 4),
            "step_norm": round(step_norm, 4),
            "predicted_norm": round(predicted, 4),
            "matches": np.isclose(step_norm, predicted),
        })
df = pd.DataFrame(results)
print("All absolute homogeneity checks pass:")
print(f" {df['matches'].all()}")
print(f"\nSample rows (lr=0.01):")
print(df[df["learning_rate"] == 0.01][["grad_norm", "step_norm", "predicted_norm", "matches"]].head(5).to_string(index=False))
print("\nConclusion: step_norm = lr * grad_norm always. Step size is exactly proportional to lr.")

Source: MML1 p. 76

#linear-algebra#spatial-distance-norms-and-analytic-geometry#distance-metrics-and-norms