Scalar Multiplication

Deep Breakdown

What it tells me (Situation Awareness): The scalar multiplication rule states that a constant factor $k$ passes through the derivative operator: $d/dx[k f(x)] = k f'(x)$. Differentiation is a linear operator, and this is one half of that linearity (the other half being the sum rule). In DA/DS: this rule appears everywhere — the learning rate $\eta$ in gradient descent $w \leftarrow w - \eta \nabla L$ scales the gradient (the $1/N$ factor in MSE loss is a constant that scales the gradient; the $2\lambda$ factor in ridge regularization gradient comes from $\lambda$ passing through the derivative); loss weighting in multi-task learning ($L = \alpha L_1 + \beta L_2$, $\nabla L = \alpha \nabla L_1 + \beta \nabla L_2$); and any normalization or scaling applied to a loss function.

Flowchart

Real-world use

Where it is used

Scaling gradients in MSE: $d/dw[(1/N)\|y-Xw\|^2] = (1/N) \cdot d/dw[\|y-Xw\|^2]$; ridge regularization gradient: $d/dw[\lambda\|w\|^2] = \lambda \cdot d/dw[\|w\|^2] = 2\lambda w$; multi-task loss gradients: $\nabla(\alpha L_A + \beta L_B) = \alpha \nabla L_A + \beta \nabla L_B$; learning rate as a scaling factor of the gradient in optimizer updates; normalizing loss by batch size.

Problem it solves

When a function has a constant overall scaling factor, the derivative should not require re-deriving the underlying function. The scalar rule makes derivations cleaner by allowing constants to be factored out, computed separately, and multiplied back in.

How it solves the problem

From first principles: $[k f(x+h) - k f(x)]/h = k [f(x+h) - f(x)]/h$. The constant $k$ factors out of the difference quotient. Taking the limit: $\lim_{h\to 0} k [f(x+h)-f(x)]/h = k \cdot \lim_{h\to 0} [f(x+h)-f(x)]/h = k f'(x)$.

Example in a business

A DS defines the MSE loss as $L = (1/N)\sum_i(y_i - \hat{y}_i)^2$. By the scalar rule: $\nabla_w L = (1/N) \cdot \nabla_w \sum_i(y_i - \hat{y}_i)^2$. The $1/N$ factor is a constant that passes through the derivative. This keeps the gradient magnitude independent of batch size $N$, making learning rates more transferable across different batch sizes.

The formula, piece by piece

• $k$: any real constant (not depending on $x$)
• $f(x)$: any differentiable function
• $d/dx[k f(x)] = k f'(x)$: constant factors out of the derivative
• Proof: $k$ factors out of the limit because limit is linear
• Combined with sum rule: $d/dx[\sum_i k_i f_i(x)] = \sum_i k_i f_i'(x)$
• Practical note: $k = -1$ gives $d/dx[-f(x)] = -f'(x)$; negation distributes through derivatives
• Special case $k = 0$: $d/dx[0 \cdot f(x)] = 0$ (consistent with constant rule)

The formula in words

"If a function is scaled by a constant, its derivative is the same constant times the original derivative. Constants factor out of derivatives just as they factor out of sums and integrals. Scaling a function vertically scales its slope by the same factor." Tiny examples: $d/dx(5x^2) = 5 \cdot 2x = 10x$. $d/dx(0.01 \cdot e^x) = 0.01 \cdot e^x$. $d/dx[(1/N) \cdot x^2] = (1/N) \cdot 2x = 2x/N$. $d/dx(-3\sin x) = -3\cos x$. $d/dx(2\lambda w^2) = 2\lambda \cdot 2w = 4\lambda w$ (ridge gradient).

Critical Concepts

• Linearity of differentiation: the derivative operator $d/dx$ is linear: $d/dx[af + bg] = af' + bg'$. This combines scalar multiplication and the sum rule. It means that gradient computation is completely compositional — you can always split, scale, and recombine.
• Learning rate is a scalar applied to the gradient: in $w \leftarrow w - \eta g$, the update is $-\eta$ times the gradient. When people say "the learning rate scales the gradient," they mean the scalar multiplication rule: the step in parameter space is $\eta$ times the gradient magnitude.
• Batch size normalization: if loss is $L = \sum_i \ell_i$ (sum over batch) vs $L = (1/N)\sum_i \ell_i$ (mean), the gradients differ by $1/N$. Using mean loss makes the gradient magnitude independent of batch size, allowing the same learning rate to work across batch sizes. This is the scalar rule in practice.
• Gradient scaling in multi-task learning: if the total loss is $L = \alpha L_A + \beta L_B$, each task's gradient contribution is scaled by its coefficient: $\nabla L = \alpha \nabla L_A + \beta \nabla L_B$. Choosing $\alpha$ and $\beta$ effectively controls the relative step sizes in each task's gradient direction.
• Negative gradient (gradient descent): $-\nabla L$ is just $(-1) \cdot \nabla L$.

Gradient descent updates $w \leftarrow w - \eta \nabla L$ because we move in the direction of the negative gradient to decrease the loss. The minus sign is the scalar $k = -1$ case.

Step-by-step calculation (tiny example)

Find the gradient of $L = (1/3)(r_1^2 + r_2^2 + r_3^2)$ where $r_i = y_i - wx_i$, w.r.t. $w$. Use $x=[1,2,3]$, $y=[2,3,4]$, $w=1$.

Step 1: Factor out the constant $1/3$. $dL/dw = (1/3) \cdot d/dw(r_1^2 + r_2^2 + r_3^2)$ (scalar rule).

Step 2: Sum rule + power rule on each term. $d/dw(r_i^2) = 2r_i \cdot (-x_i)$ where $r_i = y_i - wx_i$.

Step 3: Residuals at $w=1$: all $r_i = 1$. $d/dw(r_1^2 + r_2^2 + r_3^2) = -2[r_1 x_1 + r_2 x_2 + r_3 x_3] = -2[1+2+3] = -12$.

Step 4: Apply scalar factor. $dL/dw = (1/3)(-12) = -4$.

Code examples

Beginner example (Python)

import numpy as np

def scaled_deriv(k: float, f_prime_x: float) -> float:
    """d/dx[k * f(x)] = k * f'(x)."""
    return k * f_prime_x


# Verify for MSE: d/dw[(1/N) * ||y - wx||^2]
x_vals = np.array([1.0, 2.0, 3.0])
y_vals = np.array([2.0, 3.0, 4.0])
w = 1.0
N = len(x_vals)


def mse(w: float) -> float:
    return float((1 / N) * np.sum((y_vals - w * x_vals)**2))


# Analytic gradient: (1/N) * (-2) * sum(r_i * x_i)
residuals = y_vals - w * x_vals
grad_analytic = (1 / N) * (-2) * np.dot(residuals, x_vals)

# Numerical
h = 1e-7
grad_numerical = (mse(w + h) - mse(w - h)) / (2 * h)
print(f"MSE gradient: analytic = {grad_analytic:.6f}, numerical = {grad_numerical:.6f}")
print(f"Scalar rule: (1/N) factors out of d/dw[||y - wx||^2]")

# Show that scaling the loss scales the gradient proportionally
print("\nEffect of scalar k on gradient:")
print(f"{'k':>8} {'Gradient':>14}")
for k in [1.0, 1/N, 2.0, 0.5, -1.0]:
    # d/dw[k * mse_unnormalized] = k * grad_unnormalized
    grad_unnormalized = (-2) * np.dot(residuals, x_vals)
    scaled_grad = k * grad_unnormalized
    print(f"{k:>8.4f} {scaled_grad:>14.6f}")

Beginner example (R)

# d/dx[k * f(x)] = k * f'(x)
x_vals <- c(1, 2, 3)
y_vals <- c(2, 3, 4)
w <- 1.0
N <- length(x_vals)
mse <- function(w) (1/N) * sum((y_vals - w * x_vals)^2)
residuals <- y_vals - w * x_vals

# Analytic: (1/N) passes through derivative
grad_analytic <- (1/N) * (-2) * sum(residuals * x_vals)
h <- 1e-7
grad_numerical <- (mse(w + h) - mse(w - h)) / (2 * h)
cat(sprintf("MSE gradient: analytic = %.6f, numerical = %.6f\n", grad_analytic, grad_numerical))
cat("Scalar (1/N) factors out of derivative.\n")

# Multi-task loss: grad(alpha*L_A + beta*L_B) = alpha*grad_A + beta*grad_B
alpha <- 0.7; beta <- 0.3
grad_A <- 2.0  # hypothetical
grad_B <- 5.0  # hypothetical
grad_combined <- alpha * grad_A + beta * grad_B
cat(sprintf("\nMulti-task: alpha*grad_A + beta*grad_B = %.1f*%.1f + %.1f*%.1f = %.2f\n", alpha, grad_A, beta, grad_B, grad_combined))

Professional example (Python)

import numpy as np
from sklearn.datasets import make_regression
np.random.seed(42)

# -- Scalar rule in multi-task learning: gradient scaling by task weight --
# Use case: a DS trains a model with two losses (regression + sparsity),
# weighted by alpha and beta. The gradient is a weighted sum of individual gradients.
N, D = 300, 6
X, y_reg = make_regression(n_samples=N, n_features=D, noise=3.0, random_state=42)
y_reg = y_reg / y_reg.std()

# Two tasks: MSE regression loss and L2 sparsity regularization
ALPHA = 0.8   # regression weight
BETA = 0.2   # sparsity weight
LAMBDA = 0.01   # L2 regularization intensity

def mse_loss(w: np.ndarray, X: np.ndarray, y: np.ndarray) -> float:
    """L_A = (1/N) * ||y - Xw||^2."""
    return float(np.mean((y - X @ w)**2))

def mse_gradient(w: np.ndarray, X: np.ndarray, y: np.ndarray) -> np.ndarray:
    """grad L_A = (-2/N) * X^T (y - Xw). Scalar 1/N factors out."""
    residuals = y - X @ w
    return -2.0 / N * X.T @ residuals

def l2_penalty(w: np.ndarray, lam: float) -> float:
    """L_B = lambda * ||w||^2."""
    return float(lam * np.dot(w, w))

def l2_gradient(w: np.ndarray, lam: float) -> np.ndarray:
    """grad L_B = 2 * lambda * w. Scalar 2*lambda factors out."""
    return 2.0 * lam * w

def total_loss_and_gradient(
    w: np.ndarray, X: np.ndarray, y: np.ndarray, alpha: float, beta: float, lam: float
) -> tuple:
    """
    L = alpha * L_A + beta * L_B.
    Scalar rule: grad L = alpha * grad_A + beta * grad_B.
    """
    loss_a = mse_loss(w, X, y)
    loss_b = l2_penalty(w, lam)
    grad_a = mse_gradient(w, X, y)
    grad_b = l2_gradient(w, lam)
    total_loss = alpha * loss_a + beta * loss_b
    total_grad = alpha * grad_a + beta * grad_b   # scalar rule
    return float(total_loss), total_grad


# Gradient descent
w = np.zeros(D)
LR = 0.05
N_ITER = 300
for _ in range(N_ITER):
    _, grad = total_loss_and_gradient(w, X, y_reg, ALPHA, BETA, LAMBDA)
    w -= LR * grad
total_loss_final, grad_final = total_loss_and_gradient(w, X, y_reg, ALPHA, BETA, LAMBDA)
grad_a_final = mse_gradient(w, X, y_reg)
grad_b_final = l2_gradient(w, LAMBDA)
print("Scalar Rule: gradient of weighted multi-task loss")
print(f"\nalpha={ALPHA}, beta={BETA}, lambda={LAMBDA}")
print(f"L = alpha * MSE + beta * L2_penalty")
print(f"grad L = alpha * grad_MSE + beta * grad_L2 (scalar rule)")
print(f"\nFinal weights: {w.round(4)}")
print(f"Final total loss: {total_loss_final:.6f}")
print(f"\nGradient decomposition at final w:")
print(f"{'Param':>8} {'alpha*grad_A':>14} {'beta*grad_B':>14} {'total grad':>14}")
for j in range(D):
    print(f"{'w['+str(j)+']':>8} {ALPHA*grad_a_final[j]:>14.6f} {BETA*grad_b_final[j]:>14.6f} {grad_final[j]:>14.6f}")
print(f"\nGradient norm at convergence: {np.linalg.norm(grad_final):.6f}")

Source: 1300 p. 218

#calculus#differential-calculus#definition-and-properties-of-the-derivative