05 · Statistics for machine learning and optimisation¶
CSE273 Unit V. Sampling and populations, hypothesis testing, confidence intervals and correlation; maximum likelihood estimation and loss functions; convex and non-convex problems, minima and saddle points; gradient descent, the learning rate and its variants; and the three optimisers every deep learning library ships: Momentum, RMSProp and Adam.
The optimisers are not imported from a library. They are the ~60 lines in src/groundwork/optim.py, tested in tests/test_optim.py, and used here to train a softmax classifier on the irrigation data.
try:
import groundwork
except ImportError: # running on Kaggle
%pip install -q git+https://github.com/LuShadowX/Groundwork
import groundwork
import time
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats, optimize
import conviction as cn
from groundwork import data, plotting, optim
from groundwork.stats import bootstrap_ci, mle_normal, normal_nll, mse, mae, huber, log_loss, hinge
plotting.setup()
df = data.load()
rng = np.random.default_rng(5)
1. Statistical foundations¶
1.1 Sampling and population¶
Here the 630,000 training fields are treated as the population, so the true answer is known and every sampling idea can be checked. The population mean soil moisture is a fixed number (a parameter); the mean of a random sample is a statistic that changes from sample to sample. Its spread is the standard error, σ/√n.
pop = df["Soil_Moisture"].to_numpy()
mu, sigma = pop.mean(), pop.std()
print(f"population mean = {mu:.3f}, sd = {sigma:.3f}")
fig, ax = plt.subplots()
rows = []
for n, color in zip([5, 30, 200], ["#e9c46a", "#2a9d8f", plotting.ACCENT]):
means = np.array([rng.choice(pop, n).mean() for _ in range(5000)])
ax.hist(means, bins=60, density=True, alpha=0.55, color=color, label=f"n = {n}")
rows.append({"n": n, "sd of sample means": means.std(), "σ/√n": sigma / np.sqrt(n)})
ax.axvline(mu, color="black", ls="--", lw=1)
ax.set(xlabel="sample mean of soil moisture", ylabel="density", title="Sample means pile up around the population mean")
ax.legend()
plotting.save("g05-sampling")
plt.show()
pd.DataFrame(rows).round(3)
population mean = 37.304, sd = 16.377
| n | sd of sample means | σ/√n | |
|---|---|---|---|
| 0 | 5 | 7.380 | 7.324 |
| 1 | 30 | 3.024 | 2.990 |
| 2 | 200 | 1.154 | 1.158 |
Soil moisture itself is flat (uniform-like), but the sample means are bell-shaped and narrower for bigger n: the central limit theorem, measured. The standard error matches σ/√n.
1.2 Hypothesis testing¶
Question: do High-need fields have a higher temperature than Low-need ones?
- H₀: the mean temperature is the same; H₁: it differs.
- Welch's t-test (no equal-variance assumption) on a sample of each group.
- Reject H₀ if p < α = 0.05.
sample = data.sample(4000, seed=5)
hi_t = sample.loc[sample[data.TARGET] == "High", "Temperature_C"]
lo_t = sample.loc[sample[data.TARGET] == "Low", "Temperature_C"]
t_stat, p = stats.ttest_ind(hi_t, lo_t, equal_var=False)
print(f"High: n={len(hi_t)}, mean={hi_t.mean():.2f} | Low: n={len(lo_t)}, mean={lo_t.mean():.2f}")
print(f"t = {t_stat:.2f}, p = {p:.2e} -> {'reject' if p < 0.05 else 'keep'} H0")
# a categorical question needs a different test: is mulching independent of need?
table = pd.crosstab(sample["Mulching_Used"], sample[data.TARGET])
chi2, p_chi, dof, _ = stats.chi2_contingency(table)
print(f"chi-square = {chi2:.1f}, dof = {dof}, p = {p_chi:.2e}")
# and a question where H0 is true: is soil pH different between the classes?
t2, p2 = stats.ttest_ind(sample.loc[sample[data.TARGET] == "High", "Soil_pH"], sample.loc[sample[data.TARGET] == "Low", "Soil_pH"], equal_var=False)
print(f"soil pH: t = {t2:.2f}, p = {p2:.3f} -> {'reject' if p2 < 0.05 else 'keep'} H0")
High: n=133, mean=34.58 | Low: n=2349, mean=25.42 t = 17.92, p = 5.58e-41 -> reject H0 chi-square = 354.8, dof = 2, p = 9.02e-78 soil pH: t = 0.01, p = 0.994 -> keep H0
A permutation test makes the logic concrete without any distribution formula: if H₀ is true the labels are exchangeable, so shuffle them many times and see how often a difference as large as the real one appears by chance.
both = np.concatenate([hi_t, lo_t])
observed = hi_t.mean() - lo_t.mean()
perm = np.empty(5000)
for i in range(len(perm)):
shuffled = rng.permutation(both)
perm[i] = shuffled[:len(hi_t)].mean() - shuffled[len(hi_t):].mean()
fig, ax = plt.subplots()
ax.hist(perm, bins=60, color="#cbd5e1", label="differences under H₀ (shuffled labels)")
ax.axvline(observed, color="#e76f51", lw=2, label=f"observed difference = {observed:.2f} °C")
ax.set(xlabel="mean temperature, High − Low", title="Permutation test")
ax.legend()
plotting.save("g05-permutation")
plt.show()
print("permutation p-value:", (np.abs(perm) >= abs(observed)).mean())
permutation p-value: 0.0
Not one of 5,000 shuffles comes close. And α = 0.05 really does mean "5% false alarms": testing two random halves of the same group (H₀ true by construction) many times rejects about 5% of the time.
lows = df.loc[df[data.TARGET] == "Low", "Temperature_C"].to_numpy()
rejections = np.mean([stats.ttest_ind(*rng.choice(lows, (2, 200)), equal_var=False).pvalue < 0.05 for _ in range(2000)])
print(f"type I error rate over 2000 A/A tests: {rejections:.3f} (expected 0.05)")
type I error rate over 2000 A/A tests: 0.053 (expected 0.05)
1.3 Confidence intervals¶
A 95% CI is a range built so that, over repeated samples, 95% of such ranges contain the true value. Three ways for the mean temperature of High fields: the t-formula, the bootstrap from groundwork.stats, and conviction, my uncertainty library, which carries the bootstrap distribution around as a value.
true_high = df.loc[df[data.TARGET] == "High", "Temperature_C"].mean()
m, se = hi_t.mean(), hi_t.std(ddof=1) / np.sqrt(len(hi_t))
t_ci = stats.t.interval(0.95, len(hi_t) - 1, m, se)
_, b_lo, b_hi = bootstrap_ci(hi_t.to_numpy(), n_boot=4000, seed=0)
boot_means = [rng.choice(hi_t.to_numpy(), len(hi_t)).mean() for _ in range(4000)]
belief = cn.empirical(boot_means) # an Uncertain value
c_lo, c_hi = belief.interval(0.95)
pd.DataFrame({
"method": ["t-interval", "bootstrap (groundwork)", "conviction.empirical"],
"low": [t_ci[0], b_lo, c_lo], "high": [t_ci[1], b_hi, c_hi],
}).assign(contains_true_mean=lambda d: (d.low <= true_high) & (true_high <= d.high)).round(3)
| method | low | high | contains_true_mean | |
|---|---|---|---|---|
| 0 | t-interval | 33.628 | 35.526 | True |
| 1 | bootstrap (groundwork) | 33.609 | 35.465 | True |
| 2 | conviction.empirical | 33.628 | 35.501 | True |
conviction also answers questions about the uncertain value directly, returning a probability instead of a yes/no. For example, are High fields at least 9 °C hotter than Low ones?
diff = cn.empirical([rng.choice(hi_t.to_numpy(), len(hi_t)).mean() - rng.choice(lo_t.to_numpy(), len(lo_t)).mean() for _ in range(4000)])
claim = diff > 9.0
print(f"difference: {diff.mean():.2f} °C, 95% interval {np.round(diff.interval(0.95), 2)}")
print(f"P(difference > 9 °C) = {claim.probability():.3f}")
print("decide at 95% confidence:", claim.at_confidence(0.95))
difference: 9.16 °C, 95% interval [ 8.13 10.12] P(difference > 9 °C) = 0.631 decide at 95% confidence: False
Checking what "95%" means: draw 500 samples, build a t-interval from each, and count how many contain the true mean.
high_pop = df.loc[df[data.TARGET] == "High", "Temperature_C"].to_numpy()
fig, ax = plt.subplots(figsize=(8, 4))
covered = 0
for i in range(500):
smp = rng.choice(high_pop, 60)
lo, hi = stats.t.interval(0.95, 59, smp.mean(), smp.std(ddof=1) / np.sqrt(60))
ok = lo <= true_high <= hi
covered += ok
if i < 60:
ax.plot([lo, hi], [i, i], color="#2a9d8f" if ok else "#e76f51", lw=1.5)
ax.axvline(true_high, color="black", ls="--")
ax.set(xlabel="temperature (°C)", ylabel="sample", title=f"{covered / 500:.1%} of 500 intervals contain the true mean (first 60 drawn)")
plotting.save("g05-ci-coverage")
plt.show()
1.4 Correlation¶
A correlation from a sample is also an estimate with uncertainty. Its p-value tests H₀: ρ = 0, and the Fisher z-transform gives a CI. With 630,000 rows, even tiny correlations are "significant", which is why the size of r matters more than its p-value.
def corr_ci(x, y, level=0.95):
r, p = stats.pearsonr(x, y)
zf, se = np.arctanh(r), 1 / np.sqrt(len(x) - 3)
q = stats.norm.ppf(0.5 + level / 2)
return r, p, np.tanh(zf - q * se), np.tanh(zf + q * se)
need = df[data.TARGET].cat.codes
rows = []
for col in ["Soil_Moisture", "Wind_Speed_kmh", "Temperature_C", "Humidity", "Soil_pH"]:
for n, label in [(500, "sample of 500"), (len(df), "all 630,000")]:
idx = rng.choice(len(df), n, replace=False) if n < len(df) else slice(None)
r, p, lo, hi = corr_ci(df[col].to_numpy()[idx], need.to_numpy()[idx])
rows.append({"feature": col, "data": label, "r": r, "p-value": p, "95% CI": f"[{lo:.3f}, {hi:.3f}]"})
pd.DataFrame(rows).round(4)
| feature | data | r | p-value | 95% CI | |
|---|---|---|---|---|---|
| 0 | Soil_Moisture | sample of 500 | -0.4785 | 0.0000 | [-0.543, -0.408] |
| 1 | Soil_Moisture | all 630,000 | -0.4555 | 0.0000 | [-0.457, -0.454] |
| 2 | Wind_Speed_kmh | sample of 500 | 0.2472 | 0.0000 | [0.163, 0.328] |
| 3 | Wind_Speed_kmh | all 630,000 | 0.2582 | 0.0000 | [0.256, 0.260] |
| 4 | Temperature_C | sample of 500 | 0.2546 | 0.0000 | [0.171, 0.335] |
| 5 | Temperature_C | all 630,000 | 0.2529 | 0.0000 | [0.251, 0.255] |
| 6 | Humidity | sample of 500 | -0.0151 | 0.7356 | [-0.103, 0.073] |
| 7 | Humidity | all 630,000 | -0.0219 | 0.0000 | [-0.024, -0.019] |
| 8 | Soil_pH | sample of 500 | 0.0371 | 0.4082 | [-0.051, 0.124] |
| 9 | Soil_pH | all 630,000 | 0.0002 | 0.9050 | [-0.002, 0.003] |
Humidity has r ≈ −0.02: in a sample of 500 that's indistinguishable from 0, but with all 630,000 rows the p-value is tiny. It is real but far too weak to be useful. Correlation also isn't causation: wind doesn't make fields thirsty directly, it raises evaporation.
2. Estimation and loss functions¶
2.1 Maximum likelihood estimation¶
MLE picks the parameters that make the observed data most probable. For a normal distribution the answer has a closed form (sample mean, and the 1/n standard deviation); a numerical optimiser on the negative log-likelihood has to land on the same place.
x = df["Temperature_C"].sample(2000, random_state=1).to_numpy()
closed = mle_normal(x)
res = optimize.minimize(normal_nll, x0=[20.0, 5.0], args=(x,), method="Nelder-Mead")
print("closed form: μ = %.4f, σ = %.4f" % closed)
print("numerical: μ = %.4f, σ = %.4f (%d iterations)" % (*res.x, res.nit))
# MLE for a probability: P(High) is the share of High fields
k, n = (df[data.TARGET] == "High").sum(), len(df)
ps = np.linspace(0.02, 0.05, 400)
loglik = k * np.log(ps) + (n - k) * np.log(1 - ps)
print("Bernoulli MLE of P(High):", round(ps[np.argmax(loglik)], 4), "| k/n =", round(k / n, 4))
closed form: μ = 27.1160, σ = 8.5716 numerical: μ = 27.1159, σ = 8.5716 (45 iterations) Bernoulli MLE of P(High): 0.0334 | k/n = 0.0333
For classification, maximising the likelihood of the labels under a softmax model is exactly minimising the cross-entropy loss. That's the link between statistics and the optimisers in section 4.
2.2 Loss functions¶
The loss decides what "wrong" costs. Squared error punishes big mistakes heavily (and leads to the mean), absolute error is robust (and leads to the median), Huber is quadratic near 0 and linear far away. For classification, log loss and hinge loss are written in terms of the margin y·f(x).
r = np.linspace(-3, 3, 300)
mg = np.linspace(-3, 3, 300)
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(r, mse(r), label="squared (MSE)")
axes[0].plot(r, mae(r), label="absolute (MAE)")
axes[0].plot(r, huber(r, 1.0), label="Huber, δ = 1")
axes[0].set(title="Regression losses", xlabel="residual y − ŷ", ylim=(0, 4))
axes[1].plot(mg, (mg <= 0).astype(float), label="0–1 loss")
axes[1].plot(mg, log_loss(mg) / np.log(2), label="log loss (scaled)")
axes[1].plot(mg, hinge(mg), label="hinge")
axes[1].set(title="Classification losses", xlabel="margin y·f(x)", ylim=(0, 4))
for ax in axes:
ax.legend()
fig.tight_layout()
plotting.save("g05-losses")
plt.show()
values = np.append(df["Soil_Moisture"].sample(99, random_state=0).to_numpy(), 999.0) # one broken sensor
grid = np.linspace(0, 120, 4001)
best = {name: grid[np.argmin([f(values - c).sum() for c in grid])] for name, f in [("MSE", mse), ("MAE", mae), ("Huber", huber)]}
print("best constant prediction with one 999 reading:", {k: round(v, 2) for k, v in best.items()})
print("mean =", round(values.mean(), 2), "| median =", round(np.median(values), 2))
best constant prediction with one 999 reading: {'MSE': np.float64(43.26), 'MAE': np.float64(29.7), 'Huber': np.float64(29.76)}
mean = 43.26 | median = 29.9
One broken reading drags the MSE answer (the mean) far away; MAE (the median) barely moves.
3. The optimisation landscape¶
3.1 Convex vs non-convex functions¶
A convex function has no bumps: every chord lies above the curve, so any local minimum is the global one. Logistic and softmax regression losses are convex; neural network losses are not.
xs = np.linspace(-2.2, 2.2, 400)
convex = xs**2
nonconvex = xs**4 - 3 * xs**2 + xs
fig, axes = plt.subplots(1, 2, figsize=(12, 3.8))
axes[0].plot(xs, convex, color=plotting.ACCENT)
axes[0].plot([-1.8, 1.2], [1.8**2, 1.2**2], "o--", color="#e76f51", label="a chord stays above")
axes[0].set_title("Convex: x²")
axes[0].legend()
axes[1].plot(xs, nonconvex, color=plotting.ACCENT)
for start, color in [(-2.0, "#2a9d8f"), (2.0, "#e76f51")]:
path = optim.minimize(lambda v: np.array([4 * v[0]**3 - 6 * v[0] + 1]), [start], optim.GD(0.02), 300)[:, 0]
axes[1].plot(path, path**4 - 3 * path**2 + path, "o", ms=3, color=color, label=f"GD from {start:+.0f} ends at {path[-1]:.2f}")
axes[1].set_title("Non-convex: x⁴ − 3x² + x")
axes[1].legend()
fig.tight_layout()
plotting.save("g05-convex")
plt.show()
On the non-convex function, the same algorithm ends in different minima depending on where it starts: the one on the right is only a local minimum.
3.2 Minima and saddle points¶
Where the gradient is zero, the Hessian (matrix of second derivatives) tells what kind of point it is: all eigenvalues positive → minimum; mixed signs → saddle point. f(x, y) = x² − y² has a saddle at the origin: a minimum along x and a maximum along y.
H_saddle = np.array([[2.0, 0.0], [0.0, -2.0]])
print("Hessian eigenvalues of x² − y² at (0, 0):", np.linalg.eigvalsh(H_saddle))
g1, g2 = np.meshgrid(np.linspace(-2, 2, 200), np.linspace(-2, 2, 200))
pts = np.stack([g1, g2], axis=-1)
fig, ax = plt.subplots(figsize=(5.5, 5))
ax.contour(g1, g2, optim.saddle(pts), levels=25, cmap="coolwarm")
path = optim.minimize(optim.saddle_grad, [1.8, 1e-3], optim.GD(0.05), 80)
ax.plot(path[:, 0], path[:, 1], "o-", ms=3, color="black", label="GD starting almost on the ridge")
ax.plot(0, 0, "*", ms=14, color="#e76f51", label="saddle point")
ax.set(xlim=(-2, 2), ylim=(-2, 2), title="GD slows down near a saddle, then escapes")
ax.legend(loc="lower left", fontsize=8)
plotting.save("g05-saddle")
plt.show()
Hessian eigenvalues of x² − y² at (0, 0): [-2. 2.]
from sklearn.metrics import balanced_accuracy_score
from sklearn.model_selection import train_test_split
s = data.sample(40_000, seed=6)
X, y = data.xy(s)
X = X.to_numpy()
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
mu_tr, sd_tr = X_tr.mean(0), X_tr.std(0) + 1e-9
X_tr, X_te = (X_tr - mu_tr) / sd_tr, (X_te - mu_tr) / sd_tr
W, hist = optim.train_softmax(X_tr, y_tr, optim.GD(0.5), epochs=200)
print(f"loss {hist[0]:.4f} -> {hist[-1]:.4f}; test balanced accuracy {balanced_accuracy_score(y_te, optim.predict_softmax(W, X_te)):.4f}")
loss 1.0986 -> 0.3163; test balanced accuracy 0.6288
4.2 The learning rate¶
The single most important setting. Too small: painfully slow. Too large: every step overshoots the minimum, so the loss bounces instead of settling (η = 12 below), and larger still it diverges to infinity (checked in tests/test_optim.py).
fig, ax = plt.subplots()
for lr, color in [(0.01, "#e9c46a"), (0.5, "#2a9d8f"), (3.0, "#264653"), (12.0, "#e76f51")]:
_, h = optim.train_softmax(X_tr, y_tr, optim.GD(lr), epochs=100)
ax.plot(np.minimum(h, 3), label=f"η = {lr}", color=color)
ax.set(xlabel="epoch", ylabel="training loss (capped at 3)", title="Learning rate: too small, steady, fast, unstable")
ax.legend()
plotting.save("g05-learning-rate")
plt.show()
4.3 Variants of gradient descent¶
- Batch GD uses every example for each step: smooth but one step per epoch.
- Stochastic GD uses one example per step: noisy but many steps.
- Mini-batch GD uses small batches: the practical compromise, and what everyone actually runs.
fig, ax = plt.subplots()
rows = []
for name, bs, lr in [("batch", None, 0.5), ("mini-batch (256)", 256, 0.1), ("stochastic (1)", 1, 0.002)]:
t0 = time.perf_counter()
Wv, h = optim.train_softmax(X_tr, y_tr, optim.GD(lr), epochs=15, batch_size=bs)
rows.append({"variant": name, "final loss": h[-1], "test balanced acc.": balanced_accuracy_score(y_te, optim.predict_softmax(Wv, X_te)),
"seconds": time.perf_counter() - t0})
ax.plot(h, marker="o", ms=3, label=name)
ax.set(xlabel="epoch", ylabel="training loss", title="Same number of epochs, very different progress")
ax.legend()
plotting.save("g05-gd-variants")
plt.show()
pd.DataFrame(rows).round(4)
| variant | final loss | test balanced acc. | seconds | |
|---|---|---|---|---|
| 0 | batch | 0.4347 | 0.5916 | 0.3249 |
| 1 | mini-batch (256) | 0.3039 | 0.6903 | 0.4197 |
| 2 | stochastic (1) | 0.2911 | 0.7727 | 17.5636 |
5. Advanced optimisers¶
5.1 Momentum¶
Keeps a running velocity v ← βv + ∇L, so steps build up along consistent directions and oscillations cancel out.
5.2 RMSProp¶
Divides each parameter's step by a running root-mean-square of its gradients, so steep directions get small steps and flat ones get big steps.
5.3 Adam¶
Momentum and RMSProp together, plus a bias correction because both running averages start at zero.
The classic stress test is the Rosenbrock function: a long, curved, flat-bottomed valley with the minimum at (1, 1).
gx, gy = np.meshgrid(np.linspace(-2, 2, 300), np.linspace(-1, 3, 300))
Z = optim.rosenbrock(np.stack([gx, gy], axis=-1))
start = [-1.5, 2.5]
runs = {
"Gradient descent": optim.GD(0.0012),
"Momentum": optim.Momentum(0.0012, 0.9),
"RMSProp": optim.RMSProp(0.01),
"Adam": optim.Adam(0.05),
}
fig, ax = plt.subplots(figsize=(8, 6))
ax.contour(gx, gy, np.log1p(Z), levels=30, cmap="Greys", alpha=0.6)
summary = []
for (name, opt), color in zip(runs.items(), ["#e9c46a", "#2a9d8f", "#264653", "#e76f51"]):
path = optim.minimize(optim.rosenbrock_grad, start, opt, steps=2000)
ax.plot(path[:, 0], path[:, 1], lw=1.6, color=color, label=name)
summary.append({"optimiser": name, "final point": np.round(path[-1], 3), "distance to (1, 1)": np.linalg.norm(path[-1] - 1),
"steps to get within 0.05": next((i for i, p in enumerate(path) if np.linalg.norm(p - 1) < 0.05), None)})
ax.plot(1, 1, "*", ms=16, color="black", label="minimum (1, 1)")
ax.plot(*start, "o", color="black")
ax.set(title="2,000 steps on the Rosenbrock valley", xlabel="x", ylabel="y")
ax.legend(loc="upper right")
plotting.save("g05-rosenbrock")
plt.show()
pd.DataFrame(summary)
| optimiser | final point | distance to (1, 1) | steps to get within 0.05 | |
|---|---|---|---|---|
| 0 | Gradient descent | [0.706, 0.497] | 0.582486 | NaN |
| 1 | Momentum | [1.0, 1.0] | 0.000051 | 638.0 |
| 2 | RMSProp | [0.93, 0.879] | 0.139986 | NaN |
| 3 | Adam | [1.0, 1.0] | 0.000003 | 979.0 |
Adam and Momentum reach the minimum; RMSProp gets into the valley quickly but then bounces along its floor (a fixed learning rate and no momentum), and plain gradient descent crawls: after 2,000 steps it is still far away. On a badly conditioned problem like this, the adaptive and momentum methods are a big win.
And on the real problem, the irrigation classifier with mini-batches of 256. Comparing optimisers is only fair if each one gets its own best learning rate, so first a small sweep:
sweep = {"Gradient descent": (optim.GD, [0.03, 0.1, 0.3, 1.0]), "Momentum": (optim.Momentum, [0.01, 0.03, 0.1, 0.3]),
"RMSProp": (optim.RMSProp, [0.001, 0.003, 0.01, 0.03]), "Adam": (optim.Adam, [0.001, 0.003, 0.01, 0.03, 0.1])}
rows = []
for name, (cls, lrs) in sweep.items():
for lr in lrs:
_, h = optim.train_softmax(X_tr, y_tr, cls(lr), epochs=25, batch_size=256, seed=0)
rows.append({"optimiser": name, "η": lr, "loss after 25 epochs": h[-1]})
sweep_df = pd.DataFrame(rows)
best_lr = sweep_df.loc[sweep_df.groupby("optimiser")["loss after 25 epochs"].idxmin()].set_index("optimiser")
sweep_df.pivot(index="η", columns="optimiser", values="loss after 25 epochs").round(4)
| optimiser | Adam | Gradient descent | Momentum | RMSProp |
|---|---|---|---|---|
| η | ||||
| 0.001 | 0.3425 | NaN | NaN | 0.3126 |
| 0.003 | 0.3008 | NaN | NaN | 0.2903 |
| 0.010 | 0.2897 | NaN | 0.2961 | 0.2898 |
| 0.030 | 0.3016 | 0.3199 | 0.2892 | 0.3069 |
| 0.100 | 0.3360 | 0.2962 | 0.2931 | NaN |
| 0.300 | NaN | 0.2896 | 0.3164 | NaN |
| 1.000 | NaN | 0.2997 | NaN | NaN |
Then each optimiser at its best learning rate. The last row is scikit-learn's LogisticRegression (L-BFGS solver, effectively unregularised) as an independent reference for the true minimum.
from sklearn.linear_model import LogisticRegression
fig, ax = plt.subplots()
rows = []
for (name, (cls, _)), color in zip(sweep.items(), ["#e9c46a", "#2a9d8f", "#264653", "#e76f51"]):
lr = best_lr.loc[name, "η"]
Wv, h = optim.train_softmax(X_tr, y_tr, cls(lr), epochs=25, batch_size=256, seed=0)
ax.plot(h, marker="o", ms=3, label=f"{name} (η = {lr})", color=color)
rows.append({"optimiser": name, "η": lr, "loss after 1 epoch": h[1], "loss after 25": h[-1],
"test balanced accuracy": balanced_accuracy_score(y_te, optim.predict_softmax(Wv, X_te))})
ref = LogisticRegression(C=1e6, max_iter=3000).fit(X_tr, y_tr)
ref_loss = optim.cross_entropy(np.vstack([ref.intercept_, ref.coef_.T]), np.hstack([np.ones((len(X_tr), 1)), X_tr]), np.eye(3)[y_tr])[0]
ax.axhline(ref_loss, ls="--", color="black", lw=1, label="scikit-learn minimum")
rows.append({"optimiser": "sklearn LogisticRegression (reference)", "η": np.nan, "loss after 1 epoch": np.nan,
"loss after 25": ref_loss, "test balanced accuracy": balanced_accuracy_score(y_te, ref.predict(X_te))})
ax.set(xlabel="epoch", ylabel="training cross-entropy", ylim=(0.27, 0.5), title="Softmax regression on the irrigation data")
ax.legend(fontsize=8)
plotting.save("g05-optimisers-real")
plt.show()
pd.DataFrame(rows).round(4)
| optimiser | η | loss after 1 epoch | loss after 25 | test balanced accuracy | |
|---|---|---|---|---|---|
| 0 | Gradient descent | 0.30 | 0.3555 | 0.2896 | 0.7756 |
| 1 | Momentum | 0.03 | 0.3480 | 0.2892 | 0.7747 |
| 2 | RMSProp | 0.01 | 0.3876 | 0.2898 | 0.7867 |
| 3 | Adam | 0.01 | 0.4148 | 0.2897 | 0.7877 |
| 4 | sklearn LogisticRegression (reference) | NaN | NaN | 0.2865 | 0.7925 |
Two honest findings. First, once each optimiser has a good learning rate, all four land within 0.004 of scikit-learn's minimum, which confirms the from-scratch implementations. On a convex, well-scaled problem like this one, the learning rate matters more than the choice of optimiser; the clever ones only pull ahead on hard landscapes like Rosenbrock. Second, they stop slightly above the true minimum, because mini-batch gradients are noisy and the step size never shrinks (a learning-rate schedule would close the gap).
The balanced accuracy itself is limited by the model, not the optimiser: a linear boundary can't capture the threshold and interaction effects found in the EDA. That is the motivation for the tree ensembles in CSE274.
| Syllabus topic | Where |
|---|---|
| Sampling and population | 1.1 (CLT, standard error) |
| Hypothesis testing | 1.2 (t, χ², permutation, type I error) |
| Confidence intervals | 1.3 (t, bootstrap, conviction, coverage) |
| Correlation | 1.4 (significance, Fisher CI) |
| Maximum likelihood estimation | 2.1 |
| Loss functions | 2.2 |
| Convex vs non-convex, minima and saddle points | 3.1, 3.2 |
| Gradient descent, learning rate, variants | 4.1–4.3 |
| Momentum, RMSProp, Adam | 5 (from scratch, Rosenbrock and real data) |