The numbers first
Everything below is from a walk-forward backtest over 7,401 FBS games, 2017–2025,
training only on prior seasons — the model never sees a game before predicting it.
Walk-forward results vs. baselines
| Metric | This model | Benchmark |
|---|
| Straight-up accuracy | 75.5% | Vegas closing lines: 76.3% |
| Margin RMSE | 16.37 | Always-pick-home: 23.06 · SP+ differential: 18.75 |
| Brier score | 0.162 | — |
Win probabilities are kernel-smoothed and isotonic-calibrated, with calibration
reported out-of-fold so the reliability table isn’t perfect by construction.
Across ten calibration buckets the worst deviation is 1.1 standard errors: when the model says
65%, teams win about 65% of the time. Landing within a point of the closing line — a benchmark
that aggregates enormous amounts of information — is the headline for a from-scratch system.
What the model can’t do, measured
The honesty section
Across seven spread buckets and seven total buckets, no betting edge is statistically
significant. Hit rates run 48.6%–55.5% against the 52.4% break-even, non-monotonically —
exactly what chance produces across seven looks. The system says so on its own report card:
the backtest prints a z-score and a “Real?” flag per bucket, and the weekly projection repeats
the verdict so it can’t be forgotten between runs.
Building the significance test that kills your own headline is the point: a model you can’t
trust to admit failure can’t be trusted to claim success.
Architecture
-
Data layer: 17-endpoint REST API ingestion (collegefootballdata.com) into a
local Parquet cache, with API-budget awareness built in — the free tier allows
~1,000 calls/month, a full historical backfill costs 200–400, and the client hard-stops before
exceeding the budget.
-
Ratings: a ridge-regression team rating layer — offense and
defense per team, solved as one penalized regression per season, shrinking each team toward the
mean in proportion to how little evidence supports it. Anchored on preseason priors: published
SP+ blended with an in-house regression on returning production, recruiting, and prior ratings,
so the cold start is handled without double-shrinkage.
-
Models: two LightGBM models (margin and total) over ~28
features — deliberately not a neural network. About 10,000 rows against 28 features is
squarely boosted-tree territory; a network with enough capacity to be interesting just
memorizes the data.
-
Season simulation: a 10,000-run Monte Carlo simulator with
empirically measured correlated team-strength uncertainty (sd 3.4 points, measured three
independent ways). Projected records are exact by linearity of expectation — the simulation
supplies the spread around them.
-
The lab: an isolated experiment harness. Candidate model ideas run against two
fixed walk-forward folds and must beat the baseline on both to be adopted.
Several plausible ideas split the folds and died as noise — which is the harness working.
-
Weekly operational loop: project the slate before kickoff, grade the saved
projections after — against a backtest baseline matched to the same weeks — so
drift is a number rather than a feeling. Grading the file saved before kickoff
matters: a re-derived backtest can quietly benefit from information that didn’t exist that week.
-
Tests: an automated suite covering the data layer, ratings, pipeline, grading,
and season simulation — run against synthetic leagues with known ground truth, including leakage
guards (rewriting a week’s results must move zero features, and the model must
not beat a market built from ground truth — if it does, that’s leakage, not skill).
Why calibrated probabilities, not a normal curve
College football margins are fat-tailed and cluster on key numbers (3, 7, 10, 14), so a normal
CDF is overconfident in the middle and underconfident in the tails. Instead, the empirical
distribution of out-of-sample margin errors is kernel-smoothed, then calibrated with isotonic
regression — and evaluated out-of-fold, so the calibration claim is earned rather than assumed.
A research finding worth keeping
Games involving a first-year head coach carry RMSE 15.53 vs 14.99 for all other
games, on essentially identical outcome variance — they’re harder to predict, not wilder.
That points toward wider uncertainty on those games rather than different point predictions:
the features identify the problem without fixing it, and honest measurement says what the next
step should be.
What this project is evidence of
- Evaluation-first ML: walk-forward validation, calibrated probabilities, drift measurement, and negative results reported as prominently as wins.
- Production habits applied to a model: cached ingestion with budget guards, an experiment gauntlet, a graded weekly loop, and a test suite that catches lookahead bugs before they become quietly wrong projections.
- Judgment about tools: boosted trees over a neural network for 10,000 rows, ridge shrinkage where 130+ teams play 12 games each, exact expectations where simulation would only add noise.
← Back to portfolio