Calibration Plots¶
Summary¶
Calibration plots (also called reliability diagrams or calibration curves) are visual tools for checking whether a model's predicted probabilities match actual outcome frequencies. A perfectly calibrated model that predicts 60% win probability should win exactly 60% of those predictions over a large sample.
The plot shows bins of predicted probability on the x-axis and actual outcome frequency on the y-axis, with a diagonal reference line (perfect calibration). Points above the diagonal = model is underestimating probability (conservative); points below = model is overconfident.
Calibration is fundamental to sports betting because the model outputs probabilities that are compared to de-vigged bookmaker odds to find +EV bets. If the model is poorly calibrated, the probability estimates are systematically wrong, and EV calculations will be unreliable.
The client's spec requires calibration testing as part of the backtesting framework, alongside CLV and walk-forward validation.
Key Concepts¶
- Reliability diagram: The standard name for a calibration plot. Shows "reliability" of probabilities — how reliable they are as confidence estimates.
- Bin-based calibration: Group predictions into bins (e.g., 0-5%, 5-10%, ..., 95-100%), compute empirical frequency within each bin, plot against bin midpoint.
- Expected Calibration Error (ECE): Single number summarizing calibration quality. Weighted average of |predicted - empirical| across bins.
- Underconfidence (conservative): Points above diagonal — model predicts 50% but actually wins 55%. The model is underestimating its own accuracy.
- Overconfidence: Points below diagonal — model predicts 70% but actually wins 60%. The model is too certain.
- Platt scaling / isotonic regression: Post-hoc calibration methods to adjust model outputs to better match actual frequencies.
Formulas¶
Expected Calibration Error (ECE):
$$ECE = \sum_{b=1}^{B} \frac{n_b}{N} \cdot |acc_b - conf_b|$$
Where:
- B = number of bins
- n_b = number of predictions in bin b
- N = total predictions
- acc_b = actual accuracy (win rate) in bin b
- conf_b = average predicted probability in bin b
Maximum Calibration Error (MCE): Maximum |acc_b - conf_b| across bins — worst bin.
Confidence interval for calibration: With small samples (64 World Cup matches), calibration plots have wide confidence intervals. Use bootstrapping or binominal confidence intervals.
Python Implementation¶
import numpy as np
import matplotlib.pyplot as plt
def calibration_plot(predicted_probs, actual_outcomes, n_bins=10, model_name='Model'):
"""
Generate a calibration plot (reliability diagram).
Args:
predicted_probs: array of predicted probabilities
actual_outcomes: array of actual outcomes (0 or 1)
n_bins: number of probability bins
model_name: label for the plot
Returns:
dict with bin statistics and matplotlib figure
"""
bin_edges = np.linspace(0, 1, n_bins + 1)
bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2
bin_counts = np.zeros(n_bins)
bin_accuracy = np.zeros(n_bins)
bin_confidence = np.zeros(n_bins)
for i in range(n_bins):
mask = (predicted_probs >= bin_edges[i]) & (predicted_probs < bin_edges[i+1])
# Last bin: include right edge
if i == n_bins - 1:
mask = (predicted_probs >= bin_edges[i]) & (predicted_probs <= bin_edges[i+1])
bin_counts[i] = mask.sum()
if bin_counts[i] > 0:
bin_accuracy[i] = actual_outcomes[mask].mean()
bin_confidence[i] = predicted_probs[mask].mean()
# ECE
ece = np.sum((bin_counts / len(predicted_probs)) * np.abs(bin_accuracy - bin_confidence))
# Plot
fig, ax = plt.subplots(figsize=(8, 6))
ax.plot([0, 1], [0, 1], 'k--', label='Perfect calibration', linewidth=2)
ax.scatter(bin_centers, bin_accuracy, s=bin_counts * 5, alpha=0.6,
label=f'{model_name}', c='steelblue')
ax.set_xlabel('Mean Predicted Probability', fontsize=12)
ax.set_ylabel('Fraction of Positives (Actual Win Rate)', fontsize=12)
ax.set_title(f'Calibration Plot — {model_name}\nECE = {ece:.4f}', fontsize=14)
ax.legend()
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
return {
'bin_centers': bin_centers,
'bin_counts': bin_counts,
'bin_accuracy': bin_accuracy,
'bin_confidence': bin_confidence,
'ece': ece,
'figure': fig
}
def expected_calibration_error(predicted_probs, actual_outcomes, n_bins=10):
"""
Calculate ECE without plotting.
"""
bin_edges = np.linspace(0, 1, n_bins + 1)
ece = 0
for i in range(n_bins):
if i < n_bins - 1:
mask = (predicted_probs >= bin_edges[i]) & (predicted_probs < bin_edges[i+1])
else:
mask = (predicted_probs >= bin_edges[i]) & (predicted_probs <= bin_edges[i+1])
n_b = mask.sum()
if n_b > 0:
acc_b = actual_outcomes[mask].mean()
conf_b = predicted_probs[mask].mean()
ece += (n_b / len(predicted_probs)) * abs(acc_b - conf_b)
return ece
def platt_calibration(probs, outcomes):
"""
Platt scaling: fit a logistic regression to calibrate probabilities.
Can improve calibration if model is systematically over/under-confident.
"""
from sklearn.linear_model import LogisticRegression
# Clip for logit stability
p = np.clip(probs, 1e-5, 1 - 1e-5)
logit = np.log(p / (1 - p)).reshape(-1, 1)
calibrator = LogisticRegression()
calibrator.fit(logit, outcomes)
calibrated_probs = calibrator.predict_proba(logit)[:, 1]
return calibrated_probs
# Example
np.random.seed(42)
probs = np.random.beta(2, 2, size=500) # simulated predictions
outcomes = (np.random.random(500) < probs).astype(int) # stochastic outcomes
result = calibration_plot(probs, outcomes, n_bins=8)
plt.savefig('/tmp/calibration.png', dpi=150)
print(f"ECE: {result['ece']:.4f}")
Notes¶
- Calibration is a necessary but not sufficient condition for a good betting model — a model can be perfectly calibrated but have no predictive value (predict 50% for everything)
- For World Cup with only ~64 matches: use wider bins (5-10 bins), expect wide confidence intervals on calibration
- Combine calibration plots with Brier score for comprehensive model assessment
- Overconfidence is the most common calibration issue in sports prediction models — neural networks and boosted trees tend to be overconfident
- Post-hoc calibration (Platt scaling, isotonic regression) can fix calibration without improving discrimination — useful if the model is well-ranked but poorly calibrated