Backtesting Framework¶

Summary¶

A backtesting framework systematically evaluates a sports betting model by simulating how it would have performed on historical data. The framework must mimic real deployment conditions: using only information available at prediction time, tracking realistic transaction costs (vig), computing proper Kelly stakes, and measuring not just ROI but also CLV, calibration, and risk metrics.

The client's spec requires walk-forward validation across 2010, 2014, 2018, and 2022 World Cups — training on all prior data before each tournament and generating predictions before each match. This gives an honest picture of how the model performs out-of-sample.

Key metrics for a sports betting backtest:
1. ROI on staked: Total profit/total staked. Accounts for varying stake sizes.
2. Hit rate: % of bets that win. Alone, this is misleading without odds context.
3. CLV: Whether the model beats the closing line. Higher standard than just ROI.
4. Brier score: Probability calibration quality.
5. Max drawdown: Worst peak-to-trough decline. Critical for bankroll management.
6. Sharpe ratio: Risk-adjusted return. Allows comparison across strategies.

Framework Architecture¶

class SportsBettingBacktester:
    def __init__(self, bankroll=10000, kelly_fraction=0.5, min_edge=0.03):
        self.bankroll = bankroll
        self.kelly_fraction = kelly_fraction
        self.min_edge = min_edge
        self.history = []

    def run_tournament(self, model, fixtures_df, closing_odds_df):
        """
        Run backtest for a single World Cup tournament.

        Args:
            model: trained prediction model
            fixtures_df: match dates, teams, home/away
            closing_odds_df: Pinnacle closing odds for all matches
        """
        results = []

        for _, match in fixtures_df.iterrows():
            # 1. Generate prediction
            pred = model.predict(match)
            prob_home = pred['home_win_prob']
            prob_draw = pred['draw_prob']
            prob_away = pred['away_win_prob']

            # 2. Get bookmaker odds (The Odds API or historical data)
            bookie_odds = self.get_odds(match)

            # 3. De-vig odds
            fair_probs = devig_multiplicative(bookie_odds)

            # 4. Compute EV for each outcome
            ev_home = prob_home * bookie_odds['home'] - 1
            ev_draw = prob_draw * bookie_odds['draw'] - 1
            ev_away = prob_away * bookie_odds['away'] - 1

            # 5. Find best EV bet
            evs = {'home': ev_home, 'draw': ev_draw, 'away': ev_away}
            best_bet = max(evs, key=evs.get)
            best_ev = evs[best_bet]

            if best_ev < self.min_edge:
                continue  # no bet

            # 6. Compute Kelly stake
            b = bookie_odds[best_bet] - 1
            p = pred[f'{best_bet}_prob']
            f_full = max((p * b - (1-p)) / b, 0)
            stake = self.bankroll * f_full * self.kelly_fraction

            # 7. Record bet (before outcome is known)
            self.history.append({
                'match': match['id'],
                'date': match['date'],
                'bet_on': best_bet,
                'odds': bookie_odds[best_bet],
                'model_prob': p,
                'stake': stake,
                'close_odds': closing_odds_df.loc[match['id'], best_bet],
                'ev': best_ev
            })

            # 8. Simulate result and update bankroll
            result = self.settle_bet(match, best_bet)
            self.update_bankroll(result, stake, bookie_odds[best_bet])

        return self.compute_metrics()

    def compute_metrics(self):
        """Compute all backtest metrics."""
        df = pd.DataFrame(self.history)

        metrics = {
            'n_bets': len(df),
            'n_wins': (df['result'] == 1).sum() if 'result' in df.columns else None,
            'hit_rate': (df['result'] == 1).mean() if 'result' in df.columns else None,
            'roi_on_staked': df['pnl'].sum() / df['stake'].sum() if 'pnl' in df.columns else None,
            'total_pnl': df['pnl'].sum() if 'pnl' in df.columns else None,
            'final_bankroll': df['bankroll'].iloc[-1] if 'bankroll' in df.columns and len(df) > 0 else self.bankroll,
            'max_drawdown': self.max_drawdown(df),
            'clv_mean': ((df['odds'] - df['close_odds']) / df['close_odds']).mean(),
            'brier_score': ((df['model_prob'] - df['result'])**2).mean()
        }
        return metrics

    def max_drawdown(self, df):
        """Calculate maximum drawdown."""
        if 'bankroll' not in df.columns:
            return 0
        peak = df['bankroll'].cummax()
        dd = (df['bankroll'] - peak) / peak
        return dd.min()

Key Metrics Definitions¶

Metric	Formula	Target
ROI on Staked	total_pnl / total_staked	> 0% (positive edge)
Hit Rate	wins / total_bets	Varies by odds
CLV Mean	mean((bet_odds - close_odds) / close_odds)	> 0%
Brier Score	mean((pred_prob - result)²)	< 0.20
Max Drawdown	max(peak - trough) / peak	< 30%
Sharpe Ratio	mean(pnl/stake) / std(pnl/stake) × sqrt(252)	> 1.0

Notes¶

A backtest without CLV tracking is incomplete: even profitable strategies can be worse than just betting the closing line
The client specifically requires walk-forward validation across 2010-2022 World Cups — implement this as the core validation framework
Simulating Kelly staking introduces bankroll dependency: early losses reduce later stake sizes, which is realistic but means backtest results are path-dependent
Transaction costs: always account for vig (use de-vigged odds for decision-making, actual odds for P&L)