Backtest reliability

Also seen as: bootstrap test, p-value, backtest significance, bootstrap confidence interval

What does "Could it be luck?" measure?

The verdict answers one question: if this strategy had no edge at all over cash, how often would its history still produce a Sharpe ratio as high as the one the backtest shows? That share is a p-value, printed as "p = …", and the sentence under it says it in words: "A strategy with no edge over cash and the same volatility would look this good in … of cases."

The verdict reads that share on three rungs:

In prose: a low share means a no-edge strategy rarely matches this result, so luck is an unlikely explanation; a high share means an edgeless strategy often does, so the result proves little. The p-value is never exactly zero.

What does the verdict leave out?

It tests one strategy, as if it were the only one you ever tried. Tried twenty variants and kept the best? Among twenty strategies with no edge, one will often clear the 5% rung by chance alone, and this verdict cannot see the other nineteen — the app says so under it: "It speaks about the past, not about what the strategy will do, and it does not account for how many variants you tried before this one." That is data-snooping bias, and its cousin overfitting is the reason a strong verdict on a heavily tuned strategy deserves less weight than the same verdict on a strategy set up once.

How does resampling show how much the figures could have varied?

Resampling rebuilds the backtest's history many times by drawing its own days again, with replacement, and recomputes the figures on each rebuilt history. The spread of those recomputed figures is what the panel draws under How much they could have varied · 5–95% band: for Average annual return (arithmetic), Sharpe and Maximum drawdown, a band from the 5th to the 95th percentile of the resampled values, with a dot for the backtest's own value. This is the textbook bootstrap confidence interval.

The dot can fall outside the band — the app notes it — and does so most often for the maximum drawdown, because drawing days again breaks up the long losing runs a drawdown is made of. The return is the arithmetic average annual return, the average daily return scaled to a year, so it does not match the compound growth rate, CAGR, on Performance Metrics.

Two methods decide how days are drawn: Blocks of days draws runs of consecutive days, keeping calm and turbulent spells together; Single days draws each day on its own. More resamples make the band's edges steadier; they do not make the backtest more trustworthy.

What is the average annual return (arithmetic)?

The Average annual return (arithmetic) is the average daily return of the strategy multiplied by 252, the trading days in a year. It is the return the Robustness tab and contribution analytics use, and it is a different figure from both the compound growth rate, CAGR, and the AAGR on Performance Metrics.

where the average is the plain mean of the backtest's daily returns. Because it averages returns instead of compounding them, it ignores the drag volatility puts on growth: a day at −10% followed by a day at +10% averages to 0%, yet leaves the capital 1% lower. The higher the volatility, the wider that gap, so on a very volatile strategy the arithmetic figure can be positive while the strategy lost money over the period. Read it as the strength of the average day, not as what the capital did.

How does Fincanva handle it?

• It runs on request, from the Robustness tab: choose a method and a number of resamples — 100 or 1,000 — and press Calculate. Nothing is computed before that.
• Years in profit counts how many of the years in the Year by year table had an average daily return, scaled to a year, above zero — "7 of 10", say. In a very volatile year that can disagree with the sign the same year shows on Monthly Returns, which compounds.
• The Year by year table gives each calendar year's days, average annual return, Sharpe and maximum drawdown. Each year's Sharpe uses that year's own risk-free rate. A year needs at least 40 trading days to appear; if none has, the table is replaced by a note saying so.
• The figures follow the tab's Simulation settings: with Costs & interests or Taxes on, the check runs on the net returns.
• A simulation stored before the panel existed has to be run again first; until then the panel says it isn't available.
• The same request on the same backtest always returns the same result.

How many resamples one calculation may use is set by your plan, and a plan that allows none does not include the panel:

Set by your plan: 0 on Free and Starter, 100 on Advanced and 1000 on Ultimate and Professional. See what each plan includes.

A number of resamples beyond your plan stays in the list, marked with the plan that includes it, and choosing it opens a window instead of calculating.

What does it look like in practice?

A strategy's backtest shows a Sharpe of 0.9 over twelve years. The panel returns p = 0.03 — "Probably not": only 3 in 100 no-edge histories reach a Sharpe of 0.9. The band for the average annual return runs from 4% to 14% around a backtest value of 9%, and Years in profit reads 9 of 12. Read together: the result is unlikely to be pure chance, but the same days could plausibly have produced anything from a modest 4% to a strong 14% a year. Now suppose the strategy was the best of thirty variants: the 3% no longer means much, because among thirty edgeless tries it is more likely than not that at least one lands at 3% or below.