Cherry-picking bias is quoting only the periods, instruments, or runs that flattered a strategy while leaving the rest out, so every number reported is true and the picture they add up to is not. It is an error of reporting: the underlying test can be entirely correct, and the distortion enters only when its results are described.
Also seen as: Cherry picking, selective reporting
Nothing has to be falsified — a good year really was a good year — but the reader is shown a slice of the evidence chosen because it was favorable, and has no way to see what was left out.
How does Fincanva handle it?
A Fincanva backtest reports the whole period it ran, not a chosen stretch of it, and reports the falls alongside the gains: the run that produces the return also produces its worst peak-to-trough drawdown, so the bad part of the history arrives attached to the good part. Because the run is a walk-forward replay of the full span, no version of the result covers only the years that worked.
Start-date sensitivity answers the window form directly: it re-runs the same strategy across many entry dates and holding windows and reports the range of outcomes — the best start against the worst, and the share of start dates that ended positive. A claim that holds for one entry month shows up as a wide range.
What the product cannot do is decide what you tell other people, or yourself: one favorable screenshot from a full run is still available to anyone. Turned inwards, that is confirmation bias; when the numbers themselves are wrong, it is data-quality bias.
What gets cherry-picked in a backtest?
Four things are usually picked, and each is picked the same way — after the results are already known:
- The window. A start and end date chosen because the stretch between them went well. This is the most common form, because a backtest makes any window one setting away.
- The instruments. The names in the report are the ones that contributed; the ones that dragged get described as "not really part of the idea".
- The metric. The return is quoted and the worst drawdown is not, or the gross curve is shown while the net-of-costs one — the number a real account would have kept — is not (see cost-ignoring bias).
- The run. One configuration out of many tried is presented as the strategy, with no mention of the others.
One test catches all four: could someone reproduce the claim without knowing which slice you chose? If the claim only holds on your slice, the slice is doing the work, not the strategy.
What changes when you report 2019 and omit 2022?
A strategy is backtested over the ten calendar years 2015–2024. Two of those years stand out: 2019 returned +31% and 2022 returned −29%.
A report built around 2019 — the year label, the rising curve, the +31% — states a fact. Here is what the same run also says:
| Figure from the same run | Value |
|---|---|
| 2019 calendar year | +31% |
| 2022 calendar year | −29% |
| Full period, total return | +72% |
| Full period, annualized (CAGR) | 5.6% a year |
| Worst peak-to-trough fall in the period | −34% |
Quoting 2019 alone invites the reader to treat +31% as what the strategy does in a year; the run's own annualized figure is about a fifth of that. Put the omitted year back and the pair alone leaves the strategy below where it started: 1.31 × 0.71 = 0.93, or −7% across the two years together. None of those five numbers contradicts the others — they all come from one run. The distortion is entirely in which of them got quoted.
How is cherry-picking bias different from selection bias and data-snooping bias?
The three differ by which step goes wrong: cherry-picking bias is about what you report, selection bias is about what you tested, and data-snooping bias is about how many things you tested before something looked good.
They also stack, in that order. A researcher who tries fifty variants (data-snooping), keeps the one with the most flattering instrument list (selection), and then presents its best three years (cherry-picking) has committed all three, and the final report shows no trace of the first two.