← All articles

Backtest overfitting: 1,344 variants, zero survivors

11 min read · Published 27 Oct 2026

We tested 1,344 variants of simple rules built from popular indicators and picked the 13 best in 2018–2022. On average they made +0.20 R per trade. In 2023–2025, years we did not look at while choosing, the same rules made +0.05 R, and none of them passed validation. About three quarters of the “edge” was the cost of picking the best out of many.

R is a trade’s result in units of risk (−1 R = a loss equal to the distance to the stop). PF (profit factor) is total gains divided by total losses; a PF of 1.00 means breaking even. The t-statistic tells you how many times the average result exceeds its random error; in a single test, a t of about 2 is usually taken as the threshold of significance. ATR is the average range of a candle, a measure of volatility.

Hypothesis

Anyone who has looked at a backtest result knows the thought: now that we know which version worked, all we have to do is pick the best one. You can add a filter that would have cut out the losses, or change an exit that was giving back profit. In our case the question was put bluntly: why shouldn’t fitting a rule to a known result work?

Instead of answering with theory, we did exactly what the question was about: a deliberate, wide search. Stated at its best, the hypothesis is this: among combinations of popular indicators on the hourly chart there is one that makes money after costs not only in the years in which it was found, but also in the years that follow.

We wrote down our expectation before the start: zero survivors. Every component of the grid (levels, sessions, RSI, moving averages, breakouts) had already failed when we tested it on its own. The value of the test lay elsewhere: we wanted to show, in numbers, what the act of choosing costs.

How we tested it

The grid. We took six instruments: bitcoin, EUR/USD, gold, the German index, USD/JPY and the US technology index, all on hourly candles. Each variant consisted of four elements:

• trigger (8 options): a 20-candle breakout, a breakout of the Asian session range, a breakout of yesterday’s high or low, RSI crossing back through 30 or 70, a move back inside the Bollinger Bands, a crossover of the 20- and 50-period moving averages, a cross of the daily volume-weighted average price (VWAP), and the SMC setup from chapter 16;
• direction (2): with the trigger or against it;
• filter (7): none, the trend from the 200-period moving average, the daily trend, RSI above or below 50, elevated volatility, London and New York hours only, high volume;
• exit (2): trend-following (a wide stop, no fixed target, a limit of a few days) or short (a 1 ATR stop, a 2 ATR target, 24 hours).

6 × 8 × 2 × 7 × 2 gives 1,344 variants, and over a million trades in the training years. Costs came from ticks and included the account markup; we also charged swap. Entry was at market on the open of a candle, and all conditions used only candles that had already closed, which we verified with a data truncation test. Each variant could have one position open at a time.

Three separate periods. The whole point of the test was to keep searching apart from checking:

1. Training, 2018–2022. This was the only place where searching was allowed. A candidate needed at least 100 trades, an average above zero, PF ≥ 1.2 and PF ≥ 1.05 after removing the three best trades. At most the five best per instrument, ranked by t-statistic, went on to validation.
2. Validation, 2023–2025. The selected candidates went through this period without any change. To pass, they needed an average above zero and PF ≥ 1.15. The result had to beat 95% of draws with random entries and beat plain “always buy” at the same moments. We also required significance after correcting for the number of candidates (q < 0.10).
3. One shot at 2026, only for those that survived validation.

We froze the grid, the selection rules and the criteria with a checksum before the first run.

Arithmetic before the result. Each instrument had 224 variants. Before the start, we calculated what to expect if none of them had an edge. The best of 224 would still come out with a t-statistic of about 2.8. With about 300 trades, that means roughly +0.25 R per trade and a PF of about 1.3. In other words, a good result for the best variant in training was predictable even if none of them had an edge.

Result

Training. Of the 1,273 variants with at least 100 trades, 268 ended in profit, or 21%. The average trade across the whole family came to −0.060 R, because costs take their toll. The best t-statistic per instrument ranged from 1.99 to 3.01 — roughly what chance alone predicted. The candidate criteria were met by 46 variants, and 13 went on to validation. EUR/USD had no candidate at all; gold, the German index and the technology index had one each.

Validation:

All the rules except the last one used the trend-following exit. None of the 13 candidates passed validation, so we left 2026 unopened and closed the search.

The closest was bitcoin with the daily VWAP cross: 1,008 trades, +0.246 R, PF 1.38 and p = 0.008. It failed for two reasons. After correcting for 13 candidates, q came out at 0.104 against a threshold of 0.10, and the total result, +248 R, was lower than “always buy” at the same moments (+261 R). In 2023–2025 bitcoin was in a bull market. The bitcoin rules traded in the direction of the move, with an exit that lets profits run, so in practice they gave largely long exposure to a rising market — and the best of them made less than simply buying.

Why it doesn’t work

A training result is edge plus luck. Every measured result has two parts: what the rule can really do, and the chance of that particular sample. When we pick the best of 224 variants, we are mainly picking the one that got the luckiest. A real edge, if there is one, carries over into the following years; luck does not. That is why the best of many variants is overestimated by the very nature of the choice, however carefully it was calculated.

The shrinkage can be measured. The average of the 13 candidates fell from +0.201 R to +0.049 R, a drop of about three quarters. Outside bitcoin, the drop was total: the other eight rules averaged +0.16 R in training and about zero in validation. Bitcoin kept part of its result, but for a reason we already know: the whole market was rising, and the signal had nothing to do with it.

Thirteen candidates are fewer than thirteen ideas. Two of the USD/JPY rules differed only in the RSI filter and gave almost the same result in validation. Picking “the five best” tends to collect near-copies of a single phenomenon, so the apparent variety of candidates offers no protection against chance.

Not everything was noise, but it is not enough. In validation the candidates did slightly better than the average trade across the whole family (+0.049 R against −0.031 R), although part of that difference is the bitcoin bull market. So the selection from training carried a little information — just nowhere near the level that would beat costs, a simple benchmark and the correction for multiple testing.

Adding conditions after the fact is the same trap on a smaller scale. Every filter you add and every tweak to the exit “because it helped in the history” is one more variant in the search, even if nobody counts it. On the same family, we separately tested a “don’t trade against the yearly trend” filter: it worsened the average result in each of the three periods (chapter 22). In chapter 16, in turn, we describe a single cell chosen after the result, out of six. On the years 2018–2023, where we picked it, it had a PF of 1.38; on the unseen year 2026, a PF of 1.15, and after removing the three best trades only 0.33.

The problem has its own literature in finance. The “reality check” for data snooping (White, 2000) and the Sharpe ratio adjusted for the number of trials (Bailey and López de Prado, 2014) start from the same observation: what counts is not only the result of the best strategy, but also how many strategies were tested before it was found.

The lesson for you

Anyone who tests several versions of an idea is running a small search, even if the versions differ by a single setting. Treat the best version from your backtest as an upper bound on what can be achieved, not as a forecast.

How to protect yourself:

1. Split the data before you start searching: into training, validation and a final piece for a single shot. Search only in the training data. If you adjust the rule after validation, the validation data become just more training data.
2. Count every variant, including the ones you “only checked quickly”. Compare the best of 20 variants with what the best of 20 random throws gives, not with zero.
3. Freeze the selection rule and the validation criteria before you open the validation data: how many candidates, by which measure, which thresholds. Otherwise the choice of criterion itself becomes one more fit.
4. Expect shrinkage. In our case the best variants lost about three quarters of their result in the following years. If a strategy stops paying after shrinkage like that, there is nothing to deploy.
5. Compare with simply holding the instrument at the same moments. In a rising market, many rules “work” only because they buy.

In your trading journal, record not only your trades but also the date of every rule change and how many versions you tested before choosing the current one. The “Luck or edge?” calculator evaluates a single series of trades, so if that series is the best of many you have tested, read the range it shows as too optimistic.

Educational content, not investment advice. Past results do not guarantee future results. We describe the result of our search of a single family of rules, on our data and with our costs.