← All articles

A strategy generator always finds a winner. Why our tester doesn’t search for you

19 min read · Published 30 Oct 2026

We built 40 artificial markets on which, by construction, no one can make money, and on each of them we searched 100,000 variants of a simple breakout strategy. In the training years the best variant looked like a finished system: on average +0.24 R per trade and a PF of 1.47. In the validation years: −0.02 R and a PF of 0.97. This is not the bad luck of one simulation but the arithmetic of picking the best out of many. That is why our strategy tester deliberately has no search function.

R is a trade’s result in units of risk (−1 R = a loss equal to the distance to the stop). PF (profit factor) is total gains divided by total losses; a PF of 1.00 means breaking even. The standard error is the spread of an average caused by chance alone: the standard deviation of the results divided by the square root of the number of trades. The t-statistic is the average divided by its standard error; in a single test, a t of about 2 is usually taken as the threshold of significance. Training data is the data on which you may search and choose; validation data is set aside and looked at once, after the choice has been made.

Hypothesis

Strategy generators promise something that sounds reasonable. In one night a computer will check more combinations than a person could in a lifetime, throw out those that fail the filters and robustness tests, and keep those that passed everything. In its best version the hypothesis goes like this: if an edge exists among simple rules, a mass search will find it more reliably than manual guesswork, and robustness tests will separate it from luck.

In chapter 7 we described our own search of 1,344 variants, after which none of the 13 best survived validation. Here we ask about the scale generators work at, a hundred thousand and a million trials, and whether robustness tests run after the selection change anything. We looked for the answers on a market where we know the truth in advance: there is no edge there at all.

How automated search works

Generators differ in the details, but the scheme is usually similar:

1. Building blocks: entry conditions from popular indicators, filters (hours, trend, volatility) and exits (stop, target, trailing stop, time exit).
2. Random generation: thousands or millions of combinations with random parameters, each run on the same history.
3. Result filters: what remains are strategies with a large enough number of trades, PF or “quality” score.
4. Evolution: a genetic algorithm crosses and mutates the best strategies, evaluating each generation on the same history. A population of 1,000 strategies over 50 generations is 50,000 evaluations, even though the screen shows only the current population.
5. Robustness tests: reshuffling trades, higher spread, changed parameters, other markets, an “out-of-sample” segment, sometimes walk-forward.
6. Output: a bank of strategies that passed everything, often with ready-made code.

These are technically polished tools, and for many traders they are their first contact with systematic testing. The problem is not the code but the statistics of picking the best.

The arithmetic: how much the best of M is inflated

A backtest result is the sum of what the rule can really do and the chance of that particular sample. The size of the chance is measured by the standard error. If we test M variants with the same true average and pick the best, its result will on average be inflated by E_max(M) standard errors. E_max(M) is the expected maximum of M independent draws from a normal distribution.

The exact values come from numerical integration (for M = 10 and M = 100 we also checked them with 200,000 random draws), and the approximation from Blom’s formula (1958): Φ⁻¹((M − 0.375) / (M + 0.25)), where Φ⁻¹ is the inverse of the normal cumulative distribution function. In the table the two versions differ by at most 0.02, and our tester uses the approximation.

So the best of 100 variants without any edge has on average a t of about 2.5, a “significant” result by the usual threshold. In units of result: with 300 trades and a standard deviation of 1.5 R, the standard error is 1.5 / √300 ≈ 0.087 R, so the best of 10,000 independent variants without an edge will show on average about +0.33 R per trade. Variants from one family share many trades, so they count as fewer than M independent trials, and a correction for the full M is an upper bound. On the other hand, every evaluation counts: each generation of the algorithm, each parameter tweak and each rejected candidate.

Robustness tests run after the selection are multiple testing too. If, on the same data, we keep candidates that make money in both halves of the sample, at double the cost and with neighbouring parameters, we are only adding more filters to the same search. What passes them is the variants whose luck was more widespread, because a neighbour in the grid has almost the same trades and shares the same luck. The most dangerous step is using the “out-of-sample” segment as a filter. When, out of a thousand candidates, we keep only those that also did well there, that segment becomes a second part of training.

How we tested it

A market with no edge. We built 40 independent artificial markets. The price of each is a random walk: an hourly candle consists of 12 random steps, and volatility is 0.2% per hour, or about 15% a year at 6,000 candles per year. The steps are chosen so that the price is a martingale: its expected future value equals the current one. What follows is a theorem, not a hypothesis: every rule that decides only on the basis of the past has an expected value of zero before costs and minus the cost after costs. So no variant has an edge.

The strategy family. A Donchian channel breakout: a candle closing above the highest high or below the lowest low of the previous N candles, entry at the open of the next candle, one position at a time. The grid:

• channel length: 20 values from 10 to 240 candles;
• direction: with the breakout or against it;
• filter: none, one of three 8-hour windows of the day, or the trend from a 200-candle moving average;
• stop: 10 values from 0.5 to 5 ATR; target: none, or 1, 2, 3 or 5 R;
• time exit: 10 values from 4 to 120 candles.

In total 100,000 variants. We filled stops and targets at the price of the first step that crossed them, like a market order (filling exactly at the level would create a fake edge here). The trading cost was about 2.2% of the average hourly ATR, roughly what, according to our measurements, it costs to trade a large US index on the hourly chart (1.5–2.8%).

Periods and selection. Each market had 25,200 training candles (70%, about 4.2 years), 10,800 validation candles (30%) and 12,000 candles of “future”, needed for one demonstration below. Out of M variants, we picked the one with the highest t in training, with at least 100 trades. For M below 100,000 we drew subsets of the grid, for example a thousand subsets per market for M = 10. We also calculated the result after correction, that is, the average minus E_max(M) standard errors, the way our tester calculates it. The script has a fixed random seed and gives the same result every time.

Result

The best of M variants, averaged over 40 markets:

The more trials, the better the winner looks in training: from a PF of 1.12 with ten to a PF of 1.47 with a hundred thousand. Already the best of a hundred has a median PF of 1.25, the threshold we ourselves treat as the minimum for evidence. In validation all rows give the same: from −0.007 to −0.022 R per trade, roughly the cost (0.009–0.014 R for these variants), and 40–46% of validations end in profit. The average of all variants was −0.016 R in training and −0.013 R in validation, with a cost of 0.016 R, in line with the construction of the market.

An example from one market. The winner out of 100,000 variants had a t of 4.07 in training, a result that in a single test would count as very strong: 358 trades, +0.312 R per trade, PF 1.58. It was a 120-candle channel, entry against the breakout, a stop of 4 ATR, a target of 2 R and an exit after 120 candles. In validation it made 179 trades, −0.103 R per trade and a PF of 0.84.

The correction works, with room to spare. After subtracting E_max(M) standard errors, the winner’s average result is negative in every row, and for M = 10,000 and M = 100,000 it was never positive (for M = 10 it was positive in 22% of draws). Because the variants resemble each other, the correction calculated for independent trials is too large here: the winner out of 100,000 had a t of 3.14, not 4.37, and the actual inflation came to between 75% (M = 100,000) and 86% (M = 10) of the correction. We would rather subtract a little too much than show an edge that isn’t there.

Robustness tests after the selection, run one after another on the same training data, on the full grid:

On average 2,546 variants out of 100,000 passed the full set of tests run on training data, and on none of the 40 markets fewer than 576. In validation they made on average −0.008 R, roughly zero minus the cost. The double-cost test did not remove a single variant, because with a cost of about 0.016 R per trade the selected variants had a cushion of +0.16 R each, and that cushion was luck. The neighbour test removed 2%. The last row shows the most dangerous mistake: when validation served as a filter, the surviving variants made +0.142 R per trade in it and looked confirmed twice over, and in the following two years, which we used for nothing else, they made −0.024 R.

Real data: our own search of 1,344 variants

On real data we did the same on a small scale; chapter 7 has the details. The grid had 1,344 variants (6 instruments × 8 triggers × 2 directions × 7 filters × 2 exits); training covered 2018–2022 and validation 2023–2025. In training, 268 of the 1,273 variants with at least 100 trades had a positive result, or 21%. The best t-statistic per instrument ranged from 1.99 to 3.01, and the expected maximum of the 224 trials per instrument is about 2.8, roughly the level of chance alone. Of 46 candidates, the 13 best went into validation, and none of them passed. Their average fell from +0.201 R to +0.049 R per trade, a drop of about three quarters. What remained came largely from the bitcoin bull market: the bitcoin variant closest to passing had +0.246 R and a PF of 1.38 in validation, but in total made +248 R, less than simple “always buy” at the same moments (+261 R).

Why it doesn’t work

A generator does not show the bar. The winner’s result appears on the screen without any information on how many trials it came from, and with a hundred thousand trials a PF of 1.47 and a t of 3.14 are on average what a market with no edge at all delivers.

Tests on the same data mean more trials, not more evidence. Luck can be “robust” too, because it covers a whole group of similar variants. When a channel of length 36 makes money, channels of 30 and 40 usually do too, because they take almost the same trades: in the simulation, 98% of the variants that passed the earlier tests were also profitable for all their neighbours in the grid. A parameter “plateau” is a feature of the history on which it was found. An out-of-sample segment works only once: once it decides which strategies stay, it stops testing anything.

A small edge disappears under the bar. We are not saying that there is no real edge among a million combinations. We are saying that the result of the best one does not let you tell it apart from luck. The candidates from our real search kept +0.049 R per trade in validation. With 300 trades and a standard deviation of 1.5 R, that gives a t of about 0.6, while the bar for a million trials is almost 4.9.

The problem has its own literature: the “reality check” for data snooping (White, 2000), the Sharpe ratio adjusted for the number of trials (Bailey and López de Prado, 2014) and the paper on backtest overfitting (Bailey, Borwein, López de Prado and Zhu, 2014), whose authors note that almost nobody reports the number of trials. Harvey, Liu and Zhu (2016) calculated that after decades of searching the same data, a new factor in academic finance should need a t above about 3, not 2.

What we do instead

Our strategy tester, RBR Strategy Lab, is a program (EA) for the Strategy Tester of the MetaTrader 5 platform, built the opposite way to a generator:

1. One strategy, described before the result. You assemble it from building blocks: entry, filter, stop, target, time exit. Before the Lab calculates the first trade, it saves the strategy description with a checksum (hash) and a date. Changing a single parameter gives a new hash, that is, a new strategy.
2. Real ticks and real costs: bid and ask prices from ticks, commission and markup from your account history.
3. Validation once: the last 30% of the period. The Lab shows its result for a given hash on the first run, and after that only the saved one.
4. Variant counter and correction: every new hash on the same instrument, timeframe and period is one more trial. The report says which variant this is, by how much the result of the best is inflated on average at this number of trials (Blom’s approximation), and how much remains after the correction. The result after correction must be positive, otherwise the strategy gets neither the verdict “passes” nor “trace”.
5. A comparison with something, not with zero: with 1,000 series of random entries with the same stop and target geometry, calculated with the same cost, and with holding the market during the same hours.
6. A verdict based on criteria written into the code: passes, trace or does not pass, with a list of the criteria the strategy did not meet. The thresholds cannot be changed in the settings.

The Lab has no search, meaning no mass generation and no evolution of strategies. That is a decision, not something left over in the plan. For most strategies the Lab will say “does not pass”, and we consider that its value.

Walk-forward: tuning that can be counted. The full version of the Lab has walk-forward. You enter the parameter grid, for example a few channel lengths and stop widths, before the test, and it goes into the hash. The Lab picks parameters on consecutive training windows by a fixed rule and checks them on the next, unseen segments. Only the joined test segments are evaluated, by the same criteria as everything else, and the final validation stays untouched. Each grid combination counts as a variant, and the grid has an upper limit (up to 2 parameters and 50 combinations), so that walk-forward does not become search under another name. What gets evaluated is the tuning procedure, not one lucky set of parameters.

Honestly about the limits. Our tester will not stop anyone from searching by hand, variant after variant. We checked in the simulation how often the winner out of M trials would pass a simplified set of criteria similar to ours: in training at least 100 trades, PF ≥ 1.25 and t ≥ 1.645 (the equivalent of p < 0.05), and in validation a result above zero. We left out the years, the five best trades and the comparison with holding the market, so the figures below are an upper bound:

Without the correction, the winner out of 100 trials on a market with no edge passes this set in 21% of cases, and the winner out of a thousand or more in 37–40%, because for a strategy without an edge a validation result in profit is almost a coin toss. With the condition that the average after correction must be positive, 3.9–4.7% remain at 10 and 100 trials, roughly what a p < 0.05 threshold allows, and almost zero at a thousand or more. This table convinced us that the correction cannot be just a number next to the verdict. Today it is the Lab’s eighth criterion: “passes” or “trace” requires a positive average after correction, counted over all variants tested on the same instrument, timeframe and period. It does not protect against everything: the counter only sees trials made in the Lab in your terminal, and the same idea tested earlier elsewhere or on a different period starts counting from zero. So before the verdict, read the variant counter and remember the trials the Lab has not seen.

The lesson for you

If you search anyway, with a generator, your platform’s optimiser or by hand, count how many trials you have made.

1. Record every trial, including the rejected ones and the ones “only checked quickly”. In a generator, the number of evaluations is at least the population size times the number of generations, not the length of the results list.
2. Compare the winner’s t with the bar. t = average result / (standard deviation / √number of trades); a spreadsheet is enough. The bar is about 1.5 for 10 trials, 2.5 for 100, 3.85 for 10,000 and 4.9 for a million. A result below it is what chance alone delivers.
3. Set aside a piece of data that no phase will see, neither the search nor the robustness tests, and look at it once, for one strategy.
4. Don’t choose among the strategies that passed validation. If you did, validation has become training and you need a new one: more months going forward, or a demo account.
5. Compare with random entries and with holding the market, not with zero.
6. Expect shrinkage. In our real search the best variants lost about three quarters of their result in the following years, and in the simulation without an edge all of it.

In your trading journal, record how many versions of the strategy you tested before choosing the current one. The “Luck or edge?” calculator evaluates a single series of trades. If that series is the best of many, first compare its t with the bar.

Educational content, not investment advice. Past results do not guarantee future results. We describe the result of our simulation on artificial markets with no edge and of our search of a single family of rules, on our data and with our costs.