What is backtesting, and how does survivorship bias distort it?
By Ghost Man Trades · Last reviewed · Education only, not advice
Backtesting in plain English
A backtest asks: "If I had followed these exact rules in the past, what would have happened?" You write down the rules precisely (when to buy, when to sell, how much), run them over years of historical prices, and count the results: number of trades, win rate, average gain and loss, worst drawdown.
Done carefully, it is the best way to find out whether an idea had any edge at all before risking money. Done carelessly, it produces beautiful numbers that fall apart in real trading. The most common reason is not bad math. It is bad data.
Survivorship bias
Suppose you backtest a strategy on the stocks in an index today, going back 20 years. Every one of those stocks survived 20 years, and many of them grew enough to join the index. The companies that went bankrupt, got delisted or shrank out of the index are missing. Your test only ever "bought" winners, because the losers were removed from the list before you started.
That is survivorship bias, and it is well documented. Researchers studying mutual funds found that databases that drop dead funds overstate average performance, because poorly performing funds are the ones that tend to close.
Example (hypothetical numbers).
- In 2006, 100 stocks meet your rule. Over the next 20 years, 80 survive and return an average of +150%. 20 fail or get delisted, losing an average of 80%.
- A backtest using only today's survivors sees 80 stocks and reports an average of +150%.
- The real average across all 100 stocks: (80 × 150% + 20 × −80%) ÷ 100 = (12,000 − 1,600) ÷ 100 = +104%.
- Same rule, same years. The survivor-only test overstated the result by 46 percentage points, and it never showed you a single wipeout.
Other backtest traps
- Look-ahead bias: using information that wasn't known yet, like buying at today's close on a signal that only exists after the close.
- Overfitting: adjusting settings until the past looks perfect. More rules and more tuning usually mean worse real results.
- Ignoring costs: commissions, bid-ask spreads, slippage and taxes all reduce real returns.
- Too few trades: ten trades can look great by luck.
- One market regime: a test that only covers a long bull market says little about a bear market.
Common mistakes when reading someone else's backtest
- Not asking which stocks were included. If the answer is "today's index," assume survivorship bias.
- Looking only at win rate. Check the size of losses and the worst trade too. See win rate vs expectancy.
- Treating hypothetical returns as a promise. In the U.S., registered investment advisers face specific SEC requirements before they can advertise hypothetical performance, for good reason.
How Ghost Man uses this
Every signal record on this site is a backtest of the Ghost Man rules on one stock and one chart: buy at the next open after a buy signal, sell at the next open after a take-profit signal, long only, no stop loss. Trades are filled at the next open, not at the signal's close, to avoid look-ahead bias. We list every trade so anyone can check it.
Here is the limitation we want you to know: the records are calculated on stocks that are listed and trading today. Companies that went bankrupt or were delisted are not in our universe, so our records carry survivorship bias, and the true historical results of the same rules across all stocks would likely be worse. The records also leave out commissions, spreads and taxes, and the open trade is shown separately rather than counted. Treat them as evidence about how the rules have behaved, not as a forecast.
Related terms
- Win rate vs expectancy · What is the difference between win rate and expectancy in trading?
- Position sizing · What is position sizing, and how much should you put in one stock?
- Stop loss vs time stop · What is a stop-loss order, and what is a time stop?
- 200-week moving average · What is the 200-week moving average (the 4-year average)?
- RSI (Relative Strength Index) · What is RSI (the Relative Strength Index)?
- Dividend growth and Dividend Aristocrats · What are dividend growth stocks and the Dividend Aristocrats?
- Fibonacci retracement · What are Fibonacci retracement levels?
- Max drawdown · What is maximum drawdown?
- Value trap · What is a value trap?
Learn it step by step in Swing School
Frequently asked questions
Is a backtest the same as real trading results?
No. A backtest is hypothetical: it shows what rules would have done on past data, with no real money, emotions or execution problems. Real results usually come in worse.
How do you avoid survivorship bias?
Test on a historical universe that includes companies that were later delisted, acquired or went bankrupt, using point-in-time index memberships and prices. This data costs more and is harder to get, which is why many free backtests skip it.
What is look-ahead bias?
Using information in a test that was not available at the time of the trade, for example today's index members, later-revised earnings or a closing price before the close happened.
What is overfitting?
Tuning a strategy's settings until it matches the past almost perfectly. An overfit strategy has learned the noise in old data and usually performs much worse on new data.
How many trades does a backtest need?
There is no magic number, but a handful of trades tells you very little. Results based on dozens or hundreds of trades across different market conditions are more meaningful.
Sources
We check every definition and formula against these sources. Our own rules come from the Ghost Man scan code that runs this site.
- Elton, Gruber and Blake, Survivorship Bias and Mutual Fund Performance, Review of Financial Studies 9(4), 1996
- Carhart, Carpenter, Lynch and Musto, Mutual Fund Survivorship (Review of Financial Studies, 2002), working paper
- SEC: Investment Adviser Marketing compliance guide (hypothetical performance)