A practical guide to what backtests, paper/demo accounts and small live tests can and cannot tell you about a bot.
In plain English
This guide explains what a test can and cannot prove. Backtests, demo environments and small live tests answer different questions and should not be treated as interchangeable.
Why weak results are useful
A screen full of losing optimization candidates is not wasted work. It can reveal that the idea is fragile, the search space is wrong or the assumptions only worked on a narrow period.
Backtest: logic and historical behavior
A backtest is useful for checking strategy logic, parameter sensitivity and historical distributions. It usually cannot reproduce every detail of latency, partial fills, outages or broker-specific rejection rules.
Demo/paper: integration and workflow
Demo is where you prove authentication, symbol mapping, position checks, order lifecycle, restart behavior and logging. It is also the right place to intentionally trigger errors.
Live: execution reality
Only live trading can reveal the exact live environment, but that does not justify skipping earlier stages. If a live test is used, size should be constrained and the purpose should be defined before starting.
| Test | Good for | Weak at |
|---|---|---|
| Backtest | Historical logic, distributions | Real execution |
| Demo/paper | API integration, workflow | Exact live fills/behavior |
| Small live | Actual broker environment | Large statistical conclusions from few trades |
Keep a promotion gate
Define requirements before moving stages: no duplicate orders, stable restart behavior, known maximum drawdown in tests, enough observations, and no unresolved broker errors.
Before you rely on this in production
- Historical and demo tests are not confused with live proof.
- Failure paths are tested deliberately.
- Restart/recovery is included.
- Test logs are detailed enough to explain outcomes.