Strategy Robustness Analyzer Help
Complete Guide to Understanding Your Strategy Analysis
Overview
The Strategy Robustness Analyzer helps you answer the most important question about any trading strategy: "Is this edge real, or did I just curve-fit the past?"
A profitable backtest doesn't guarantee future profits. This tool goes beyond basic metrics to stress-test your strategy and identify warning signs of overfitting, luck, or unstable performance.
What Makes a Strategy "Robust"?
A robust strategy has these characteristics:
- Consistent performance — Works across different time periods, not just historically
- Statistical significance — Results are unlikely to be random chance
- Sequence independence — Doesn't depend on trades happening in a specific order
- Linear equity growth — Steady accumulation, not lucky streaks
- Quality entries — Entries provide edge over random timing
The Six Analysis Modules
| Tab | What It Answers | Data Required |
|---|---|---|
| Summary | What are the basic performance metrics? | Basic trade export |
| Stability | Is performance consistent over time? | Trades with dates |
| Monte Carlo | How much does luck affect outcomes? | Basic trade export |
| Edge Detection | Is the edge statistically significant? | Basic trade export |
| Trade Quality | How efficient are entries and exits? | Full NT8 export (MAE/MFE) |
| E-Ratio | Are entries better than random? | Price data + full export |
Getting Started
Exporting Trades from NinjaTrader 8
For the most complete analysis, export your trades with all available columns:
- Open the Trade Performance window in NinjaTrader
- Right-click on the trades list
- Select Export → Trades
- Save as CSV file
Supported File Formats
- NinjaTrader 8 — Full support including MAE/MFE
- Basic CSV — Any file with Date and Profit columns
- StrategyQuant — Exports from SQX
How Many Trades Do I Need?
| Trade Count | Analysis Quality | Notes |
|---|---|---|
| < 30 | Insufficient | Results are statistically meaningless |
| 30 - 100 | Preliminary | Useful for initial screening only |
| 100 - 300 | Adequate | Reasonable confidence in results |
| 300 - 1000 | Good | Statistical tests become reliable |
| > 1000 | Excellent | High confidence in all metrics |
Data Formats
The analyzer accepts CSV files with trade data. The more complete your export, the more analysis features become available.
Minimum Required Columns
At minimum, your CSV needs these columns (column names are flexible):
| Data | Accepted Column Names | Example |
|---|---|---|
| Profit/Loss | profit, pnl, p&l, net profit, trade p&l | 125.50 or $125.50 or -$50.00 |
Recommended Columns
These columns enable time-based analysis (Stability tab, monthly breakdown):
| Data | Accepted Column Names | Example |
|---|---|---|
| Entry Date/Time | entry time, entry date, open time, date | 2024-03-15 09:35:00 |
| Exit Date/Time | exit time, exit date, close time | 2024-03-15 10:22:00 |
| Direction | market pos., direction, side, type | Long or Short |
Full Export Columns (Trade Quality Tab)
To unlock MAE/MFE analysis, include these columns from NinjaTrader's full export:
| Data | Accepted Column Names | What It Measures |
|---|---|---|
| MAE | mae, max adverse, max. adverse excursion | Maximum loss during the trade |
| MFE | mfe, max favorable, max. favorable excursion | Maximum profit available during the trade |
| Entry Price | entry price, open price, avg. entry price | Price at entry |
| Exit Price | exit price, close price, avg. exit price | Price at exit |
NinjaTrader 8 Export
To get the most complete export from NinjaTrader 8:
- Open Control Center → New → Trade Performance
- Select your strategy/account and date range
- Click the Trades tab to see individual trades
- Right-click anywhere in the trade list
- Select Export → All Trades
- Save as CSV file
Example CSV Format
Price Data for E-Ratio (Optional)
The E-Ratio analysis requires separate OHLC price bar data. Export this from NinjaTrader or your charting platform:
| Column | Required | Example |
|---|---|---|
| Date/Time | Yes | 2024-01-15 09:30:00 |
| Open | Yes | 4850.25 |
| High | Yes | 4852.50 |
| Low | Yes | 4849.00 |
| Close | Yes | 4851.75 |
StrategyQuant X Export
Exports from StrategyQuant X are automatically detected. Use the "Export trades to CSV" option from the results panel.
Other Platforms
Most trading platforms can export trade history to CSV. As long as you have a Profit column (and ideally date/time), the analyzer will attempt to parse it. If your format isn't recognized, try renaming columns to match the accepted names above.
Summary Tab
The Summary tab shows the fundamental metrics every trader should understand. While these alone don't prove robustness, they establish baseline performance.
Win Rate
The percentage of trades that were profitable.
Context matters: A 35% win rate is fine if average winners are 3x larger than average losers. A 70% win rate is dangerous if losers are 5x larger than winners.
Profit Factor
The ratio of gross profits to gross losses. How many dollars you win for every dollar you lose.
Beware: Profit factors above 3.0 in backtests are often a sign of curve-fitting and rarely persist in live trading.
Expectancy
The average amount you expect to win (or lose) per trade. This is your "edge" in dollar terms.
A positive expectancy means the strategy makes money over time. The higher, the better — but sustainability matters more than magnitude.
R² (Equity Curve Linearity)
Measures how closely your equity curve follows a straight upward line. R² of 1.0 = perfect straight line.
Why it matters: An R² of 0.50 means half your profits came from lucky streaks rather than consistent edge. Low R² suggests the strategy may only work in certain market conditions.
SQN (System Quality Number)
Developed by Van Tharp, SQN measures edge quality adjusted for trade count. It answers: "How good is this edge relative to its variability?"
Note: SQN above 7.0 is suspicious and often indicates curve-fitting.
Maximum Drawdown
The largest peak-to-trough decline in equity during the backtest period.
Rule of thumb: Expect live drawdowns to be 1.5x to 2x worse than backtested drawdowns. If you can't stomach a $10,000 drawdown, don't trade a strategy with a $5,000 backtested drawdown.
Stability Tab
The Stability tab is your curve-fitting detector. It analyzes whether your strategy's edge is consistent across time periods, or if the "good" results only appear in certain portions of the data.
Why This Matters
A curve-fit strategy is optimized to fit historical data perfectly — but the "edge" doesn't exist in reality. Classic signs of curve-fitting include:
- Profits heavily concentrated in one portion of the backtest
- Dramatic improvement in metrics from first half to second half
- Equity curve that looks random in one half but linear in the other
- R² that jumps from near-zero to excellent
Stability Score (0-100)
The stability score penalizes strategies where key metrics vary significantly between time periods:
Curve-Fit Detection
The analyzer checks for 5 specific warning signals that commonly indicate curve-fitting:
| Signal | What It Detects | Why It's a Problem |
|---|---|---|
| Profit Concentration | 80%+ of profits came from the second half | Suggests optimization on recent data |
| First Half Near-Random | First half R² < 0.3 (random walk) | No edge existed before optimization period |
| First Half Barely Profitable | First half profit factor < 1.1 | Strategy didn't work before recent tweaks |
| Unrealistic Expectancy Jump | Expectancy improved 200%+ from first to second half | Too-good-to-be-true improvement |
| R² Flip | R² went from < 0.5 to > 0.8 | Equity curve only became "tradeable" after optimization |
Curve-Fit Assessment Levels
First Half vs Second Half Table
This table compares key metrics between the two halves. The "Change" column shows the percentage difference:
| Change Amount | Interpretation |
|---|---|
| ±15% or less | Normal variance — healthy consistency |
| ±15% to 30% | Noticeable drift — warrants attention |
| ±30% or more | Significant instability — red flag |
Important: A strategy that only works in the second half is equally suspect as one that degrades. Both patterns suggest the strategy may have been optimized on a specific portion of the data.
Quarters Breakdown
When you have enough trades (400+), the analyzer breaks performance into four quarters to provide finer-grained analysis:
Profit Distribution Bar
A visual representation showing what percentage of total profits came from each quarter. Healthy strategies show relatively balanced contributions across all quarters.
Back-Loaded Profits Warning
Triggers when 75%+ of profits came from the last two quarters. This pattern often indicates the strategy was optimized on recent data and may not perform as well going forward.
Detailed Quarter Metrics
Click "Show detailed quarter metrics" to see Win Rate, Profit Factor, Expectancy, R², and P&L for each quarter. Look for:
- Consistent metrics across quarters — Edge is stable
- R² that jumps dramatically — Curve-fit signal
- PF below 1.0 in early quarters — Strategy didn't work originally
Rolling Window Chart
This chart shows how profit factor and expectancy change over time using a rolling window (typically 50-100 trades). Look for:
- Stable lines — Metrics stay within a reasonable band
- Wild swings — Strategy works in streaks (regime dependent)
- Downward trend — Edge is decaying over time
- Dramatic upward trend — Only recent period is profitable
Monthly Breakdown
Shows performance by calendar month. Useful for identifying:
- Seasonal patterns (e.g., strategy only works in Q1)
- Sensitivity to volatility regimes
- Whether a few outlier months drive all the profits
Monte Carlo Tab
Monte Carlo simulation answers: "How much of my results are due to the specific order trades happened?"
How It Works
- Take your exact list of trades (same wins and losses)
- Randomly shuffle the order thousands of times
- Track the equity curve for each shuffled sequence
- Analyze the distribution of outcomes
This reveals the range of possible outcomes you could have experienced with the same trades in a different order.
Sample Equity Paths Chart
- Red line: Your actual backtest (the order trades really happened)
- Blue lines: Random shuffles of the same trades
What to look for:
- Tight bundle: Outcome doesn't depend much on trade order — robust
- Wide spread: Luck of the sequence matters a lot — risky
Final Equity Histogram
Shows the distribution of ending equity values across all simulations:
- Left tail: Worst-case outcomes (unlucky sequences)
- Center: Most likely outcomes
- Right tail: Best-case outcomes (lucky sequences)
Key Statistics
Probability of Profit
What percentage of simulations ended profitable?
If less than 80% of shuffles end profitable, the strategy's success may depend on lucky trade ordering.
5th Percentile Equity
In the worst 5% of scenarios, what's the minimum equity you'd have?
This is your "bad luck" baseline. If the 5th percentile is negative, there's meaningful risk the strategy could lose money even with a positive backtest.
95th Percentile Drawdown
The drawdown you'd experience in the worst 5% of sequences.
Important: This is often 1.5x to 2x worse than your actual backtest drawdown. Plan your position sizing for this number, not the backtest number.
Advanced Options
- Random Skip %: Simulates missing trades (you were away, system error, etc.)
- P&L Variance %: Adds random variance to trade results (simulates slippage, different fills)
Edge Detection Tab
Edge Detection uses statistical hypothesis testing to answer: "Could these results have happened by random chance?"
Understanding p-Values
A p-value measures the probability of seeing results this good (or better) if the strategy had no real edge.
| p-Value | Interpretation | Confidence |
|---|---|---|
| < 0.01 | Very strong evidence of real edge | 99%+ confidence |
| 0.01 - 0.05 | Good evidence of edge | 95-99% confidence |
| 0.05 - 0.10 | Weak evidence — more data needed | 90-95% confidence |
| > 0.10 | Results could easily be random | < 90% confidence |
Expectancy t-Test
Tests whether the average trade profit is significantly different from zero.
- Null hypothesis: True expectancy = $0 (no edge)
- If p < 0.05: Expectancy is statistically significant
Win Rate Binomial Test
Tests whether the win rate is significantly different from 50% (random coin flip).
- Null hypothesis: True win rate = 50%
- If p < 0.05: Win rate is statistically significant
Bootstrap Confidence Intervals
Bootstrap resampling creates thousands of "simulated" trade histories by randomly sampling (with replacement) from your actual trades. This generates confidence intervals for metrics.
95% Confidence Interval for Expectancy
If the lower bound is positive, you can be 95% confident the true expectancy is above zero.
If the lower bound is negative, there's meaningful risk the strategy has no real edge.
Sample Size Guidance
The analyzer estimates how many trades you need for various confidence levels. Use this to understand whether your current results are reliable.
Trade Quality Tab
Trade Quality analysis examines what happens during each trade, not just the final P&L. This requires NinjaTrader exports with MAE/MFE columns.
MAE and MFE Explained
MAE (Maximum Adverse Excursion)
The worst point each trade reached before closing. How much "heat" you took.
Example: You buy at $100, price drops to $95, then recovers and you sell at $103. Your MAE is $5 (the lowest point).
MFE (Maximum Favorable Excursion)
The best point each trade reached before closing. How much profit was "available."
Example: You buy at $100, price rises to $110, then drops and you sell at $106. Your MFE is $10 (the highest point).
Trade Efficiency
How much of the available profit (MFE) did you actually capture?
Low efficiency means you're leaving money on the table — consider adjusting your exit strategy.
Edge Ratio (MFE/MAE)
The ratio of favorable movement to adverse movement.
Higher is better — it means trades tend to move in your favor more than against you.
MAE vs Profit Scatter Plot
Each dot represents a trade:
- X-axis: How much heat the trade took (MAE)
- Y-axis: Final profit
- Green dots: Winners
- Red dots: Losers
Ideal pattern: Winners clustered in upper-left (low heat, good profit). Losers clustered in lower-left (small losses, stopped out quickly).
Problem pattern: Dots spread across the entire chart, or winners requiring high MAE to become profitable (you're getting lucky on trades that nearly stopped out).
Stop Loss Optimization Table
This table simulates different stop loss levels based on MAE multiples and shows the trade-off:
- Losers Caught: What % of losing trades would have been stopped out early?
- Winners Lost: What % of winning trades would have been stopped out?
- Net Effect: Overall impact on profitability
Look for the "sweet spot" where you catch most losers but preserve most winners. A positive net effect means a tighter stop would improve results.
E-Ratio Tab
E-Ratio analysis answers the humbling question: "Are my entries actually better than random?"
This requires uploading price bar data (OHLC) in addition to trade data.
What is E-Ratio?
E-Ratio measures the ratio of favorable price movement to adverse price movement after your entry signals, normalized by ATR (Average True Range).
An E-Ratio above 1.0 means prices tend to move more favorably than adversely after your entries.
The Chart
- Blue line (Strategy): E-Ratio of your actual entries
- Red line (Random): E-Ratio of random entries in the same market
- Gray line: Break-even threshold (1.0)
Interpreting Results
| Condition | Meaning |
|---|---|
| Strategy > 1.0 AND Strategy > Random | Your entries provide real edge over both break-even and random timing |
| Strategy > 1.0 BUT Strategy � Random | You're capturing natural market drift, not timing skill |
| Strategy < 1.0 | Entries are worse than break-even — your signal timing hurts performance |
| Strategy < Random | You'd be better off entering randomly |
Peak E-Ratio Bar
The bar number where your E-Ratio is highest shows when your entry edge is strongest. This can help optimize hold times:
- Peak at bar 1-3: Scalping edge — entries work quickly
- Peak at bar 10+: Swing trading edge — needs time to develop
Edge Decay
Most entry edges decay over time. The "Edge Decay Bar" shows where your E-Ratio drops below 1.0. Holding past this point erodes your edge.
Verdict Scoring
The Tradability Verdict synthesizes all analysis modules into a single 0-100 score and recommendation.
Score Ranges
| Score | Verdict | Recommendation |
|---|---|---|
| 75-100 | TRADABLE | Strategy shows robust, statistically significant edge. Suitable for live trading with proper risk management. |
| 60-74 | TRADABLE WITH CAUTION | Good potential with minor concerns. Start with reduced size and monitor closely. |
| 40-59 | CAUTION | Mixed signals. Paper trade longer or gather more data before committing capital. |
| 0-39 | NOT RECOMMENDED | Significant issues detected. Do not trade live until problems are resolved. |
How the Score is Calculated
The score starts at a baseline and adjusts based on findings:
Positive Factors (add points)
- Statistical significance (p < 0.05) for expectancy
- High R² (> 0.85)
- High stability score (> 70)
- Monte Carlo probability of profit > 90%
- Bootstrap CI lower bound > 0
- Trade quality metrics (if available)
- E-Ratio above random (if available)
Negative Factors (subtract points)
- Expectancy not significant (p > 0.10)
- Low R² (< 0.70)
- Stability score below 50
- Second half expectancy much worse than first half
- Monte Carlo probability of profit < 70%
- Small sample size (< 100 trades)
- Bootstrap CI includes negative values
Strengths & Concerns Lists
Below the score, you'll see itemized lists of:
- Strengths: What's working well
- Concerns: Significant issues that need attention
- Watch Items: Minor issues to monitor
Use these lists to understand why the strategy scored as it did and what to work on.
Red Flags Checklist
These warning signs suggest a strategy may not perform as expected in live trading:
Critical Red Flags (Don't Trade)
- Expectancy drops > 50% from first half to second half
- p-value > 0.10 (results not statistically significant)
- Monte Carlo probability of profit < 60%
- R² < 0.50 (equity curve is random walk)
- Profit factor > 4.0 (almost certainly curve-fit)
- SQN > 7.0 (too good to be true)
Caution Flags (Investigate Further)
- Sample size < 100 trades
- Stability score < 50
- Second half metrics worse than first half
- Bootstrap CI lower bound near zero
- E-Ratio near or below random baseline
- Few big winners drive most of the profits
- Rolling metrics show high variance
Common Curve-Fitting Patterns
- Over-optimization: Too many parameters tuned to historical data
- Look-ahead bias: Using information that wouldn't be available in real-time
- Selection bias: Only showing the backtest that worked
- Short test period: Strategy only works in specific market regime
- Excessive filters: Many conditions that reduce trades to only the "good" ones
Questions to Ask Yourself
- Would I have created this strategy before seeing this data?
- Does the strategy logic make sense, or is it arbitrary rules that happen to work?
- How many strategies did I test before finding this one?
- Would this work in a different market or time period?
- Am I emotionally attached to wanting this to work?
Glossary
- ATR (Average True Range)
- A measure of market volatility calculated from the average of true ranges over a period (typically 14 bars).
- Bootstrap
- A statistical method that resamples your data to estimate confidence intervals and uncertainty.
- Confidence Interval
- A range of values that likely contains the true parameter value. A 95% CI means we're 95% confident the true value falls within this range.
- Curve-Fitting
- Over-optimizing a strategy to fit historical data, resulting in poor forward performance.
- Drawdown
- The decline from a peak in equity to a subsequent low point before a new high is reached.
- E-Ratio
- Edge Ratio — the ratio of favorable to adverse price movement, normalized by ATR, measuring entry quality.
- Expectancy
- The average profit per trade, accounting for both win rate and win/loss size.
- MAE (Maximum Adverse Excursion)
- The maximum loss experienced during a trade before it was closed.
- MFE (Maximum Favorable Excursion)
- The maximum profit available during a trade before it was closed.
- Monte Carlo Simulation
- A technique that uses random sampling to understand the range of possible outcomes.
- p-Value
- The probability of seeing results this extreme if the null hypothesis (no edge) were true. Lower is better.
- Profit Factor
- Gross profits divided by gross losses. Values above 1.0 indicate profitability.
- R² (R-Squared)
- A measure of how closely data fits a regression line. For equity curves, it measures linearity of growth.
- Robustness
- A strategy's ability to perform consistently across different conditions, time periods, and trade sequences.
- SQN (System Quality Number)
- A metric developed by Van Tharp measuring the quality of a trading system based on expectancy and variability.
- Statistical Significance
- Results that are unlikely to have occurred by random chance, typically p < 0.05.
- t-Test
- A statistical test used to determine if a sample mean is significantly different from a hypothesized value (e.g., zero).
- Win Rate
- The percentage of trades that are profitable.
Strategy Robustness Analyzer — futures.aeromir.com
Questions? Contact [email protected]