One of the biggest issues with system development is that many trading strategies do not hold up into the future. This could be due to several reasons:
- The system is not based on a valid premise
- Market conditions have changed in a dramatic way that invalids the theoretical premises on which the system was developed
- The system has not been developed and tested with a sound methodology. For instance, (a) lack of robustness in a system due to improper parameters, and (b) inconsistent rules and improper testing of the system using out-of-sample and in-sample data
There are several approaches and methodologies to asses robustness and increase likelihood of positive real life trading performance including:
- Peak avoidance
- Limited number of optimizable parameters
- 3D surface charting
- Monte Carlo analysis
- Data scrambling
- Walk forward & walk backward analysis
One of the most robust ways of testing the reliability of a trading system and making sure the program will have the highest likelihood of performing well in real live trading is to use walk forward optimization (WFO), a method first described in the book “Design, Testing, and Optimization of Trading Systems” by Roberto Pardo.
What is walk forward analysis?
Walk forward analysis is the process of optimizing a trading system using a limited set of parameters, and then testing the best optimized parameter set on out-of-sample data. This process is similar to how a trader would use an automated trading system in real live trading. The in-sample time window is shifted forward by the period covered by the out-of-sample test, and the process is repeated. At the end of the test, all of the recorded results are used to assess the trading strategy.
In other words, walk forward analysis does optimization on a training set; tests on a period after the set and then rolls it all forward and repeats the process. We have multiple out-of-sample periods and look at these results combined. Walk forward testing is a specific application of a technique known as Cross-validation. It means taking a segment of data to optimize a system, and another segment of data to validate. This gives a larger out-of-sample period and allows the system developer to see how stable the system is over time.
The picture below illustrates the walk forward analysis procedure. An optimization is performed over a longer period (the in-sample data), and then the optimized parameter set is tested over a subsequent shorter period (the out-of-sample data). The optimization and testing periods are shifted forward, and the process is repeated until a suitable sample size is achieved.
Runs in a Walk Forward Optimization
In order to demonstrate the concept we will perform in this article a walk forward optimization on a volatility breakout trading system (VBO). For the test we will use the German DAX futures, NinjaTrader, CQG historical 1-minute data, and we will assume 3 points of slippage for each R/T trade to cover trading frictions.
The process consists of three main steps:
- Define in-sample and out-of-sample periods
- Define a robust parameters area
- Execute the walk forward
Definition of the in-sample and out-of-sample periods
We will choose as in-sample 1/1/2001 to 12/31/2009 for system design and in-sample optimization and 1/1/2010 to 12/31/2012 as out-of-sample period to evaluate the in-sample optimization robustness and execute the walk forward. We will then use a 3:1 ratio for the WFO (walk forward optimization):
- Optimize 2007 to 2009 and verify performance out-of-sample in 2010
- Optimize 2008 to 2010 and verify performance out-of-sample in 2011
- Optimize 2009 to 2011 and verify performance out-of-sample in 2012
Define the robust parameters area in the in-sample period
In this section we will define the "robust area" of the system parameters. We will optimize only 3 system parameters:
- Lookback period of the fast average
- Lookback period of the slow average
- Volatility filter
Other system parameters that we will not optimize are:
- Start time: 09:00 (GMT+1)
- End time: 22:00 (GMT+1)
- Last trade: 18:00 (GMT+1)
- Stop risk: 2%
- Maximum trades per day: 3
- Exit on close: True
Optimization of the charting resolution
As we can see the average SQN (system quality number) tends to decrease as we increase the time period of the chart in minutes. We will choose 14 minutes for all simulations going forward.
Definition of the robust parameter area for the averages
Via a 3D surface chart we can identify the robust parameter area as follows:
- Slow average: 12 to 30 minutes
- Fast average: 330 to 500 minutes
We define "robust" a parameter surface area that does not have major peaks or valleys and has generally a good performance.
Definition of the robust parameter for the volatility filter
As we can see the robust area for the filter is between 0.55 and 0.70 when the SQN is slowly increasing.
Now that we have identified the robust parameters area it is worth performing a full in-sample optimization to see how the system would have performed between 2001 and 2009.
The system generated a net profit of $120,000 between 2001 and 2009 with a profit factor of 1.56, performing 756 trades with an average of 41% trades profitable. The systems exhibits certain desirable characteristics such as a high win/loss ratio of 2.28.
Walk forward optimization
As anticipated, we will now proceed with a walk-forward optimization.
Step 1: We'll optimize between 2007-2009 and find the best parameters.
The best parameters for the 2007-09 period are:
- Fast average: 12
- Slow average: 410
- Filter: 0.55
We apply these parameters to the out-of-sample period in 2010 with the following results:
Net profit: $12,300
DD: $9,000
% profitable: 43%
Profit factor: 1.38
Step 2: We'll optimize between 2008-2010 and find the best parameters.
The best parameters for the 2008-10 period are:
- Fast average: 20
- Slow average: 500
- Filter: 0.70
We apply these parameters to the out-of-sample period in 2011 with the following results:
Net profit: $27,900
DD: $7,450
% profitable: 43%
Profit factor: 1.61
Step 3: We'll optimize between 2009-2011 and find the best parameters.
The best parameters for the 2009-11 period are:
- Fast average: 20
- Slow average: 420
- Filter: 0.55
We apply these parameters to the out-of-sample period in 2012 with the following results:
Net profit: $17,540
DD: $7,300
% profitable: 41%
Profit factor: 1.58
Conclusions
In this article we have shown how to perform a walk forward optimization on an intraday mechanical system. The walk forward results are in line with the in-sample results and this builds confidence in the robustness of the strategy.
-- By Amon Licini has been a private trader for 15 years and a senior manager with various corporates in Italy. Amon's main trading interests lie in the area of volatility and open range breakouts for intraday systems. He lives in Milan with his wife and 2 children and loves traveling when he's not developing new systems. Amon has a degree in mechanical engineering from the Polytechnic University of Milan.