Using System Parameter Randomization To Estimate Future Returns

You just spent a ton of time creating a trading system and being very careful not to over-optimize. You then tested it on the out-of-sample data segment and the performance looks good. What’s next? Jump right into the live market? Maybe. But instead, you would like to perform one more test called System Parameter Randomization.

The article, System Parameter Permutation – a better alternative?, provided a unique way to estimate possible future returns of a trading system. This method is called System Parameter Randomization (SPR) but, there was no practical example within the article. I like practical examples and I know you probably do too. So, in this article I’m going to take an example trading model from System Trader Success, and follow the guidelines found in the SPR article and estimate our possible future returns. My hope is with this practical example on how to use SPR you just might have a new and valuable tool for your toolbox.

The System

I’m going to use a trading model inspired by Ben Little in the article, The Simplest System You’ll Ever Find for the S&P E-mini. Within this article Ben creates a simple mean reversion strategy for the S&P E-mini which can be traded on a 5-minute chart. Ben carefully tests various filters until he has something workable. The rules I decided upon for this trading model are below:

• Long trades only
• Buy only at the lowest low from yesterday
• The Close of yesterday must be lower than the Open (yesterday was down day)
• Yesterday’s range must be greater than 18 points (Yesterday was a large range day)
• The market must be trading below it’s 140 simple moving average (bearish regime filter)
• $5,000 stop loss
• $5,000 profit target
• Close all trades at end of day session

Please note this trading model may not be ready for the live market! It’s an example system and needs to be fully vetted by you to ensure the system performance metrics meet your requirements.

The Trading Environment

•  25,000 account size
• Trades 1 contract per trade
• Only 1 trade per day
• $20 deducted round trip for slippage and commissions
• Backtested dates are from 1997 through July 31, 2015.

Results of Backtest

With this information we generate the following results for the trading model.

SPR Baseline EQ Curve

Let’s say it’s August 1, 2015 and we are happy with our backtested results. In fact, let’s say the last two years of our results were actually executed over the out-of-sample data and we like what we see. We’re now ready to take our system live. But before we do that, we would like to estimate what our likely future returns might be. How are we going to do that? Enter SPR.

Estimating Future Performance With SPR

We know through experience that our system will not likely perform identically to our backtest. We expect our backtest is overly optimistic. When we selected our input values for our trading model we understand this will generate one possible outcome into the future. Furthermore, the likelihood that those values will work identically into the future is not high. This is why we strive to locate stable regions when we do our optimizations because this tends to make our system more robust to market changes. System traders fall into a trap when they backtest to find the best values. That’s not what you should do. You should pick reasonable values within a stable range.

So how does SPR work to estimate future returns?

In essence we’re going to estimate future performance by changing the input values and then recording the results. We will do this many, many times. The input values will actually be picked randomly over a predetermined range. By plugging in random values for our inputs we generate one possible future result of our system. Do this many times and we get a population of possible future outcomes. Once we have this population of possible future outcomes we can calculate the median net profit. This will be our estimate of future performance. Note, you could use any metric you like such as, CAGR, profit factor, or average profit per trade.

By changing the input values and generating hundreds or thousands of different results we are introducing change to the trading model and seeing how it reacts. I would say, we are stressing the trading model. During this stress test we want to know if changing one input parameter makes a significant or small difference to our bottom line. The more our system is able to “hold-up” or perform well over a wide range of values, the more likely it will produce a population of acceptable future outcomes. This population will skew our final median results in favor of a “winning” or acceptable model. If our trading model is negatively affected by changes in the input values we are likely to see a wider range of outcomes that may skew our results in favor of a “poor” or under performing trading model.

Using EasyLanguage To Perform SPR

Here are the steps I’ll use to execute an SPR test on the example system. I’ll be using EasyLanguage, TradeStation, and Excel to perform this test.

1) Determine your critical input parameters and reasonable ranges

2) For each input value generate a random value within the range

3) Apply this set of input values to the backtest and record the results

4) Repeat this test X number of times

5) Once complete, locate the median value of the metric you desire

6) This median value is the estimated future value

Pick Input Values And Ranges

It’s important to use all the inputs when you’re performing the SPR test. In our example, our trading model is not complete. It’s a work in progress and really only two inputs are being used. The following input values and ranges we be used for our SPR test.

• Our bull/bear regime filter ( 100-200 )
• Yesterday’s range must be greater than X points (2-30)

The ranges I selected are “reasonable”. What does “reasonable” mean? Good question. Nothing in the trading world is exact so, I can only recommend to pick values that “make sense” in the context of the trading model. Only you will know what that is. It’s probably better to error on the side of providing too wide of range as apposed to a narrow range.

Update The code With Random Number Generator

We’ll write some EasyLanguage code to generate random values for each of our input values. We’ll put these within the “Once” block so the random numbers will only be generated at the start of the run. We don’t want the values changing in the middle of a backtest!

TradeStation’s “Random” number function will generate a number between zero and the value you pick. For example, if you use “Random(28)” this will generate a range of integers between 0 and 28. Well, in our case for the BB variable we don’t want a value of zero or one. We want a range between 2 and 30. To do this, we simply have TradeStation generate values between 0 and 28 and then add two. Thus, we generate a range from 2-30.

While we’re editing the code we can also comment out the “BB” and “TrendfilterPeriod” as input values. Why? These are no longer input values but variables which will be randomly chosen. So, let’s remove them from the input section and place them into the “Variables” section of the code. In their place, we’ll need to add a new input value called “Runs”. This will be an input value which we’ll optimize over using TradeStation’s optimization feature. Here is what the inputs look like now:

Update Code to Store System Results

OK, now we need a way to store the results of a run to a file. This can easily be done by adding the following code to the bottom of our strategy.

Execute!

We now use TradeStation’s optimization feature to iterate over the strategy 1,000 times. Each time the system evaluates the trading model over the backtest, it will be using random values for the parameters we defined. Once the run has completed over the backtest, the results are written to a file and the next iteration starts with a new set of random input values.

Below is an image of the output of our runs. You can see that each row represents a particular run. Each column represents:

1. The run number
2. The net profit generated during the test
3. Our bull/bear regime filter look-back period (trendFilterPeriod)
4. Points required to be considered a large range day (BB)

I did not need to include the randomly picked values used during each test. However, I thought it would be interesting to see and would also provide a way to verify the random numbers being picked are correct.

Reviewing The Results: Our Median Return

The results can now be loaded into Excel and the median value calculated. In our case, we determine that our median value is $35,058. This value is positive. Great! But we can go further.

Let’s find the standard deviation and compute a two standard deviation band around our average value. Using Excel we find that the standard deviation is $3,538. Thus, two standard deviations is $7,076.

Our upper band (+2 standard deviations form median) is $42,134 while our lower band (-2 standard deviations from median) is $27,982. We can use the lower band as an estimate on how our system might do into the future assuming poor performance. I’ll call this our pessimistic estimate of future profit. In this case, even our poor performance estimate produces a positive result of $27,982 in net profit.

The key takeaways are:

• Our median profit is positive
• Our pessimistic profit is also positive

Based on these numbers it appears our system passes. We randomly changed the input parameters many times over and we see our median net profit over all those different strategy variations remains positive. In fact, even two standard deviations away from the mean remains positive.

Conclusion

So there you have it – an example on how to use SPR to estimate future performance of your system. You can use this technique to estimate potential drawdown, annual returns, or whatever metric you wish. With a little coding and working the numbers in Excel, you just might have another tool for your toolbox to test your system before putting your money on the line.

Video Demo of Using SPR