One of the biggest issues with system development is that many trading strategies do not hold up into the future. This could be due to several reasons:

• The system is not based on a valid premise
• Market conditions have changed in a dramatic way that invalids the theoretical premises on which the system was developed
• The system has not been developed and tested with a sound methodology. For instance, (a) lack of robustness in a system due to improper parameters, and (b) inconsistent rules and improper testing of the system using out-of-sample and in-sample data

There are several approaches and methodologies to asses robustness and increase likelihood of positive real life trading performance including:

• Peak avoidance
• Limited number of optimizable parameters
• 3D surface charting
• Monte Carlo analysis
• Data scrambling
• Walk forward & walk backward analysis

One of the most robust ways of testing the reliability of a trading system and making sure the program will have the highest likelihood of performing well in real live trading is to use walk forward optimization (WFO), a method first described in the book “Design, Testing, and Optimization of Trading Systems” by Roberto Pardo.

What is walk forward analysis?

Walk forward analysis is the process of optimizing a trading system using a limited set of parameters, and then testing the best optimized parameter set on out-of-sample data. This process is similar to how a trader would use an automated trading system in real live trading. The in-sample time window is shifted forward by the period covered by the out-of-sample test, and the process is repeated. At the end of the test, all of the recorded results are used to assess the trading strategy.

In other words, walk forward analysis does optimization on a training set; tests on a period after the set and then rolls it all forward and repeats the process. We have multiple out-of-sample periods and look at these results combined. Walk forward testing is a specific application of a technique known as Cross-validation. It means taking a segment of data to optimize a system, and another segment of data to validate. This gives a larger out-of-sample period and allows the system developer to see how stable the system is over time.

The picture below illustrates the walk forward analysis procedure. An optimization is performed over a longer period (the in-sample data), and then the optimized parameter set is tested over a subsequent shorter period (the out-of-sample data). The optimization and testing periods are shifted forward, and the process is repeated until a suitable sample size is achieved.

Runs in a Walk Forward Optimization

"I’ve never seen a bad backtest” -- Dimitris Melas, head of research at MSCI.

A backtest is a simulation of a trading strategy used to evaluate how effective the strategy might have been if it were traded historically. Backtesting is used by hedge funds and other researchers to test strategies before real capital is applied. Backtests testing are valuable because they enable quants to quickly test and reject trading strategy ideas.

All too often strategies look great in simulation but fail to live up to their promise in live trading. There are a number of reasons for these failures, some of which are beyond the control of a quant developer. But other failures are caused by common, insidious mistakes.

An over optimistic backtest can cause a lot of pain. I’d like to help you avoid that pain by sharing 9 of the most common pitfalls in trading strategy development and testing that can result in overly optimistic backtests:

1. In-sample backtesting

Many strategies require refinement, or model training of some sort. As one example, a regression-based model that seeks to predict future prices might use recent data to build the model. It is perfectly fine to build a model in that manner, but it is not OK to test the model over that same time period. Such models are doomed to succeed.

Don’t trust them.

Solution: Best practices are to build procedures to prevent testing over the same data you train over. As a simple example you might use data from 2007 to train your model, but test over 2008-forward.

By the way, even though it could be called “out-of-sample” testing it is not a good practice to train over later data, say 2014, then test over earlier data, say 2008-2013. This may permit various forms of lookahead bias.

2. Using survivor-biased data

Suppose I told you I have created a fantastic new blood pressure medicine, and that I had tested it using the following protocol:

a. Randomly select 500 subjects
b. Administer my drug to them every day for 5 years
c. Measure their blood pressure each day

At the beginning of the study the average blood pressure of the participants was 160/110, at the end of the study the average BP was 120/80 (significantly lower and better).

Those look like great results, no? What if I told you that 58 of the subjects died during the study? Maybe it was the ones with the high blood pressure that died! This is clearly not an accurate study because it focused on the statistics of survivors at the end of the study.

This same sort of bias is present in backtests that use later lists of stocks (perhaps members of the S&P 500) as the basis for historical evaluations over earlier periods. A common example is to use the current S&P 500 as the universe of stocks for testing a strategy.

Why is this bad? See the two figures below for illustrative examples.

The Anatomy of a Stop

A stop can be defined by specifying four basic parameters, illustrated in Figure 1 below:

Price Reference: The price from which the stop is offset (white line) to create a stop value.

Stop Offset: The distance from the Price Reference to the stop value.

Price Trigger: The value of price that will trigger the stop. Usually this is the low or the high of a price bar. However, more sophisticated stops can be created by setting the trigger to a custom function of price, such as Average(Low, 3) to make the stop less vulnerable to isolated tall tails.

Reset Padding: The distance the stop should be reset away from the Price Trigger when the stop has been hit. Resetting the stop level serves two purposes: (1) it is much easier to see when a stop is hit, especially when the price trigger comes very close to but not quite touches the stop level, and (2) if a trade is re-entered in the same direction after stopping out, this reset padding can serve as the initial stop loss.

With these parameters defined, the following stop values can be calculated:

Stop Value: The value of the stop (yellow line). The stop is hit when the Price Trigger touches or crosses the Stop Value.

Ratchet Stop Value: The value of the ratchet stop (red line) is the Stop Value when it is constrained to move only in the direction toward the price action. The ratchet stop is hit when the Price Trigger touches or crosses the Ratchet Stop Value.

Fig. 1. Chandelier 5% Ratchet Trailing Stop (red line).

Figure 2. 6 StdErr Trailing Stop vs 1.5 ATR Trailing Stop.

HighRef is the Price Reference for the high stop (positioned above the price). LowRef is the Price Reference for the low stop (positioned below the price). Length is used to delay changes in the price reference by “Length” bars by adjusting the HighRef and LowRef values to MaxList(high, Length) and MinList(low, Length), respectively. UseKAMA is used to apply a KAMA (Kaufman Adaptive Moving Average) filter to HighRef and LowRef values. Default value is false.

EffRatioLength, FastAvgLength, and SlowAvgLength are all parameters that determine the degree of KAMA filtering. StopOffset is the offset to the bar used to calculate the stop for the current bar. Usually should be set = 1 (since strategy orders use the format: Buy/Sell NEXT bar at Value Limit). TradeDir indicates if stops will be displayed below the price (+1), above the price (-1), or both (0). Similarly, AlertDir indicates if the corresponding low (+1), high (-1), or both (0) stops will generate an alert.

I am frequently asked if I do out-of-sample testing. The short answer is not always, and when I do it is not how most people do the test. There are lots of considerations and pitfalls to avoid when doing out-of-sample testing. Out-of-sample testing is not the panacea it is made out to be. There are lots of grey areas which I will discuss below.

Definition

To do in-sample (IS) and out-of-sample (OOS) testing, one first divides their historical data into two parts. The most common methods for dividing the data are 50% IS/50% OOS and 67% IS/33% OOS. I will be using 15 years of data. Here are some ways that one can divide the data:

Today’s topic is on optimization. Some system developers don’t believe in optimization. However, I’m going to show you why you should really learn how to optimize, and second I’m going to demonstrate that even if you don’t believe in optimization then “not optimizing” is not the solution. “Not optimizing” is not the opposite of optimization!

First, we’re going to assume you want supernormal returns. Let’s just think about what it means for both the strategy and about the markets. If a strategy can deliver supernormal returns without optimization then it suggests that there is robustness to the strategy. That’s a good thing! But what does it say about the market? It suggests the market is not very efficient. We know that’s not the typical case.

The “opposite of optimization” is not “not optimizing”! Merely looking at a single performance report won’t tell you about how robust a strategy is or what the typical return is likely to be. What you are really doing when you “don’t optimize” is making an arbitrary decision, or a random decision. You are picking some random values. It doesn’t tell you conclusively about what the typical result looks like. Now, if you were to randomly pick values for your variables and do that over and over again then you’d start to get some real insights into the strategy. And that’s the idea behind Monte Carlo methods. An optimization algorithmic is just a more intelligent take on the same theme. Instead of randomly selecting values, the optimizer simply steps through the reasonable values one by one (or uses genetic or other advanced methods), and goes one step further by highlighting the very best input values for you. You see the true opposite of optimization is diversification. Diversification, in this case, means to trade multiple combinations of parameters. Instead of trading a strategy with optimal parameters with more size, you could trade a basket or “class” of similar strategies all running with slightly different parameters. In a future post, I’ll share with you an advanced technique for achieving both a higher degree of optimization and diversification, and without requiring a larger account.

But, here’s a simple way to optimize sensibly. Run your optimization and look over all returns. Do they look random or do you see any clear relationships? Look at the values above and below the best returns. Are they still reasonable? This is called sensitivity analysis testing and is important. Next, take an average of all returns say 25% above and below your optimal and average them to get a more realistic outlook on your system. You can also take an average of the top 25% of all returns and the bottom 25% of all considered returns. You can do this for drawdowns and other system measures too. You can also compute the min/max for these values which will provide more information than a single performance report. By going through these processes, you’ll start to build up a more complete perspective of the performance metrics of your system.

Examples follow:

Sensitivity analysis reveals that the returns are random. Don’t trade this system.

Have you ever started trading a strategy that performs well in the backtests but delivers a very different result when you begin trading it with real money?

Could your backtest reports be fooling you by indicating a strategy is great but really only showing you part of the overall picture?

[NB. OK, it's possible that this isn't absolutely the simplest trading system imaginable, but apart from buy and hold it is unlikely there are many systems much simpler than this one!]

The following chart illustrates such a portfolio for the FTSE 100 Index since 1995. The diamond markers indicate the decisions made at the end of each month whether to be in the market (green diamond) or in cash (red diamond).

One of the most rewarding experiences for a TradeStation trader is to pick up a performance report that proves their great strategy idea is indeed a profitable strategy. Strategy testing done properly, as is outlined in this article, can verify the efficacy of your trading strategy and give you confidence to start trading it. But be forewarned, strategy testing done improperly can lead you toward financial destruction.

Strategy testing done incorrectly can result in false hope in a losing strategy. A trader recently shared his experience of getting great results from strategy testing his idea, but after trading it live in the market, he was losing money every day. He was baffled about what he did wrong. Having gotten excellent results on his back-testing performance report, he wondered why his promising strategy was draining his trading account. The problem was he violated several of the proper steps necessary for reliable strategy testing.

With the knowledge of how to get an accurate performance report you will be able to trust your strategy in live trading and protect your trading account. In order to properly test a strategy, there are 5 main steps that are vital to follow; configure TradeStation, "in-sample data" testing, "out-of-sample data" testing, live forward testing on the simulator account, and real live trading execution.

Step 1: Configure TradeStation

Before you begin testing your data, you must configure TradeStation so that the data it pulls onto your performance report will be accurate. Follow these 3 critical items to properly configure TradeStation.

(a) In your TradeStation platform menu, go to “format symbol” and give TradeStation a starting and ending date to test. This historical date range is called the "in-sample data." Do not include the most recent six months in this “in-sample data.” The most recent six months is called the "out-of-sample data," and it will be used later during your "out-of-sample data" testing step.

(b) Next, in your TradeStation platform menu, go to "format strategy" and select "properties for all." Now select the "general" tab and enter the commissions and slippage (be as realistic as possible, or estimate too high if you are not sure). If this step is skipped, then the strategy testing performance report will be meaningless. If this is not done you might have a good looking performance report equity curve, but as soon as you enter the commissions and slippage figures the equity curve can reverse into an underwater equity curve.

(c) The last configuration step is under "properties for all" under the "general" tab. Look in the bottom left section called "strategy testing resolution." Check the "look-inside-bar back-testing" option and then select the smallest time frame available for your chart style to make the strategy testing more closely resemble live data. When strategy testing, TradeStation uses the open, high, low, and closing data, thus the larger the time frame bar, the more distorted the strategy testing performance report can be. This "look-inside bar back-testing" option will make the computer do a lot more strategy testing calculations. This may really slow down your performance report generation, so please be patient. For an accurate performance report you must use the "look-inside bar back-testing" option.

These configuration steps are critical to getting an accurate performance report, so be sure this is completed precisely before continuing. Once TradeStation has been configured correctly, you can begin testing your strategy.

Step 2: In-Sample Data Testing (also called “Back testing”)

You are now ready to start testing your strategy idea. We will begin with testing the “in-sample data” that you set up for testing during the configuration steps. Begin with bringing up a TradeStation performance report. Right now I have a performance report in front of me that I will refer to, but you will be looking at your own performance report to analyze your own numbers. This is what we will be referring to in the steps below. There are 7 sub-steps to “in-sample data” testing, as follows:

First, look at how many trades the strategy made. To reduce strategy testing errors, where error is defined by [ error = 1 / Square Root (Number Trades In Test) ], you want at least 400 trades to reduce the margin for error to 5% in your strategy testing results. At 100 trades you have a 10% margin for error. The greater the number of inputs in your strategy that you optimize, the greater the number of trades you need to keep from over optimizing your strategy.

Also look at how many times the strategy traded on average per day. The more often a strategy trades the more profit it can generate.

In the performance report that I am looking at, it traded 397 trades in the last 3 1/2 months, averaging 5.3 trades per day.

Second, look at the “Average Trade Amount.” It needs to be large enough that slow order fills and/or larger than normal slippage does not kill the profitability of the strategy.

In my report the “Average Trade Amount” is $162.32. The commissions and slippage amount as defined in the set up steps is already subtracted in this performance report.

65% of the time this strategy trades 1 contract.
35% of the time this strategy trades 3 contracts.
10% of the time this strategy trades 5 contracts.

Third, look to see if the “Profit Factor” and “Ratio Average Win-Average Loss” are both above 1.5 and the percentage of winning trades around 45% or better

This strategy had a “Profit Factor” of 1.83.
This strategy had a “Ratio Average Win-Average Loss” of 2.28 (2.28 means breakeven is around 28% “Percent Winning Trades”)
On this strategy the “Percent Winning Trades” was 44.58%.

Fourth, look at the trade list page and assess the profit run ups and draw downs column. Notice how many trades made money and how much money they made before the trade exit occurred. Looking at what amount of money was made in relation to the profit run up and draw down, you want to know if managing the trades could generate more profits. The example used here shows that a good percentage of trades made much higher profits than where the automated exit points occurred.

Fifth, look at the three draw down (DD) numbers. I like to see the largest number at 15% or less of the “Total Net Profit” and the “Max DD” at 5% or less of the “Total Net Profit” (these numbers tell about the draw down risk level during your trades).

Total Profit - $64,440
Peak to Valley DD - $8,960 is 13% of Total Profit
Close to Close DD - $7,120 is 11% of Total Profit
Max DD - $3,420 - 5% of Total Profit

Sixth, Looking at the “Largest Losing Trade” on the report, I like to see 5% or less of the “Total Net Profit.” In my report the “Largest Losing Trade” that occurred was $2,580 which is 4% of “Total Net Profit.”

Seventh, I review the length of time in the average trade. Does the average time in a trade comply with the golden rule of trading; "cut your losses quickly and let your profits run?" You will also want to see if the strategy is built using only profit exits (no real stop loss exits). It might have a nice looking report, but it could show a messed up ratio between average bars per winning trade verses average bars per losing trade if there are no stop loss exits. Here are my average bars:

Average bars per winning trade 7.24
Average bars per losing trade 3.51 bars

This strategy complies with the golden rule of trading. Notice how it cuts losses quickly, at an average of 3.51 bars, and lets the profits run for an average of 7.24 bars.

So what does this all mean? It means this strategy has passed the historical strategy testing phase of strategy testing.

Step 3: Out-of-Sample Data (also called "Walk Forward Testing")

Once you have tested your “in-sample data” and have determined that your strategy is worthy of continued testing, you can now test your strategy against the “out-of-sample data.” If you have not yet tested your “in-sample data, do that before proceeding.

To test the “out-of-sample data” we use the most recent 6 months of data available that you reserved in step 1(a). In step 1(b) of this article, we talked about configuring TradeStation and covered entering commissions and slippage and using the “look-inside-bar back-testing” option, which must be used to run any performance report used in your strategy testing. Be sure you have configured TradeStation correctly before moving on.

Go into “format symbol” and change the date range to include ONLY the "out-of-sample data" date range that was NOT used during the strategy testing on "in-sample data." This is referred to as testing on the "out-of-sample" data.

Begin with bringing up a TradeStation performance report on the "out-of-sample" data and review all the items that we discussed in step 2 above on this "out-of-sample" performance report. The closer it performs to the Step 2 “in-sample data” performance report, the more robust the strategy is. This suggests that the results were not from curve fitting and you have a good chance of having a viable strategy. This “out-of-sample” date range test is much more important than the strategy testing step on the “in-sample-data” for finding a successful strategy. It is a good idea to test multiple different "out-of-sample" date ranges, which is called “Walk Forward Analysis.”

Robustness: Perry J. Kaufman stated, "Practically speaking, a robust trading strategy is one that produces consistently good results across a broad set of parameter (input) values applied to many different markets tested for many years."

If the strategy fails during this "out-of-sample data" test, do NOT optimize using your reserved “out-of-sample data.” This would defeat this vitally important step in strategy development. You can go back to your strategy and fix it, or else drop it and develop a new strategy idea.

One caveat - if your strategy is capitalizing on a certain market condition, like the current volatility, and then you “out-of-sample data” test a non-volatile date range, it may not perform well, However in our next phase of testing, “ Live Forward Testing,” it could prove to be successful since we are still in a volatile market. You must understand why your strategy works, under what market conditions it performs well, and in what market conditions it does not perform well.

Now that you have tested your “out-of-sample” data and your strategy is promising, you are ready to live forward test your strategy on the simulator account.

At this point you have configured TradeStation so your performance report will be accurate, you have tested your “in-sample data” and your “out-of-sample” data and your strategy still looks great. Now you are ready to live forward test your strategy on the simulator account.

Step 4: Live Forward Testing on the Simulator Account

During your live forward testing on the simulator, you want to verify that the live data feed entries and exits are similar to historical entries and exits. After you have made live data trades for a day, save the live trade list. Now reload this same chart so the strategy recalculates based on the historical for this same day. Record the historical trade list and compare the live entries and exits to the historical entries and exits. Are they the same or at least similar? Do you understand the differences and the impact your "Live Data" test says about your strategy?

Only by monitoring the program daily can performance be seen under real "live" market conditions. Continue Live Forward Testing on the simulator until you are totally comfortable that your strategy works on live data. Real time results will often be less profitable than your historic results. The key question is does the real time testing show that you have a profitable strategy that is worth trading?

Step 5: Real Live Trading Execution

Once you have done your due diligence and are comfortable with your strategy results on the simulator, you are ready to trade live. Since you are trading a brand new strategy with real money, start with a significantly reduced position sizing risk of 1/4 of 1% of your account equity at risk per your stop loss point per trade. Continue trading with minimal risk until you have verified everything is working properly within your new strategy during live order executions.

Once your strategy is making money in the live market, slowly over time begin to increase your position sizing risk. Move your risk upward from 1/4 of 1% toward 1%. Continue trading at 1% risk until you have several weeks to months of consistent trading performance. If you want to be aggressive and use more, you can continue to increase toward the maximum 3% of your account equity at risk per trade.

If you follow the 5 steps as outlined in this article, you will now be able to confidently strategy test any strategy idea that you have. Keep this article for reference so the next time you are inspired with a great idea, you will be able to prove it out, protect your TradeStation trading account, and have confidence in live trading your strategy.

Click on this link to learn how to quickly and accurately strategy test any trading idea you might have.

-- by Mark David Johnson from Customizedtrading.

Mark David Johnson is a full time TradeStation Trader and TradeStation EasyLanguage Programmer for the past 6 years. He has been involved in programming for over 26 years and involved in trading for over 16 years.

Using a stop on a position is a very popular risk management technique used by traders. My research and experience has led me to believe they are appropriate for some – but not all – types of trades. Today I will discuss when I believe they aren’t appropriate.

In Larry Connors' new book, “Short-term Trading Strategies That Work”, he dedicates a chapter to stops. It’s entitled, “Stops Hurt”.  The chapter discusses how Larry’s research team ran hundreds of tests to try and find optimal stop levels. In doing so, they came to the conclusion that for the trades they were looking at, the optimal stop was consistently none at all. In every case they found that instituting stops hurt system performance.

You should keep in mind that Larry Connors trades mean reversion strategies. Much of what I do is mean reversion based also. For instance, the Catapult system which makes up the CBI looks to buy stocks that are undergoing capitulative selling. It enters long positions in stocks or ETFs that are extremely oversold. When I first designed the system in 2005 I went through a massive series of tests to find a way to successfully incorporate stops into the methodology. Like Larry I failed to find a stop technique that would enhance the performance of the system.

I’ve gone through numerous other exercises and found the same thing time and time again. When looking to trade overbought/oversold techniques, stops generally don’t work well. If the system suggests the security should bounce when it drops to $20 and it continues to $18, then it is REALLY overdue for a bounce. Any level of stop ensures you are selling an extremely oversold security that is making a low. Those are buying conditions for oversold systems – not selling conditions.

One stop technique for oversold systems that I will sometimes use that in testing hurts performance less than the other techniques I evaluated is this:

Wait until the security bounces for a bar or two. Look for a higher high, higher low, and higher close – or at least 2 of those 3. Then place a stop under the swing low that was just made. In cases like this even if the security doesn't hit your target exit price, it still ensures that you won’t have to suffer through the entire next leg down. While it seems logical and can sometimes help avoid catastrophic trades in the long run, you’re normally better off just waiting for the mean reversion to occur and exiting at your target level.

Not using stops does not equal not controlling risk. Position sizing becomes very important. Traders could also consider using options to trade their short-term positions. Options provide a natural stop (zero). I wrote a series back in the spring (when the VIX was a lot lower) on how I sometimes use options for my short-term trading. You can find the link here:

While stops do not work well for overbought/oversold trading, they DO work well with breakouts or trend following systems. Traders that buy on a pattern breakout do so because their analysis indicates a trend could emerge in the stock or security they are trading. A reversal back into the base or below it would invalidate the pattern from a technician’s standpoint. Therefore, in such cases I believe a stop is completely appropriate. Once the pattern “fails” you should no longer be in the trade. Of course some people may want to give a little extra leeway rather than putting a stop right at a support point, but even so, there is a point where the breakout simply didn’t take.

For my own breakout trading I tend to use very tight stops. I also tend to show little patience for a trade to work once I enter it. Most successful breakouts have a tendency to work right away, and when the market environment is conducive to breakouts there’s just no point in sitting around with dead wood. I’d rather exit with a small profit or loss and try the next one.

The edge in breakout or trend trading is not in the winning percentage. It’s in the risk/reward. A big winner can gain several hundred percent if it goes on a tear. That makes up for an awful lot of scratches and small losses. As an example, in 2003 I did a lot of breakout trading. Cup & Handles, Flat Bases, High Tight Flags, Double Bottoms, etc. – all based primarily on daily bars. It was a tremendous year for trading breakouts because the ones that took caught fire rather quickly. It was also one of my best market years. Yet, at the end of the year I went back and tallied the percentage of breakout trades I took that “worked”.  In other words, they did better than a very small gain or scratch. My success rate? A little under 15%.  And it was a great year. Why?  Tiny losses and massive gains.

Everyone’s style is different and someone with more patience would likely have had a better win rate, but the win rate wasn’t important. What was important was controlling the losses, and one way to do that was through the use of stops.

-- By Rob Hanna from Quantifiable Edges. Rob Hanna has been a full-time market professional since 2001.  He has served as president of Hanna Capital Management, LLC since that time.  He first began publishing his market views and research in 2003.  From 2003 to 2007 his column “Rob Hanna’s Putting It All Together” could be found twice a week on TradingMarkets.com. In January of 2008 Rob began Quantifiable Edges.  In 2012 Rob opened his 2nd website, Overnight Edges.  Both sites use historical analysis to asses current market action and odds.  Rob utilizes price action, volume, breadth, sentiment, seasonality, liquidity flows and more to conduct his research.  Some of the indicators he uses are well known and publicly available.  Others were created in-house.  His work has been widely referenced and quoted over the years, and is often linked to in blogs, tweets, Stocktwits messages, magazine articles and more.  Below are some other places you may have seen some of Rob’s work.