The Perils of Buying Trading Systems

Ugh. I’ve done it again. I bought a “can’t miss” trading system I saw on the Internet. What looked so good in the advertisement turned out to look just as bad when I traded it with real money. This seems to happen to me EVERY time I buy a system. Why?

Sound familiar? It certainly should, if you are in the habit of buying trading systems. That is not to say there are not good ones out there – there certainly are. But by and large, purchasing trading systems is an exercise fraught with peril. Here are some reasons why it is so dangerous, and how to protect yourself.

Inherent Dangers in Buying a Trading System

When you buy a trading system based on an advertisement you see in an e-mail, magazine, or website, you typically will see some summary statistics, and a smooth looking equity curve. It probably will look too good to be true (your first warning sign!), but certainly it will look enticing, like the equity curve shown in Figure 1.

Although in the back of your mind you will wonder why a vendor would sell the “Holy Grail” so cheaply, you probably won’t dwell on his motivations for selling it (likely, he needs the money to make up for his trading losses!). Surely, you’ll reason, such great performance is possible, and you’d be happy with even 25% or 50% of the profits shown. At that point, the hook is in your mouth, and the vendor only needs to jerk the line and set the hook to bring you on his boat.

Unfortunately, those glorious results are almost always a result of incorrect testing. Here are a few of the more common tricks deployed by many unscrupulous vendors.

Over-optimization

Take some market data, and subject it to enough tests, and eventually a great equity curve will result. This occurs all the time, through the abuse of a testing process known as optimization. If a vendor puts enough rules in the strategy, with enough parameters to create millions of different trading systems, by random luck a few systems will have extraordinary performance. In effect, the data has been tortured long and hard enough to give up its secrets. But those secrets almost never apply to future data.

The tough part about optimization is that many times you cannot tell if optimization was utilized properly during development. The key for me is that if it looks too good to be true, it probably was over-optimized.

Incorrect Development Process

Most trading system vendors have no clue how to properly develop a trading system. That explains why they are primarily vendors, not traders! Developing a system the correct way is tough work, and many times extremely frustrating. For example, the process I use is shown in Figure 2. It is long, tedious, and difficult. Most of the trading ideas I test turn out to be worthless. Although there are other legitimate ways to develop a system, this approach has worked time and time again for me.

As with over-optimization, you’ll almost never be able to know for sure if a trading system vendor developed the system with a flawed process. Because of this, I always assume the vendor did it incorrectly. That assumption can save you thousands.

Deceiving the Backtest Software

Another common approach that “snake oil” vendors employ is to trick the backtest software into giving phony results. Every piece of software, including industry leaders such as Tradestation, Ninja Trader and Multicharts, has multiple ways to exploit the strategy backtest engine. Sneaky vendors know this, and take advantage whenever they can.

One popular exploit deals with limit orders. In the real money world, most limit orders are only filled when your limit price is exceeded. Most software, however, can be configured to provide limit fills as soon as the price is touched. This can make most scalping systems look terrific in backtest, when in real life they would fail miserably.

Here is an example, shown in Figure 3. The equity curve on the left assumes that limit orders are filled as soon as they are touched (the incorrect way). The curve on the right shows trades that fill only when the limit order is penetrated. What a difference!

Although many times this exploit is hard to detect, one sure sign is to look at a chart of trades, as shown in Figure 4. If there are a number of buys at the low, or sells at the top, you can bet the limit order exploit was used and that results should be ignored.

Protect Yourself

Knowing three common ways trading system results are usually not what they seem, what can you do to protect yourself? Here are some possibilities:

Realize the Holy Grail is Not for Sale

What kind of person sells a golden goose for a few dollars? Wouldn’t it be better to keep it for himself, and trade his way to financial riches from the proverbial tropical beach? Think about that before you buy a trading system. On the other hand, there are legitimate vendors who sell decent performing systems, and likely trade those themselves. But who wants to buy “decent” when you could instead buy the seemingly “Holy Grail” of systems? Good legitimate vendors with good systems get buried under a pile of shiny looking garbage.

Run in Real Time with No Money

Many times, an over-optimized system will fall apart with future data. It might take a week, or a month, but eventually it will fail. So, if you must buy a trading system, be patient before dedicating real money to it. Watch its performance for 3-6 months, and see what happens. You’ll definitely be glad you did.

Run Small Size, Check for Fill Matching

If you are worried about a system getting overly optimistic fills, which occurs many times with scalping systems, you can always try running a handful of trades with real money. If you then watch the real money performance and live strategy performance at the same time, you’ll frequently uncover the mystery behind the optimistic fills. Such a limited test is usually a small price to pay to reveal the truth.

Develop Your Own Systems

If you have the capability and desire, developing your own system is the way to go. It is the only way I do things now. Of course, you must utilize the correct process, otherwise you’ll make one of the hundreds of development mistakes that inexperienced traders make. But, do it correctly and you can create good trading strategy after good trading strategy.

Conclusion

Buying a trading system is more often than not a bad idea, especially if you only know the vendor from the equity curves in his ads. Most vendors are either failed traders, or wannabe traders. They make their money from vending, not trading.

After 20+ years in the retail trading world, I have concluded that building your own trading systems is the only way to go. While you have to watch out for many of the same tricks that vendors use to sell garbage, once you have a solid development process, comprising of sound practices, you’ll be a lot closer to being a successful trader than you would by relying on a hack trading system vendor. In other words, your trading success will be tied to your abilities, not the phantom success of a trading system vendor.

If you enjoyed this article and want to learn more techniques for building profitable trading systems, then checkout my book, “Building Winning Algorithmic Systems”.

— Kevin J. Davey of KJ Trading Systems