Finding Out What Works, And What Doesn’t Work – Part II

In Part 1 of this series, found here, I ran a simple study of a trading system, in order to look at some of the common components (such as end of day closes, short versus long bar periods) that many system developers add to their trading system. Could some of these strategy building blocks be hurting their development efforts? To answer the question, I analyzed one trading system over 7 different markets for a 5 year period. Even though I was able to extract some conclusions from the study, I also said the following:

“Of course, I made these conclusions based on one study. What if the strategy was different? What if the timeframes or markets were different? What if different years were used for the test period? Will the conclusions reached here still hold? I’ll examine those questions in Part 2.”

To summarize the earlier work, the conclusions from Part 1 were:

1. Striving for profit in multiple markets, as a confirmation of a strategy, is indeed possible.
2. Swing trading is likely more profitable than intraday trading.
3. Longer timeframes are generally superior to short time periods, but this is not a major effect.

In this second part of the study, I’ll now look at a different group of markets, a completely different strategy and some different timeframes. I’ll also try to answer the following questions: Will the conclusions still hold up? What type of insights, if any, do conducting studies like these actually provide? Finally, can studies like these improve your own strategy development?

Ground Rules

Previously in Part 1, I tested 7 different markets across a range of commodities.

(Group 1):

• Wheat (W)
• 10 Year Treasury Notes (TY)
• Lean Hogs (LH)
• Australian Dollar (AD)
• Heating Oil (HO)
• Cotton (CT)
• the e-mini Nasdaq (NQ)

I picked these at random, one from each major market group. The test period will be from 1/1/2007 to 12/31/2011, a 5 year period that includes some quiet, and some very chaotic, markets.

For Part 2, I will test another basket of commodities.

(Group 2):

• Live Cattle (LC)
• Canadian Dollar (CD)
• Copper (HG)
• Natural Gas (NG)
• Sugar (SB)

I will also test the period 1/1/2012 to present, a smaller test period outside of the range previously tested.

Finally, I will assume $5 round turn commissions and $30 round turn slippage.

The Systems

I will use two extremely simple base strategies for the tests I am running: a simple breakout based on closing prices (Strategies A and B), or a reverse breakout (countertrend – Strategies C and D). In all cases, the breakout length X will be varied from 5 bars to 100 bars.

Here is the code for all four versions of the strategy:

Results – Repeat of Part 1 Test, Different Instruments

First, I will conduct a test of Strategies A and B for the same period, with the second group of commodities. To keep things manageable, only the group summary results are shown below:

Strategy A- Breakout, Swing Trading

$ Thousands ($K) Maximum Net Profit, 2007-2011

$ Thousands ($K) Maximum Drawdown for Max Net Profit case, 2007-2011

% of Intereaction Profitable (100%=goal) 2007-2011

Strategy B- Breakout, Exit of the day

$ Thousands ($K) Maximum Net Profit, 2007-2011

$ Thousands ($K) Maximum Drawdown for Max Net Profit case, 2007-2011

no data shown for cases where maximum net profit <$0

% of Iterations Profitable (100%=goal), 2007-2011

no data shown for cases where maximum net profit <$0

The results of Group 2 are generally in line with Group 1 results, which show that swing trading (strategy A) is better than intraday trading (Strategy B), and that longer timeframes are slightly better than short timeframe bars.

Results–Different Timeframes, Different Time Period, Different Strategy

Of course, concluding that long timeframe bars and swing trading is superior based on only a few tests is dangerous. So, let’s look at Strategies C & D (the opposite Strategies of A & B), and see if the results have the same tendencies for different timeframes and test period, on a random subset of Group 1 and Group 2 instruments.

Strategy C- Reverse Breakout, Swing Trading

Strategy D- Reverse Breakout, Exit End of the day

$ Thousands ($K) Maximum Net Profit, 2012 - present

% of Iterations Profitable (100%=goal), 2012 - present

For timeframes, the results show a tendency for the smaller timeframes to be better than the longer timeframes, exactly the opposite of what was found earlier. But this effect is small, which could mean it is not actually true, and could be just an artifact of the limited testing. This is a good example why it is dangerous to draw conclusions from only a few tests.

For the swing vs. intraday testing, the results are pretty clear – intraday testing is not nearly as good as swing trading. This agrees with the earlier conclusion.

Conclusion

Through both Part 1 and Part 2 of this study, a lot of tests have been run over various bar timeframes, instruments, strategies and time periods. Yet, it would be dangerous to make any sweeping conclusions based on these tests. Although these results show that intraday trading is less profitable than swing trading, there are surely cases where intraday trading is preferable over swing trading.

So, what can this study actually teach us? I think the following lessons come out of this study:

1. Intraday trading is less profitable than swing trading. This does not mean this always holds, but if you do most of your testing intraday and you are struggling to develop a strategy, perhaps opening up the strategy to allow swing trading is a good idea. It could be that you have a good swing strategy, but the end of day exit alone makes the strategy look bad.
2. Longer timeframes are better (Part 1 conclusion), except when they aren’t (Part 2 conclusion). So, be careful of making conclusions when the effect is small, or results are contradictory. Since many strategy developers tend to find something that does or does not work and then focus only on that (or avoid it), the lesson here is that the developer should not “box” themselves in. Maybe short timeframes rarely work with your strategies, but that does not mean you should never test them. There will be other strategies where shorter time bars are an asset.
3. Since a developer can never test everything, it may be worthwhile to vary your test approach. By looking at different timeframes, instruments, etc., you may discover new paths to take in your strategy development.

The key to successful strategy testing and development is to keep an open mind, and allow the data to point you in the right direction. At the same time, though, don’t let the data trick you into making false conclusions or sweeping generalizations. Remaining flexible in your development approach can lead to new and profitable strategies – ones that you might now be dismissing based solely on past testing.

If you would like to learn more about building trading systems be sure to get a copy of my latest book, Building Winning Algorithmic Trading Systems.

— Kevin J. Davey of KJ Trading Systems