In a recent article entitled, “Backtesting the MAC-System – How Long is Long Enough?“, the author presented a simple rotational trading system based upon moving averages. The system is one of the most basic rotational systems you could devise. Based upon signals generated from the moving averages, your investment is either within the S&P index or a bond fund. Several readers voiced their concerns over the possibility of the system being over optimized to the historical data. This is a valid point and worth looking into.
When it comes to making a profitable trading system we depend upon finding market edges we can exploit. All successful trading relies on recurring patterns found in the historical market data and exploiting those patterns to make a profit. This is true for discretionary traders as well as system traders. Optimizing our system to take advantage of market edges is not the problem. We want to optimize our system to take advantage of an edge.
Yet, much of the historical market movement is noise or random movement. So, the problem of over optimizing our system to the historical data starts to happen when our system begins to key-into patterns within the noise instead of exploiting a true market edge. Finding a pattern in the noise can produce great looking equity curves on historical data, but on out-of-sample data the equity curve can break down rather quickly.
In summary, we want a trading system that exploits a true market edge and avoids keying-off patterns in the market noise. Optimization is not the problem. Over fitting is the problem and having the ability in separating a true market edge from the noise is what this game is all about.
So, how can we test the MAC system to help us determine if it’s overly optimized? When looking at a trading model, it’s important that the parameters demonstrate robustness across a wide range. That is, the system should remain profitable over many different values. We should be able to change some of the key input values and we should have a viable looking trading model. The output (net profit in this case) should not drastically change.
In order to explore the robustness of the basic premise of this trading model I created a stripped-down version in EasyLanguage that buys and sells the S&P cash market only. The complete trading model, as presented by the author of the MAC system, has rules to move cash into a bond fund during “bear” markets, but I’m particularly interested in testing the buy/sell signals generated by the various moving averages as these seemed to produce the most amount of questions by readers.
Trading Model Rules
Our trading model is very simple with signals to go-long the S&P cash market and when to close our position. The buy/sell rules are below.
Buy Signal
The 34-day exponential moving average (EMA) of the S&P 500 becomes greater than 1.001 times the 200-day SMA.
Sell Signal
The 40-day simple moving average (SMA) of the S&P crosses below the 200-day SMA.
Testing Environment
I coded the above rules in EasyLanguage and tested them on the S&P cash market. Before getting into the details of the results, let me say this: all the tests within this article are going to use the following assumptions:
- Starting account size of $100,000
- Dates tested are from October 1969 through September 2014
- Shares were scaled based upon price and volatility
- The P&L is not accumulated
- There were no deductions for slippage and commissions
- There were no stops
Please note we are not adding our profits to our trading account! We are always trading a small percentage of our starting capital. Here is the position sizing formula used:
Shares = $5,000 per trade / 5 * ATR(10)
Baseline Results
Just using the simple rules above we get the following equity curve and results.
MAC Baseline Model
MAC Baseline | |
|---|---|
Net Profit | $279,023 |
Profit Factor | 11.65 |
Total Trades | 27 |
%Winners | 70% |
Avg.Trade Net Profit | $10,334.19 |
Annual Rate of Return | 2.71% |
Max Drawdown(Intraday) | $55,741 |
Expectancy | 3.16 |
Expectancy Score | 1.73 |
Testing Trend Filter Period
The baseline rules call for a 200 SMA to act as a major trend filter. Is this 200 value just an outlier? Or, is this parameter robust for a wide range of values? To test the robustness of this input, I will use TradeStation’s optimization feature which will allow me to quickly test a range of values. I will test the range 60 – 260 in increments of 10 for the input value used in the SMA calculation. The results of my test are below. The x-axis displays the look-back period in days while the y-axis displays the net profit.
The optimal value (260) is not the default value (200). Many values show profitable results. The default value and above all show very similar results.
Testing Buy Period
The baseline rules call for a 34 EMA to rise above the 200 SMA a given amount to trigger a buy signal. Is the 34 period used for the EMA calculation overly optimized? To test the robustness of this input, I will use TradeStation’s optimization feature which will allow me to quickly test a range of values. I will test the range 5 – 75 in increments of 5 for the input value used in the EMA calculation. The results of my test are below. The x-axis displays the look-back period in days while the y-axis displays the net profit.
The optimal value (20) is not the default value (34). All values show profitable and very similar results.
Testing Buy Factor
The baseline rules call for a 34 EMA to rise above the 200 SMA by a factor over 1.001. Is this factor overly optimized? Again, I will use TradeStation’s optimization feature. I will test the range 0.990 – 1.010 in increments of .001 for the input value used in the factor calculation. The results of my test are below. The x-axis displays the factor value while the y-axis displays the net profit.
The optimal value (.98) is not the default value (1.001). All the values show profitable and very similar results to the default value.
Testing Exit Period
The baseline rules call for a 40 SMA cross below the 200 SMA to trigger a sell signal. Is the 40 period used for the SMA calculation overly optimized? Once again, I will use TradeStation’s optimization feature to quickly test a range of values. I will test the range 5 – 75 in increments of 5 for the input value used in the SMA calculation. The results of my test are below. The x-axis displays the look-back period in days while the y-axis displays the net profit.
The optimal value (55) is not the default value (20). All values show profitable results. Values between 35 and 60 all show similar results.
Conclusion
It appears this trading model is not overly optimized. Through my simple testing I was able to demonstrate that the core of the MAC trading system shows profitability over a wide range of values. In fact, some combinations produce better results. The default values do not appear overly optimized. Note, many different values appear to work just fine.
It’s my opinion the MAC system will likely work into the future. It works because it positions a trader into bull trends as seen from 2009 to today (late 2014) while more importantly, it gets a trader out of the market during prolonged bear markets such as seen in 2007-2008. The exact parameters don’t matter! You can use a 260 SMA or a 180 SMA for the major trend filter. You can enter the market using a 40 EMA or a 20 EMA. You can use a factor of 1.001 or .0098 as the buy threshold. They all produce positive results which speaks to the robustness of the system.
It’s true the testing period within this article only produced 16 trades (samples) and this is a very low value. Part of this is due to the fact I don’t have all the historical data at my disposal to test. However, based upon the authors original work of going back to 1955, I would feel confident that we would have a sufficient number of samples.
Simple systems on daily charts applied to a broad market index provide a systematic approach that often will do far better than what most people do. That is, make buying and selling decisions on emotional reactions.
What About SPY?
Several people have asked about this model on SPY. Strictly speaking, the MAC-System presented in this article is not the complete MAC-System. The EasyLanguage code I wrote for the above tests was designed strictly to test the robustness of the input values – not to test the overall performance of the strategy. The trading model presented is a stripped down version of the MAC-System. I would refer you to the article “Backtesting the MAC-System – How Long is Long Enough?” for a more complete description of the performance of SPY on the MAC-System
However given the strategy code presented in this article is not the complete MAC-System, below are the results when applied to SPY. Profits are reinvested. All trading equity is invested for each trade. $16 commissions and $.04 slippage per share was deducted per round trip. The TradeStation Performance Report can be download here MAC_Test_SPY.
Jeff, when was trade #8? It looks like around half the total profit came from that one trade.
Ross, the largest trade was during the dot-com boom. That trade was from 1995-1998 when the S&P climbed from around 465 to 1020. This generated a profit of $118,610. The current open trade may be within the top three when it closes. The open profit is $44,900.
Jeff, a curve-fitted system will be always robust with respect to parameter changes. Actually you proved that the system was curve-fitted not the opposite as you argue. You have to rethink your whole philosophy.The reason that the MAC system worked on SPX data is because there are no prolonged sideways periods like in some other markets. Actually the 1990s trend makes the system robust to paarmeter variations.
Hello Alex. I disagree that “a curve-fitted system will be always robust with respect to parameter changes”. I’ve seen overly optimized systems where you change a parameter and the equity curve falls apart. This would be a red flag in my book. With a system that shows robustness across many different parameters, I feel much better. Besides, curve-fitting is not the problem. Every system is curve-fitted to the historical data. I rely on historical patterns or behavior repeating into the future. Of course if the market changes dramatically (stops trending for a prolonged period), the system will stop working. This is nothing new as the MAC system is a trend following system. Of course a trend following system will stop working when a market no longer trends. Discretionary traders as well as system traders adapt over time as edges appear and disappear. Knowing when to stop trading a system is part of any trading plan. Some edges last decades, others don’t. For the MAC system you can remove the 1990’s trend and it still works and the parameters remain robust. The edge exploited by the MAC system may last for another 100 years, or it may die over the next couple of years. No one knows. All I can do is make best estimates based upon history and then remain flexible at what the future holds. If one had to wait for 100% certainty no trades would be taken.
Despite everything I believe it would be really interesting to see how the system behaves with RUSSEL 2000 or NASDAQ, just to see if we can boost returns more.
https://nightlypatterns.wordpress.com
Hello Marco. I just ran the system on SPY, ES, QQQ, DOW, and IWM. All produce positive results. For fun I ran it on APPL, GLD and USO. Again, positive results except for USO. While the USO equity curve is still in the green, it’s an ugly falling equity curve. It’s clear the system was designed for the S&P, but it will hold-up on other markets.
“Besides, curve-fitting is not the problem”
Curve-fitting IS the problem. There’s NO edge to the MAC system. It’s all an illusion you have fallen victim of.
Jeff, curve-fitting and optimization are not equivalent notions. You seem to use them interchangeably. The formal definition of optimization is that it is a global maximum that is robust when parameters are perturbed. What you have seen that broke down was an unstable local maximum. The MAC system parameters have been selected for a global maximum for SPX. Why don’t you apply it to some commodity futures?
Jeff the MAC system has 5 parameters. Any claim that it is not fitted is ludicrous to say the least. BTW just trying to help as I have fallen a victim of such systems in the past and have lost money.
Hello Alex. I appreciate your opinion and desire to help. Thanks. However, I do disagree. In my opinion curve-fitting is not the problem as all systems are curve fitted to the historical data. Over-optimization would be the potential problem. So, yes curve-fitting and over optimization are two different aspects which was one of the points of the article. As you state, selecting local maximum values for each individual input parameter would likely increase the chances for having an over-optimized system and this would be a problem. As stated in the article, I found the system not to be overly-optimized. Could the system be overly optimized? Maybe, but I did not find evidence to support this. Might the system not work into the future? Maybe. If the market drastically deviates from it’s historical behavior, it will stop working. Furthermore, it has been my experience that systems don’t need to work across different markets. Ideally they should work on similar markets. In this case, the system was built for the stock index markets. Specifically the S&P cash market. However it should fair well on the S&P futures, the DOW, QQQQ and others of that nature. Developing the system across similar (highly correlated) markets would be another method to avoid over-optimization as would using in-Sample/Out-of-Sample data during the development period. In the end, if the system does not do well on Gold, Oil or Apple stock, that would not necessarily be a “bad” thing. I’ve seen far too many systems work for years and years when build for specific markets. I just ran the system on SPY, ES, QQQ, DOW, and IWM. All produce positive results. For fun I ran it on APPL, GLD and USO. Again, positive results except for USO. While the USO equity curve is still in the green, it’s an ugly falling equity curve. It’s clear the system was designed for the S&P, but it will hold-up on other markets. This is another bit of evidence that the system is not overly-optimized to the historical data.
Some more comments:
I get 27 trades in the same period in SPX with data from Yahoo finance and a backtest in excel. Would it be possible to reply with your EL code in text form since I have no TS? Do you check for a cross before you enter and exit? Any additional rules other than those stated?
Then I suggest the following:
Please report the max % DD, not $$.
Try SPY because in reality you cannot buy SPX shares and the backtest is unrealistic.
What are the units of expectancy and what it stands for? Is it average trade divided by average loser?
What is the Sharpe ratio? I get about 0.60. This should tell you whether this is just a fluke or not. At 0.60 it is a fluke. You should have more than 1.50 to be sure.
Correction: It is 21 trades I get, not 27 trades.
Alex, I posted some results at the bottom of the article. As for your questions, there is no check for a cross on a buy. The 34-EMA must be 1.001 times the value of the 200-SMA before a long will be entered. When a 40-SMA crosses below the 200-SMA the position is closed. Hope this helps.
Hi Jeff,
“The 34-EMA must be 1.001 times the value of the 200-SMA before a long will be entered.”
But in the article you say:
“The 34-day exponential moving average (EMA) of the S&P 500 becomes greater than 1.001 times the 200-day EMA.”
SMA or EMA?
Then, you say
“The P&L is not accumulated”
What do you mean here? The CAR you got cannot be true unless you fully reinvented the closed equity on each trade.
Then, here you say:
“Dates tested are from October 1969 through September 2014”
But below the equity graph the starting date is shown as 10/11/1979
As you see, besides the curtve-fitting issues, there are some other issues with the article. Regardless, what you have proved is that this is a very robust curve-fitted system on SPX data. It will work only on price series with strong uptrends followed by downtrends. If there is no uptrend for a few years but only sideways action, those who use this system will go bust big way.
Alex, Sorry for the confusion on the rules. Please check the article for the entry/exit rules. They should be stated correctly. Likewise, the annualized return was wrong along with the equity curve was not correct. Those images were from a different MAC-System run run I was testing. Thanks for pointing that out.
I almost did not even bother to include a performance report and equity graph in this article because that’s not the point of the article. The point was to demonstrate a technique on testing the robustness of input values. A technique anyone could apply to a system to help gauge if it’s overly optimized.
“The P&L is not accumulated” means profits are not added to the trading equity. Instead, each trade is simply based upon the starting equity.
I agree that this is a trend following system. Thus, it requires a trending market. This is really no different than nearly all trading systems I’ve ever seen. They work because they take advantage of historical market characteristics. When those historical characteristic fade and die, so does the system. This is nothing new and something we all live with as a traders. In short, you can always find an excuse no matter how well built and tested a trading system is, on why you should not trade it.
Thanks Jeff but please note that although the number of trades was matched after the clarification you made, the net profit does not because you number of shares rules seems not to make any sense. The position sizing formula you suggest may generate more shares that the available equity depending on the value of ATR(10).
Shares = $5,000 per trade / 5 * ATR(10)
Actually you are not “always trading a small percentage of our starting capital” with this formula. If you want to do that you should select the amount and divide by the price to open the position.
Unfortunately, your results cannot be reproduced and that also casts doubts on your claims that this is not an overly optimized system. My analysis shows that this is actually a highly optimized system to follow steep trends. It would be good if you posted your EL code in test form.
More importantly, the procedure you followed actually tests if a system curve-fit is robust, not if a system is overly optimized.
Hello Alex, the formula is actually a function and will not allow more shares to be purchased than the given account size. Please note, the results on this page are not the focus here. This is not the MAC-System as defined by the author of the original system and defeats the purpose of this article which is testing of the input parameters – not the performance of the trading model. I would urge you to contact the original creator of the MAC-System to discuss exact backtesting and historical performance. I disagree that this system is overly optimized. I also understand that you feel it is overly optimized. Maybe you can post your results on why you feel it’s overly optimized?
Jeff, thanks, Your articles are good to stimulate thinking but I think many will agree with me on this one. I wrote:
“…the procedure you followed actually tests if a system curve-fit is robust, not if a system is overly optimized.”
“Optimized” is a vague term. With systems we are interested to know whether there is curve-fit and how strong it is. You proved beyond doubt the opposite of what you think in my opinion, i.e. that the system is well curve-fitted because it responds well to parameters changes. You think that it means it is not overly optimized because of that. But this is a distracting issue in a way. Think about it.