In the previous article, “Testing A Simple Gap Strategy“, we were looking at two other filters in an attempt to improve the Gap #1 strategy. The first filter was a day-of-the-week filter while the other filter was based upon the size of the gaps. These filters were tested on our in-sample data segment and appeared to help our trading performance. In this article, let’s see how these filters perform on the out-of-sample (OOS) data segment.
Baseline OOS Performance
We’ll need a metric to compare our new filters against, and the original baseline system will be perfect. The baseline system will be applied to the OOS and the results will be used as our benchmark.
Gap Fade #1 OOS
Baseline | |
|---|---|
Total Net Profit | $3,835 |
Profit Factor | 1.27 |
Total Trades | 147 |
% Winners | 72% |
Avg.Trade Net Profit | $26.09 |
Annual Rate of Return | 1.39% |
Max Drawdown (Intraday) | 3.18% |
We can see the baseline performance is performing as expected based upon the in-sample results (see table below). The profit factor, percent winners, and annual return are similar between the two data segments. The OOS actually shows a larger profit per trade.
Gap Fade #1
Baseline | |
|---|---|
Net Profit | $24,118 |
Profit Factor | 1.2 |
Total Trades | 1,118 |
% Winners | 72% |
Avg.Trade Net Profit | $20.42 |
Annual Rate of Return | 1.44% |
Max Drawdown (Intraday) | 5.2% |
Day Of Week Filter OOS Performance
Now we’ll apply our trading model which contains the day-of-week filter to the OOS data segment. The results are below.
Gap Fade #1 OOS DOW Filter
Baseline | Dow | |
|---|---|---|
Net Profit | $3,835 | $433 |
Profit Factor | 1.27 | 1.04 |
Total Trades | 147 | 89 |
%Winners | 72% | 71% |
Avg.Trade Net Profit | $26.09 | $4.86 |
Annual Rate of Return | 1.39% | 0.16% |
Max Drawdown (Intraday) | 3.18% | 2.5% |
Well, this does not look that great. Our equity curve flounders around like a fish out of water. I’d like to see the equity continue to climb and make new equity highs. Not happening here.
Gap Size Filter OOS Performance
Now we’ll apply our trading model which contains the gap size filter to the OOS data segment. The results are below.
Gap Fade #1 OOS Gap Size Filter
Baseline | Dow | |
|---|---|---|
Net Profit | $3,835 | $3,975 |
Profit Factor | 1.27 | 1.3 |
Total Trades | 147 | 140 |
%Winners | 72% | 72% |
Avg.Trade Net Profit | $26.09 | $28.29 |
Annual Rate of Return | 1.39% | 1.44% |
Max Drawdown (Intraday) | 3.18% | 3.13% |
This looks better than the day-of-week filter, but the improvement is not much. We have 140 trades on our OOS data segment, but the results are about the same.
It appears that our filters provide no real edge or at least, no long lasting edge. I would like to see our OOS performance with our given filter(s) outperform our baseline. It’s not enough to simply be a little better, we would like to see significant difference for the better. In this case we see less performance with our DOW filter and about the same performance with our gap size filter. This may very well mean we should scrap our filter ideas and test other possible improvements.
A word of caution! What we don’t want to do is go back and tweak our current filters to make them perform better on the OOS data. This might be your first natural instinct. This can be a powerful motive if our OOS results looked decent such as with our gap size filter. While going back to our in-sample segment and making them look better is tempting, it’s a big mistake! By making decisions based upon the OOS data segment and using that information to make changes to our system, we’re actually transforming our OOS segment into our in-sample. We are using knowledge gained during our OOS and transferring that modification of the code. Think about that! We want our OOS segment to remain “clean” yet, every time we make a decision based on the OOS results to change our system, we are destroying the very essence of why we have an OOS segment in the first place. We must be careful here.
Now granted, you can’t avoid this completely. I may decide to test another filter and this decision is based upon our OOS results. So, there is some level of information leakage that will taint our design process. This is largely 100% unavoidable. We all bring biases and future information into our historical development. What we hope to avoid is large, glaring mistakes.
What Can We Do?
Since our performance in our OOS is not an improvement I see three possible actions we can take.
- Scrap existing filters and design new filters on the in-sample data segment
- Delay our final judgement until more data is collected
- Accept our baseline as good-enough and move on
Scrap Our Filters
First, we could decide our filters provide no real edge and we must go back to the in-sample segment to try different filters. Note, if we follow this line of action we are taking information obtained during our OOS segment and using that knowledge to modify our system on the in-sample segment. However, this is a reasonable risk I’m willing to accept. I would not go back and tweak the existing day-of-week or gap size filters. We already selected reasonable values based upon our in-sample data segment. It’s a done deal and I would not modify them. I either accept them or discard them.
Delay Our Final Judgement
Second, maybe our filters are working and our system is simply experiencing a normal drawdown or flat period. It may very well resume its upward trend later. Maybe in another six months one of our new filters (or both) may be producing better returns than our baseline for the next five years. It’s possible. Over the years I’ve seen this happen a handful of times. It’s not a likely case but if we were to pursue this I would first go back in history during the in-sample data segment and see if we can find a similar period where our performance languished. I want to see if there has been similar periods in our in-sample backtest where the equity curve looks similar to our OOS over a similar time period or number of trades. This would help tell me if the poor performance on the OOS segment is something this system has experienced before. If it has, just maybe we are not experiencing anything more than a pause before a new push up to equity highs.
To further test this concept, I will set these systems aside and check on them in six months or so and see how they are performing.
Our Baseline Is Good Enough!
Third, it’s likely our filters provide no edge and we can simply accept the baseline system as good-enough and move on to the next stage of testing. As the baseline stands, $26 net profit per trade is not large enough for me to risk my capital.
While these three ideas are all valid, I’m going to go back to our in-sample and try a different filter. That will be for the next article in this series.
What trading hours do you use for this? The ES trades pretty much continuously… I’m new to futures.
The gaps strategy trades from 8:30 to 15:15 Central time. Thus, the gap is based upon yesterdays close vs today’s open.
By continually going back to the in sample data and searching for a filter that works with the OOS data, you’re performing an optimisation that might create a false signal. The monkey at the key board problem. When does your trading ‘gut’ tell you that you’re trying to fit and square peg through a round hole and it’s time to move on to another opportunity?
Hey Ryan. The issue you bring up is a concern. However, it may be worth the risk. Since we only tested two filters, testing different filters is hardly a concern. There are many more logical and simple filters which could be tested. Only after my usual battery of filter testing would I begin to see this concept as unproductive and move on. Yes, this is subjective but if you find yourself spending more time than usual trying to make an idea work, that’s when it may be time to move on. Putting a time limit on your development process may help. Maybe you find that most good ideas pan out in about 2-3 weeks of work. Thus, if you can’t get an idea to work within 4 weeks it may be time to move on. However, we are not there yet.
Maybe cutting the profit target without waiting for the gap to fill should improve the odds.
http://nightlypatterns.wordpress.com
After the first oos test that data sample became part of the in-sample and it makes absolutely no sense to refer to it as an oss in subsequent tests. This is too basic to dismiss and one major cause of trader failures.
I would say undercapitalization and/or poor risk management is numero uno when it comes to traders failing.
That two, you are correct. But data-snooping of the kind you did makes things a lot worse and you do not want that. Obviously, you have to admit that your second oos test is not an oos test.
Yes, this was discussed in the article. There are different risks based upon what actions we next take.
Having experience with or ‘feel’ for a market certainly cuts down on the process time. I also have a simple battery of high level tests that I run which assists in determining if my trade idea as any merit.
Don’t give up on gap size filter. Take a look at short and long term market environment and various seasonality filters.
https://daxgaptrading.wordpress.com/