Thanks much, Herman.

Herman

Given ES, you add two ticks per trade (1 buy & 1 sell ) that’s $25 right there. Then you add $5.00 for a commission fee. That’s $30 per trade for execution costs.

While this article does not talk specifically about out-of-sample vs in-sample, the same metrics apply. Not during all instances will OOS performance available when looking to buy a system however, this is an important step in testing. if you are developing a system you should always test OOS data as it gives you a better idea of how your system performs. The next step is to actually test it on live data. Many people are shocked to see their system fail to perform on the live market as bars form in real-time. Often this is due to an incomplete understanding on how bars are built tick-by-tick and how the trading code is executed against that data. This is very important for intraday trading systems.

” …is this not dependent on the nature of the strategy in question?”

Actually this procedure REVEALS the nature of the system. Is it made up of evenly sized profits (small impact of outliers = smooth equity curve with little variance) or a few “homeruns” (big impact of outliers => rugged equity curve with high variance).

TK

P.S. Jeff, feel free to delete the first repost.

I am not sure if I understood your comment correctly, so forgive me if my answer should not match what you meant.

The purpose of the “cutting procedure” is to find out to what degree the test results were impacted by outliers. If there is a big impact then you have a high risk that the results of the SAMPLE (test) are NOT representative of the real life performance later on.

>>>” …is this not dependent on the nature of the strategy in question?”<< rugged equity curve with high variance).

>>>”For instance, given a sample size of 100 and an exreme outlier-dependent strategy with a 10% win rate, then cutting the top 10% means disregarding one single trade. Though, over an average of 10,000 trades, the top X% may on average be only slightly more profitable than the top X+n%, in the case of the sample size of 100 the top X% trade (i.e. the one single trade selected), may be many multiples more profitable than the X+n% remaining.”<<>>”In other words, what you describe, with certain types of systems and without a very large sample size, risks creating a counter-productive “black swan” style exclusion that is not representative of the intended effect of this process. Instead of mitigating the impact of outliers, you risk introducing a further outlier counter-measure.”<<<

I am not sure what you meant with this paragraph. What I talk about is eliminating the trades from the statistical evaluations. Of course you should NOT introduce any rules to your system that cut "home runs" short if they should really happen. My point is to see if the system can hold up if the excellent trades are a lot less frequent (in real life) than the sample may make you believe they might be.

Looking forward to your reply,
TK

@Jeff

Any comment regarding the lookback period vs. sample size? By the way: the system I described is called "Millenium Bull" and available for a few Galactic Credits at http://www.JabbaTheHut.com =:p

TK,

Thanks again for the thoughtful reply. So sorry for getting back to this thread days later. It was a hectic week last week. I’ve heard a lot of good things about “Thinking Fast and Slow”. I’m currently reading “The Big Short” and will add your recommendation to my reading list.

Of course, above should have read X-n% – there’s no “edit” button like on the forums!