This article will show you my best tips to crush curve fitting while building your trading system. Doing so will help you create systems that work on the live market.
As you know, curve fitting is a danger that all system developers must constantly be aware of. Curve fitting destroys your trading system and can give you false hope that your system will work on the live market, which results in you losing your hard-earned money.
Curve fitting often hits novice system developers who need to learn about it. This can be particularly confusing and disheartening as you spent so much time on the strategy only to have it fail on the out-of-sample or new live data. But curve fitting can creep into the development process of professional developers. So, it's your job to be aware of curve fitting and how to avoid it. You must be vigilant about it.
Let's dive into how to minimize this common and costly problem.
Curve Fitting vs. Overfitting
First, let's clear up some terms.
When people in the trading world talk about curve-fitting, they refer to a trading system's negative behavior when moving to the out-of-sample data segment. This often results in a failing out-of-sample performance.
The term curve fitting is not the correct term. A better word is overfitting. I may mix curve fitting with overfitting in my work, which is sloppy. However, because so many associate curve-fitting with overfitting, I may continue to use these terms interchangeably.
From a mathematical standpoint, curve-fitting attempts to fit a line or curve through a series of data points. That's not what we are doing with a trading system. Regarding trading, we are attempting to optimize a trading system's performance. Often this is a crucial performance metric, such as net profit vs. drawdown. We do this by trying to introduce filters and optimize various parameters.
As traders, we depend upon finding market edges we can exploit. All successful trading relies on recurring patterns found in the historical market data and using those patterns to make a profit. This is true for discretionary traders as well as system traders.
However, much of the historical market movement is noise or random action. So, the problem arises when we over-optimize our system to the historical data, and our system begins to key into patterns within the noise instead of exploiting a true market edge. Finding a pattern in the noise can produce great-looking equity curves on historical data, but the equity curve can break down quickly on out-of-sample data.
Put another way. We want a trading system that exploits a true market edge and avoids keying off patterns in the market noise. The idea of separating a true market edge from the noise is what this game is all about.
This strategy is likely overfit to the historical data. A great example of a "curve fit" trading system.
Overfitting occurs when a model is excessively complex and has too many parameters. If you find yourself adding filter upon filter, you may be overfitting. Too many filters or parameters will likely create an overfitted model that needs to be more generalizable to new data. Put another way. You're likely building a trading system that can only trade on the in-sample!
How to Prevent Overfitting When Building Your Strategy
To avoid overfitting, let's cover what you should do during your strategy development.
1. Don't Attempt to Build The Perfect-Looking Equity Curve
We'll start with a mind-shift change. This one took me a year or two to fully accept, but it's so important.
The novice strategy developers believe they should build perfect-looking equity curves. I used to think that. I remember spending hours or days on a single strategy, adding more filters to make a great-looking backtest.
Remember, your job as an algorithmic trader is not to build a perfect-looking equity curve. That's easy to do. Your job is to construct a strategy that can generalize to new data. This means following some critical steps to avoid overfitting. The remainder of my recommendations will help you do just that. Some of these steps will feel unnatural or counterintuitive. That's why so many people fail at trading.
2. Have Enough Proper In-Sample Data
Have enough in-sample data to generate hundreds of trades. I suggest 300+ trades. Your historical data should span different market regimes (bull/bear). Don't have your in-sample data segment only span a bull market. Picking the ES dates between 2009 and 2019 shows a clear bull market. You want your in-sample data to span different regimes so your trading system is exposed to these various regimes.
3. Limit Degrees of Freedom
Degrees of freedom is a fancy word for the number of pieces of information in the data. It's determined by the number of parameters in the model relative to the number of observations. We won't get into the details of degrees of freedom and keep this very practical.
We generally want to keep the degrees of freedom to the lowest possible. As algorithmic traders, we can count the degrees of freedom by counting the optimized parameters used by our strategy. The more degrees of freedom in the data set, the more likely it is to overfit. Adding more filters and parameters to optimize allows your strategy to train on the noise of the data. These types of patterns will never predictably repeat.
Suppose you can build a trading system with no parameters. Those do exist, but you will have a few parameters most of the time. Starting in algo trading, you should keep the parameters below 4. That may be challenging to do. Once you gain some experience, I've seen evidence that up to 6 parameters can be acceptable.
4. Don't Spend Too Much Time On One Systems
This technique is called Early Stopping. This is another fancy term used in machine learning. The problem is that a trading system can train too long and overfit. What does this mean for you and me as system traders? Don't spend too much time tweaking a strategy. A good strategy will work right away. You may be overfitting if you spend many hours daily on a single strategy. If it's not working right away, move on. Kevin Davey stated, don't torture your data! I agree.
5. Optimization Done Right!
Optimization can be a real source of fun when building a trading system. Seeing how optimizing the parameters gets that perfect-looking equity curve is fun! But as we now know, it's not the right thing to do.
Optimizing can be a strong source of curve-fitting. However, you can significantly reduce the risk by following a few key points.
- You want to avoid picking the best value when optimizing a parameter. Instead, look for a stable region.
- Ideally, you want to find a cluster or range of values where your system performs well.
- Abandon this parameter/filter if there are no stable regions. In this case, the optimization will often look choppy with drastic changes.
Below is a bar graph of a look-back period optimization.. The x-axis contains the look-back period, while the y-axis contains the trading system's total P&L. The blue arrow is highlighting what most people would select. The largest bar!
This is not how you optimize correctly. Don't pick the best value!
You don't want to pick the best value. In the case above, the best value is a four. Most notice algorithmic traders will pick that value. However, that's a big mistake. You want to pick a value within a stable range. See below.
In this case, above, we can see a stable range can be seen between values 10 and 26. I would simple pick the middle value 18.
Summary
When building trading systems, we want to develop simple strategies to generalize the out-of-sample data better. We want to build systems that generate signals from market edges, not market noise.When building a trading system, keep these points in mind:
- Don't attempt to build the perfect-looking equity curve
- Try to keep your system as simple as possible
- Limit 1-3 filters (rules).
- Key Idea + Rule 1 + Rules 2 + Rule 3
- Limit 1-3 parameters for optimization
- When optimizing, don't pick the best value. Pick values within a stable range.
Your historical price data:
- Have enough in-sample data to generate hundreds of trades for your key idea 300+
- Your historical data should span different market regimes (bull/bear)
- I like to use from 2007-2020 as my in-sample when building strategies for the stock index markets. It covers both bull and bear regimes.
Following these steps, you will likely significantly reduce the chances of overfitting your strategy, which means it's more likely to work on the live market.
Don’t you want a high degree of freedom? Typically, more degrees of freedom are associated with larger datasets and fewer constraints (parameters) in a model.
Larger datasets do not inherently mean more degrees of freedom. The degrees of freedom in a model are determined by the number of parameters or features that the model uses to make predictions or decisions, not directly by the size of the dataset. In the context of a trading model, degrees of freedom refer to the number of independent variables (parameters) that the model uses. Each parameter adds a degree of freedom, allowing the model to fit more closely to the data.
I put your question through Perplexity.ai and here is the results which I generally agree with.
Understanding Degrees of Freedom in Algorithmic Trading
=======================================================
Degrees of freedom in statistical models, including those used in algorithmic
trading, refer to the number of independent variables that can vary in an
analysis without breaking any constraints. In the context of trading systems,
this often relates to the number of parameters or variables that can be
adjusted in the model.
The Case for More Degrees of Freedom
————————————
1. Flexibility:
More degrees of freedom allow for greater flexibility in modeling
complex market behaviors.
2. Capturing nuances:
With more parameters, a model can potentially capture more nuanced
patterns in the data.
3. Adaptability:
Systems with more degrees of freedom might be better able to adapt
to changing market conditions.
The Case for Fewer Degrees of Freedom
————————————-
1. Overfitting risk:
More degrees of freedom increase the risk of overfitting, where the
model performs well on historical data but poorly on new, unseen data.
2. Parsimony:
Simpler models (with fewer degrees of freedom) are often more robust
and generalize better to new data.
3. Computational efficiency:
Models with fewer parameters are typically faster to train and execute.
4. Interpretability:
Simpler models are often easier to understand and explain.
Finding the Right Balance
————————-
The ideal number of degrees of freedom depends on various factors:
– Data availability:
Larger datasets can support more complex models with more degrees of freedom.
– Market complexity:
Some markets or strategies may require more complex models.
– Trading frequency:
High-frequency trading might benefit from simpler models for speed.
– Risk tolerance:
More degrees of freedom often come with higher model risk.
Best Practices
————–
– Start simple:
Begin with a parsimonious model and add complexity only as needed.
– Cross-validation:
Use techniques like k-fold cross-validation to assess model performance
and avoid overfitting.
– Regularization:
Employ regularization techniques to penalize overly complex models.
– Feature selection:
Carefully choose which variables to include in your model.
– Continuous evaluation:
Regularly assess model performance and adjust as needed.
In conclusion, while more degrees of freedom can offer greater flexibility,
they also come with increased risks and complexity. The goal in algorithmic
trading is not necessarily to maximize degrees of freedom, but to find the
optimal balance that provides good predictive power while maintaining robustness
and avoiding overfitting.