Correlation [相關性] and relationships between factors is always something we look for as actuaries. This allows us to predict future outcomes more easily. How do we determine these relationships? Just by looking at historical data [過去數據] or observable information. You might be thinking, more factors and more information might be always better for making predictions. While generally this might be true, don’t fall into the trap of overfitting [過度擬合].
Think of house prices in your city. If I asked you to guess the price of a random house that I picked in the city, how would you guess? Pretty difficult right? But if I then said that the house was maybe 1,000 square feet, you would look at the correlation between house prices and house size, then make quite a reasonable guess. What you’ll find is that there is a general trend [傾向] of larger houses being more expensive. However, you will definitely see some houses which don’t fit the trend exactly. Maybe there’s a house which is only 300 square feet but is double the price of a 400 square feet house. There may be reasons for this, such as the house being closer to the city centre, but you won’t know unless you get more information.
Overfitting happens when you try to fit your trend to every single data point. In the example above, overfitting is when you assume that house prices go up with house size, but specifically make an exception that the price doubles if it is exactly 300 square feet. It may be a silly example, but this happens in insurance especially when it is not obvious whether some points are genuine exceptions [真例外] or not. The death probability of a person is very low and goes up with age, but there’s a high probability of death at age 0. A perfect example of a genuine exception!
Similar to last week, I would advise all actuaries to utilise their common sense when doing data analysis. Don’t let overfitting be your downfall!
Blog 42: Overfitting by Xavier Lo, FIA, FRM, MBA
About the Author
