The Pitfalls of Using Predictive Models

Buffer

I joined my friend’s fantasy football league this past season. I was skeptical to join at first. My friend’s league had been together for 7 years, each participant with deep knowledge about nearly all the NFL players and the game. I, on the other hand, have not followed NFL football for nearly 20 years and had only a superficial knowledge of only the popular NFL players. Given my lack of knowledge, I thought that I could use predictive modeling to help me pick my fantasy team during the draft.

Predictive Modeling

Predictive modeling is a process by which a statistical model is chosen to best predict the probability of an outcome. The use of predictive modeling was illustrated in the excellent movie, Moneyball, in which Billy Beane, the manager of the Oakland Athletics used predictive modeling to select the players for his team. Working under a limited budget compared to other teams in the league, his predictive model identified baseball players who were undervalued by the other teams yet were predicted to get runs/points. Those would be the players he selected. As a measure of Billy Beane’s success, the Athletics, in 2006, were ranked 24th of 30 major league teams in player salaries but had the 5th-best regular-season record.

We used Yahoo!’s Fantasy Football service as the platform for our league. The site provide a predicted overall score (the total score expected across the entire regular season) for each NFL player. I used this predicted score as the basis for my draft selection, selecting the top player (most points in his position) each time it was my turn in the draft. I continued to use Yahoo!’s predictions each week when selecting my team lineup for that week’s games.

My Performance

Our league included 12 teams. Out of 12, I came in last place. No matter what metric of performance we used to rank the teams at the end of the season, my team was the worst. My team had the lowest Win/Loss ratio. My team scored the fewest points. My team gave up the most points. Although I did not expect to win the league, I did not expect to get last place. Where did I go wrong? Here are two reasons that could explain my poor performance.

1. The Model Provides Poor Predictions

The value of a predictive model is measured by how well it predicts an outcome. I did (do) not know how well Yahoo!’s model predicts actual total scores (but something I am going to calculate next season). It is quite possible that the correlation between predicted and actual end-0f-season point totals is low. Even if this correlation was non-zero, the model still may not provide much forecasting power on which to base player selection decisions. Even a correlation of .50 (rather good for predicting any type of behavior) means that you can only explain 25% of the variance in end-of-season points. Perhaps my use of Yahoo!’s predicted scores did not help me select the best players because they were simply not good at predicting actual end-of-season scores.

Yahoo!’s predictions of player’s end-of-season point totals is offered with no additional background information about what variables they use or how those variables are combined to make their predictions. Information about how the model was built (subjectively, statistically, both) would help users evaluate the quality of the model. Were the selection and weighting of variables chosen to maximize the predictive power? Were key player variables excluded (under-specified model) in the development of the predictive model?

2. I Lack Content Knowledge

My lack of knowledge about the NFL players could have impacted my performance in two ways: 1) by  limiting my ability to use Yahoo!’s predictions correctly; 2) by limiting my ability to augment Yahoo!’s model with unique, nuanced information about players/games. As I continue to play fantasy football, I expect my performance will improve based solely on the information I learn through watching games.

Predictive Modeling and Customer Experience Management

We use predictive modeling in customer experience management (CEM) to describe the likelihood that customers will engage in certain types of loyalty behaviors (e.g., advocate, open up more of their wallet, stay for a long time). Specifically, using CEM-related data, we create models to predict the level of customer loyalty given different types of interactions with and different levels of satisfaction with the customer experience. We know that both attitudinal and operational metrics impact customer loyalty. A predictive model of customer loyalty that includes important variables maximizes the value (predictive power) of your model.  These predictive models are used to create different what/if scenarios to understand how improvements across different touch points will impact customer loyalty.

Summary

When executives make decisions about how to improve customer loyalty, they rely on many sources of information, including personal knowledge, past experiences of similar situations and even predictive models of expected performance. While I admit to relying on the use of  statistical modeling in my decision-making process, I showed how my sole reliance on this type of information did not lead to any success. My experience in playing fantasy football illustrated the importance of:

1. Knowing how well the model predicts the outcome of interest. How well does your predictive model explain customer loyalty?
2. Knowing the variables in the model. Does the model include a comprehensive list of variables to predict the outcome? A simple list of all the variables will do. Ensure the measures used to create the model are reliable and valid. Poor metrics will limit both the predictive power of your model and the value of it to your company.
3. Knowing how the model was developed. Was the model built on a statistical basis? If and when judgement was introduced into the model, when was it introduced?

Bonus: Commentary on the NFL Commentators

I watched many football games on TV over the season. I saw a lot of receivers make spectacular catches. I witnessed many running backs make unbelievable runs. Even though I reestablished my love for the sport, I, unfortunately, had to suffer through the NFL commentators’ remarks about the game, during the game.  Besides stating the obvious (e.g., “To win, they’ll have to score more points than the other team.”), the commentators might be using subliminal messaging/imagery to to attract a different segment of the viewing public. Here is my top 10 list of actual remarks made by NFL commentators this past season:

10. They “double-teamed” him.
9. It’s all about the penetration.
8. He came into his own.
7. He takes advantage of the opportunity he gets.
6. You just ride that guy.
5. Watch him squirt through this hole.
4. That’s an impressive sack.
3. That guy’s got really good ball skills.
2. You cannot advance a muff.
1. Comes right up the A gap.

Go local sports team!!!!