The moment we say predictive model, all analysts will immediately think of regressions, chaid, random forest, and GBMs. The methods are mentioned here in increasing order of number of variables with increasing difficulty to provide insights but increasing power of prediction, generally.
The major challenge with these models is that even for some interpretable data variables (as against latent variables) not all analysts will agree perfectly with the same interpretation why the sign or the magnitude is right for the predictor variables in these models.
While hard core predictive analysts will pass that worry easily because prediction is the most important thing in their operations, the marketing, psychologists, sociologists, educationists, investment and financial services people could be very easily made to think hard on these models, and possibly reject the models.
One way to over come these challenges is to balance these with two types of models to express the phenomena.
The insight models are those types of models that group variables together and explicitly drive analysts to dig for latent variables that explains the predicand.
The major point here is that typically these latent models are developed upto few dimensions so that we kind of understand and explain the top latent variales that are not more than 5 or 10 (even 10 is an anathema) though it may explain only less then half of the variation in the data. It is also generally acceptable that if we can explain the insights with top three variables that supports consistent credible interpretation of business operations, the executives are very happy and very ecstatic – they could explain the challenges of operations to others in turn.
If done right, the analysts, strategists, and executives are all could be partying together!
Welcome to the new world of Insight Models – These are fascinating and provide a direct implication to create tons of insights for companies.
One of the results of insight models I found is that there is an insight model for every predictive model where the predictive power of the insight model is as good as the best selected predictive model. It may be difficult to come up with more and more latent dimensions but I will say after top 5 latent dimensions you can assign the latent dimension interpretations to the rest of the variables as the variable that dominates.
Do you know why I say that every predictive model has equivalent insight model that is as predictive or the traditionally known predictive model? Moreover, what method would you follow and why it is a new brave world in the light of big data? Do you really need to balance insight model with predictive model or a clever modification and the communication will sell this concept?
There are lots of other insightful questions that follow this.
Note that this is not anything to do with strategic approach to model building. This is an essential part of modeling so that we develop and utilize the best model in the most practical way.