Fluidity of a Health Score

Hugh Walker

Hey all, we're having a discussion internally around a health score we're looking to build out and I'd love to hear peoples thoughts.

Once you've created a health score, do you gain or lose value by having that as a fixed scoring & weighting or as the product changes, and the actions you want to encourage from customers evolves, is it better to iterate the health score too?

Michael Haygood

Hugh,

   When building health scoring methodologies, I always iterate consistently as new information is available or more assumptions are validated or invalidated, you should think about the health score as a living, breathing, ongoing thing. I do limit the updates to quarterly or semi-annually because of the potential impact on CSM compensation and because our process to backfill and adjust the historical trend line with the updated formulas or new factors is sometimes volatile. 

-Michael

Will Pagden

Hey Hugh,

Not sure there is a right or wrong answer here. I personally review mine monthly with our Senior Leadership Team, based on changes to the product, processes and so on. We are a scale-up so things are constantly changing. Whilst I have KPIs for my team on health scores I dont give bonus based on it.

The key thing for me is to look at my churn analysis, does the healthscore when they churned represent the true picture, if not what changes can I make to be more accurate. 

As you get further established with it, you will likely find you only change them once a quarter/ twice a year but at the beginning, its somewhat trial and error.

Will

Ed Powers

@Hugh Walker, it depends on if you are using objective data or subjective evaluations to establish your health scores.

If you are analyzing data to construct your health scores, then you have done so using statistical regression or numerical methods such as random forests, boosted trees, or other algorithms. All of these techniques use historical data to train and test the predictive model, so if a fundamental change has occurred in your business, then the model becomes less predictive. Generally analysts track predictive power over time, and if significant degradation occurs, then a new set of historical data are used to generate an updated model. 

If you lack good data or are using a subjective approach to health scores, then your predictive accuracy is probably not very good to begin with. Even in this case I would recommend tracking and adjusting whatever weighting you use only when the data shows it's appropriate to do so. You can use statistical control charts to detect when that time comes. Doing anything less is just 'tampering' with the system, adding more variation and reducing whatever accuracy you already have.