Handling seasonal usage within a health score

Allastair Meffen

Hello everyone,

As we move into the summer months we are seeing our usual seasonal level of usage start to drop, but this has a direct affect on the health score for all our customers.  We will get a nice uptick starting in late August going into September when the kids return to school of course.

I am wondering if anyone has thought of a way to take out the seasonal affects of usage from their health score.  We have 3 of our 5 health score metrics tied to usage (% of activate users, avg use by user, # of activate days in the product), all of which are on a 90 day look back.

While it is nice to see the steep increase of health scores coming out of the seasonal disruption it doesn't give an accurate picture of the health of the book.

Any suggestions would be appreciated.

Regards,
Allastair

Ed Powers

Allastair--

What have you learned about the relationship between seasonal usage (compared with non-seasonal usage) and resulting customer behaviors? If seasonal usage is not a factor, (i.e. churn is not dependent on seasonal usage) then it should have no bearing on your customer health score. If it is a factor, then you would condition on whether the usage is seasonal and then include that information in your health score calculation. 

If logo or product churn is your outcome of interest, then perhaps the simplest way to test for dependence is to select homogeneous customer groups and compare churn rates vs. seasonal usage in a 2x2 contingency table. This would require classifying usage as 'seasonal' or 'non-seasonal' and tallying what customers decided (renew or churn) at that time or within a reasonable preceding time period. You would then test your hypothesis using Pearson's Chi-Squared analysis, which is easily done in Excel or Google Sheets. It would look something like this: 


You would generate a table of expected frequencies and then use the CHISQ.TEST() function which returns a p-value. If p<0.05, then you would reject the null hypothesis and conclude churn depends on seasonal usage. Note that the larger the number of samples you have in each group, the greater the power of the test, and if any cell has less than 10 observations, you would use Fisher's Exact Test which uses the hypergeometric distribution instead of the chi-square. 

Of course, deriving your health scores directly from your data may not be the approach you originally used--most CS teams do it subjectively and have poor predictive accuracy as a result--but this could be a first step in walking down a path to having more reliable customer health scores. 

I'm happy to talk this and other data-driven approaches in more detail if you'd like. Shoot me an email at [email protected] and we can set up a time. 

Ed
 

Allastair Meffen

Hi Ed,

Thank you for the feedback and we have not used Pearson's in the past so something that I can work on with our operations team.

Our health score is completely informed by data so that we can take out the subjectivity by taking a correlated approach on our historical data.  We start by bringing in ~15 different metrics that we believe relate to the customer health and then pair that down to only the top 5 most correlated.  In our current version we landed on the 3 usage metrics that I mentioned earlier along with whether they have integrated into a CRM/Data Lake plus whether they have renewed with us in the past.

Appreciate the response and will let you know how things go.

Allastair