Churn Analysis

David Ellin

There's been much discussion around capturing and reporting reasons for churn. But, I haven't heard much about tracking churn back to ICP (Ideal Customer profile).

We've all had customers that should have never been sold our products. They're simply not a good fit and the effort it takes to make them successful sucks energy and resources away from customers that fit well.

I'm curious to know if anyone has looked at churn by relating it back to ICP to come up with a "% of ICP" as a predictor of churn, possibly bringing # of support tickets and average time to closure into play as indicators of % of ICP.

Here's an example.....bear with me here as I'm sure the formula could use tweaking....

The sales team closes a deal for ABC Company. The customer is a great fit for our products. During the first year, they open 4 support tickets but three of them are asking for enhancements/recommendations for new features. Only 1 ticket is a real support issue and is closed within the normal, average closure time (overall for the company).

The deal for DEF Company is sold at the same time. DEF has a few more issues but is a pretty good fit. During the first year, they open 27 support tickets all of which are for issues using the products. The average time to ticket closure is 2x the average time (measured against all tickets) - which indicates that their issues are likely not standard.

XYZ, also sold at the same time, has tons of issues and can't get anything to do what they want it to do. They average 1 ticket per week (52/year) and it takes the support team an average of 56 hours per ticket to close.

Using a formula like this: % of ICP = 100 - ((# support tickets per month/ave. overall support tickets per month) * (# hours [or days] to closure/overall ave. hours [or days] to closure))

If the average # of support tickets per customer per month is .7 and average time to closure is 16 hours per ticket:

ABC % of ICP = 100 - ((.33/.7) * (16/16)) = 100 - (.47 * 1) = 100 - .47 = 99.53%

DEF's % of ICP = ((2.25/.7) * (36/16)) = 100 - ((3.21 * 2) = 100 - 6.42 = 93.58%

XYZ's % of ICP = ((4.33/.7)  (56/16)) = 100 - ((6.19 * 3.5) = 100 - 21.67 = 78.33%

This analysis would provide insight to Sales and Marketing to target more companies like ABC and DEF and stay away from companies like XYZ.

Is anyone doing anything like this?

David Jackson

David

Not scoring in the way you described but I have implemented ICP fit scoring for lead qualification.  We scored different aspects of firmographics to calculate the ICP Fit.  The initial scoring grid was, in part, based on a churn and retention regression analysis.  We used to (don't know if they still do) periodically review the correlation between ICP score and NRR and tweak accordingly.  

I described this approach to one SaaS CEO and he suggested using ICP Fit Score to pay differential commission to new biz sales.  Don't think he ever did but that sounds like a great step to stop buying churn!

David Ellin

@David Jackson, I love the idea of an ICP Fit Score. Incent the right behavior and get more of the right results.

Will Pagden

Really interesting post @David L Ellin. I have previously used Churn analysis to tweak the ICO but this is next level! I’m not quite there yet, but will certainly refer back to this in a few months! 

gurd3v

I probably won't be able to talk through the formula as much, but what we have done is taken the churn data collected during our offboarding calls (which we conduct with about 95% of customers requesting to cancel) and tweaked our sales strategy. 

For example, we ID'd a certain segment in our market had a 3x higher monthly churn rate than another cohort. The primary reason they were churning was solved by addressing another segment in the market. As a business that's scaling, we decided to take the data collected, and changed our ideal customer profile to align with the lower churn rate customer. The entire company is rallying behind this shift and early data suggests it's paying dividends. 

This may be a long way of me suggesting - you can absolutely use churn data to change and adjust your ICP, but are you collecting that data regularly? Is the decision being made after a year-plus of collecting the data?

Hope this answers your question?

David Ellin

Thanks, @Gurdev Anand. This is a unique approach to me - changing the ICP to fit the churning segment. Was the ICP constructed incorrectly to begin with? After changing the ICP, did you find you had less fit in other segments?

gurd3v

I should clarify - we didn't change the ICP solely on our churn data. We also found our close rates and ACV was higher with a certain segment that we're now attacking with focus. Churn data just helped solidify the decision and was likely the higher weighted metric used to come to that conclusion. 

Our business has grown significantly over the last 5 years. When the business first started, I think the objective initially was to show proof of concept and product-market fit within the broader market. This was proven, and frankly we could have had a decent business if we continued on that path. But to scale, grow and dominate the industry, we felt like we needed to focus on a particular segment that still allows for us to dominate, but with more focus and intention. 

We now intentionally have a better fit with a certain segment, than we do with our not-ideal segment. 

Will Pagden

Wow @Gurdev Anand this is deep stuff. out of interest how mature is your company? I love the idea but we’re still too small to have accurate data. 

Brian Hartley

+1 @Gurdev Anand always dropping great stuff!

David Ellin

Thanks for the additional information, @Gurdev Anand. That makes sense.

gurd3v

@Will Pagden Business is about 6 years old, I've been with PatientPop for 4 years now. We started collecting this data about 2 years ago and are now implementing the change. We could have probably implemented it a year ago if we wanted to be super aggressive, but I think we wanted to be certain, hence the additional time spent to collect the data. 

Many within CS had been harping on this thought when I first started, but the stage of our business was focused more on top line revenue while still staying narrow in our market. Happy to talk more offline about your specific business and perhaps what learnings could be applied, if interested!

Matt Vadala

This is great consideration for the support org @David L Ellin  - and something I've seen missed in some areas in lieu of meetings outside of support interactions. The cases do indeed provide hard statistics that are easy to analyze and measure. I would be curious if there any other pieces of data to collect outside of these interactions to measure the customer's weight in the industry (how well connected) and if their methodology could be adopted in any conceivable way by Product. 

Of course, having worked with many curmudgeon clients, I completely understand the sentiment of not targeting or targeting customers away from those like XYZ.

Anita Toth

Love this idea @David L Ellin. It's got me thinking of a few things.

#1. Wondering if this could be useful for 90 day churn specifically

#2. Measuring long-term customers with this formula (maybe by adding a decay element)?

The idea isn't fully formulated in my head so I might not be that clear here.  Hope this makes sense.

 

David Ellin

@Anita Toth, makes perfectly good sense. My idea and formula weren't fully baked either. I had been fiddling with it for a while.

I love the idea of 90-day churn. If you have a very high number of support tickets in the first 90-days, it's likely the product is not fitting the customer's needs and there's a high chance of churn.

David Ellin

I agree, @Matt Vadala. One of my thoughts in creating the formula was to have hard statistics to share with the head of sales so that s/he understood the sales team was missing the mark and causing cost/churn/customer dissatisfaction to the rest of the organization.

Will Pagden

100% keen to hear more @Gurdev Anand! We are still in our first full year of dedicated GTM strategy. We formed as a byproduct of another company and CEO only bought in and dedicated focus since Feb this year.

Think I need to remember the crawl, walk, run approach and slow myself down!

gurd3v

Will reach out via LI to get some time in the books!

Anita Toth

@David L Ellin I'm wondering what other data would pair nicely with this formula once it's ready to deploy to make a strong case to VP Sales that would be compelling enough for them to consider changing who they sell to. 

Any thoughts?

Ed Powers

@David L Ellin, it seems @David Jackson 's approach is helpful for figuring out ICP segmentation post-hoc, but if you already have a well-defined ICP, I would suggest keeping it simple using an ICP fit rating on a scale of 1-5 where 5 = excellent (i.e. product meets the customer's need and the customer is ready and mobilized to make the change) and 1 = poor (i.e. just the opposite). There's a start-up called CogniSaaS that quantifies technical fit better, but a CSM making a subjective assessment during onboarding is probably enough to get you started. You would then use regression analysis to determine the predictive power of the rating.

You propose using support ticket data as a proxy for ICP fit, but that introduces other variables. For example, there's high skew--support response times are exponentially, not normally distributed, so mean values overstate the results. You should also avoid calculations using percentages when you have variable denominators. Statisticians say that when denominators vary by more than 25%, errors become excessive. Better to measure ICP fit directly and then study its impact on resulting churn behavior. 

David Ellin

Thanks, @Ed Powers. This is very insightful and gives us some great things to think about regarding approach and measurement.

Anita Toth

Thanks for the great reply @Ed Powers.  2 questions:

1. At what point would you think that bringing in automation to to qualify fit would be best? (So moving from CSM assessment of fit to something like CogniSaaS)
2. How long would you use ICP fit for a customer?  Only during onboarding? 6 months? And when would other scores like Customer Health come into play?

These are just rolling around in my head cause your response got me so curious. So I  thought I'd ask.  :-)

Ed Powers

Hey @Anita Toth! For #1, In my view, ideally ICP qualification happens in the sales process, even before Customer Success. My experience with salespeople, however, is that sales qualification, much less documentation, isn't a strong suit. A single, required field with a 1-5 scale in a deal qualification step in the CRM may be all one can expect. Perhaps the CSM can later validate after engaging the customer and using something like CogniSaaS or by adding their own subjective assessment. The only time I think it can be fully automated is when customers buy online and answer a series of questions. 

Regarding #2, it would depend on your assumptions. Is there a time dependency or a 'decay' factor? Perhaps. All customer needs evolve over time, so even a "bulls eye" ICP must track with the new value the provider develops and delivers over time. In my experience, ICP is a prime factor, but not the only one that contributes to customer health and the renewal decision. Others, such as the degree of adoption, reliability of the product, price, quality of support, strength and depth of trust, and replaceability can all contribute. Once again, the math can help figure out what matters and what doesn't.