CSAT/NPS in comp plans

Effie Mansdorf

Curious is to how others measure CSAT and NPS scores into comp plans. How to you approach the following challeges:

1. No response from customers ( mark it as a negative response?)
2. Customers who have 1 strong respondor only vs. customers with multi reponses - scews the result.

Are there any criteria you put into place for the calculations?

Thanks,

Effie

Matt Myszkowski

Hi @Effie Mansdorf,

Good question & I am considering this now at my new company, but have experienced this in the past.

Firstly, I always agree a response rate for this goal to be "active". So for example, you have to have a 50% RR before we consider & reward you on the actual C-Sat/NPS score.

Secondly, I think this comes down to asking the right questions at the the right time, and only measuring & rewarding your CSM on areas they can influence or impact. So if there is a specific question such as "To what effect did your CSM enable you to achieve your outcomes?" (or something like that!), then measure that. But you cannot fairly measure them (and therefore penalise them) on questions relating to things outside of their control such as the product quality or support expereince.

Does this help?

Regards,

Ed Powers

Happy Thanksgiving, @Effie Mansdorf! While we wait on turkey watch, I thought I'd chime in. 

Agree with @Matt Myszkowski that it's best to use CSAT instead of NPS and ask specific questions about factors CSMs can impact.

Additional thoughts for whenever managers use metrics to evaluate individuals or pay them bonuses:

• Beware the Law of Large Numbers. Most of what you see in the numbers is noise, not signal, because sample sizes will typically be small and standard deviations will be large. That's especially true when it comes to customer satisfaction and NPS measurements. As a result, you will have wide uncertainly unless you have a very large number of samples. 
• Beware the Central Limit Theorem. If you plot a histogram of the rating distribution by CSM, you will see the standard normal. That means at least three independent variables are combining to produce the result, the CSM + 2 others. So while you can easily show scores by CSM, what you are actually measuring is the process, not the person. That means any individual differences you reward or recognize are most likely due to chance. 
• Respond to the numbers appropriately. Mathematically speaking, management attention is only required when a CSM's numbers differ by more than 2 standard deviations from the mean, which suggests a non-random result. If in the positive direction, learn what the CSM is doing differently and share best practices with the rest of the team. If negative, figure out why they are struggling. 

In my view, managers shouldn't manage people. Instead, they should lead people and manage process. That means the focus should be on continuous improvement, raising the average performance for the process (executed by the entire team) and engaging employees in that effort. I recommend practicing Lean Six Sigma method. Consider tracking the process metric instead of using numbers to rate and rank individuals. And consider paying a team bonus for achieving sustainably better results through better processes and better execution. 

Hope that offers a different viewpoint.

Ed

------------------------------
Ed Powers
Consultant
------------------------------
-------------------------------------------
Original Message:
Sent: 11-26-2020 04:11
From: Effie Mansdorf
Subject: CSAT/NPS in comp plans

Curious is to how others measure CSAT and NPS scores into comp plans. How to you approach the following challeges:

1. No response from customers ( mark it as a negative response?)
2. Customers who have 1 strong respondor only vs. customers with multi reponses - scews the result.

Are there any criteria you put into place for the calculations?

Thanks,

Effie





------------------------------
Effie Mansdorf
Global Head of Customer Success
------------------------------

Scott Snow

Hi Effie,

A: Great question about non-responses because this is a common problem with no easy answer. I'll share some things I've tried in the past that have worked. The effectiveness will depend a great deal on who you are surveying and other circumstances. If this is a new customer and your onboarding process made it very clear that this was important to you and your ability to delight your customers, then I would say this is an indicator that something may be wrong. After all, if they love your solution and love your service, and know it's important, wouldn't they want to (or at least feel obligated) to respond? I would have someone with a degree of separation from the customer (not the CSM, maybe the head of customer success operations or the head of CS) give them a call to find out why they did not respond and encourage them to do so. Ask them if they can do it immediately after the call or by EOD. Try to get a commitment. During that call, you may also learn something really important that you can put an action plan around.  However, you may simply learn that they just don't normally do surveys or they were exceptionally swamped that week.

Human led surveys can skew the results, but here are other options that I recommend if your volume is low:

• Have the head of CS Ops actually survey the customer by calling them and walking them through the questions and documenting their responses
• If the customer doesn't stick to the script then simply document their feedback. Try to get responses on all critical areas: Product, Support, CSM, Enablement, etc. Then extrapolate that into a score for each section of the survey.

I would not do this for NPS since that is a more consistently defined metric that is used to compare your company with others. 

Only if the CSM has large numbers of customers and you have enough historical data to know what a no response means in aggregate, would I consider defaulting a negative  CSAT score. See @Ed Powers response based on statistics.

Lastly, if you assign a score based on the conversation I would have a notification back to the customer to help keep your team honest. It should thank them for the conversation and let them know the score(s) assigned. 

Q2: For CSAT, I always average multiple responses for a single customer. For each CSM, I then average those customer-specific averages for their accounts. For a manager, I average the customer averages under them (Not the CSM averages). etc. 

Does this at least give you some ideas?

Scott

Effie Mansdorf

This certainly does help Scott! Since the Survey will be going out on a quarterly basis, it's not realistic for me to call every customer of ours to follow up with them. However, averaging multiple responses from each customer with a minimum response rate seems right.

 

-------------------------------------------
Original Message:
Sent: 11/27/2020 9:43:00 AM
From: Scott Snow
Subject: RE: CSAT/NPS in comp plans

Hi Effie,

A: Great question about non-responses because this is a common problem with no easy answer. I'll share some things I've tried in the past that have worked. The effectiveness will depend a great deal on who you are surveying and other circumstances. If this is a new customer and your onboarding process made it very clear that this was important to you and your ability to delight your customers, then I would say this is an indicator that something may be wrong. After all, if they love your solution and love your service, and know it's important, wouldn't they want to (or at least feel obligated) to respond? I would have someone with a degree of separation from the customer (not the CSM, maybe the head of customer success operations or the head of CS) give them a call to find out why they did not respond and encourage them to do so. Ask them if they can do it immediately after the call or by EOD. Try to get a commitment. During that call, you may also learn something really important that you can put an action plan around.  However, you may simply learn that they just don't normally do surveys or they were exceptionally swamped that week.

Human led surveys can skew the results, but here are other options that I recommend if your volume is low:

• Have the head of CS Ops actually survey the customer by calling them and walking them through the questions and documenting their responses
• If the customer doesn't stick to the script then simply document their feedback. Try to get responses on all critical areas: Product, Support, CSM, Enablement, etc. Then extrapolate that into a score for each section of the survey.

I would not do this for NPS since that is a more consistently defined metric that is used to compare your company with others. 

Only if the CSM has large numbers of customers and you have enough historical data to know what a no response means in aggregate, would I consider defaulting a negative  CSAT score. See @Ed Powers response based on statistics.

Lastly, if you assign a score based on the conversation I would have a notification back to the customer to help keep your team honest. It should thank them for the conversation and let them know the score(s) assigned. 

Q2: For CSAT, I always average multiple responses for a single customer. For each CSM, I then average those customer-specific averages for their accounts. For a manager, I average the customer averages under them (Not the CSM averages). etc. 

Does this at least give you some ideas?

Scott