What Must You Ask? Comparing Two Customer Survey Approaches

Buffer

A new customer survey approach that requires only two open-ended questions provides customer insights that are comparable to a longer, more traditional customer survey. This post presents the new survey approach and outlines its benefits to your enterprise.

Customer feedback plays a central role in customer-focused programs. Annual customer surveys are the most popular ways of collecting customer feedback. While we have seen advancements in how we collect that feedback (e.g., web surveys), little has changed with respect to what we collect (e.g., survey questions). In this post, I will present a new customer survey approach that requires only two (2) open-ended questions. To illustrate the utility of this new approach, I will present evidence that the two-question survey provides comparable customer insights to a longer, more traditional customer survey.

The first survey method is the traditional survey that typically contains 20+ questions, including loyalty questions, several CX questions and open-ended questions. The second survey method contains only two open-ended questions.

1. Traditional Survey (13 questions): In this example, the traditional survey contains 3 loyalty questions: Overall satisfaction, Likelihood to recommend and Likelihood to buy again (0 to 10 rating scale). Also, the survey contains 10 CX questions that cover various customer touch points (e.g., product quality, technical support, ease of doing business) (0 to 10 rating scale).

2. New Survey (2 questions): The survey contains the following two open-ended questions.

1. What one word best describes Company Name?
2. If you were in charge of Company Name, what improvements, if any, would you make?

I have been developing this new survey approach over the past couple of years, using both basic research and applied research methods. For example, using over 2,000,000 online reviews, I developed a sentiment lexicon to translate adjectives into numerical values of sentiment/satisfaction. Also, I used these two questions in several engagements in both customer and employee surveys.  You can read more about the development of this sentiment lexicon here and here, and you can see a sample report using this method here.

The Goal of Customer Surveys

Customer relationship surveys are typically used by business leaders to help make decisions about where investments will have the greatest impact on improving the customer experience. To help make these strategic decisions, leaders rely on three key pieces of information from their customer relationship surveys:

1. Current level of customer satisfaction – Surveys provide an index or metric that represents the level of satisfaction of your customer base. Before you start your company’s journey of self-improvements, you need to know how satisfied your customers are today. This metric can be used in any analytics efforts to study customer satisfaction.
2. Context of customer satisfaction – Surveys provide context for the responses. It’s important to understand your brand in the context of your customers’ experiences. How customers describe the company or their problems helps highlight the underlying conditions that drive their satisfaction and dissatisfaction and could generate improvement ideas.
3. Key drivers of customer satisfaction – Surveys identify the key drivers of satisfaction. When you invest in your company to make improvements, you want to optimize the ROI of those improvement efforts. Consequently, you want to invest in CX areas that  ROI by first improving business processes that reflect key drivers. Target improvements in areas that matter most to customers.

Comparing Two Survey Methods

I will examine two survey methods by comparing the outcomes on the three criteria above. In a recent client engagement, I conducted a survey for a B2B technology company using questions from both surveys approaches above. I analyzed the respective questions for each survey method to compare the results across the two methods. Because the two methods are based on the same respondents, I expect that the two methods would result in outcomes that are comparable.

Results

Below are the results of the two survey methods.

Traditional Survey (13 Questions)

Figure 1. Customer Satisfaction metric based on ratings. (click image to enlarge)

1. Customer Metric. We can calculate a numerical index easily simply by using the ratings provided by the respondents. As you can see in Figure 1, we can use this satisfaction metric to track changes over time. The results indicate that the level of satisfaction has remained relatively stable over three-year time span reported here.

If you are more comfortable using percentages in your dashboards, you can easily calculate percentages as your metric of choice.

Figure 2. Satisfaction ratings of customer touch points represent the context of customer satisfaction. (click image to enlarge)

2. Context. We can understand the context of the satisfaction ratings by examining the different touch points measured in the survey. Figure 2 contains information about the different touch points. Customers are more satisfied with product, technical support and training than they are with the web site, purchasing/invoicing/delivery and the sales process.

3. Key Drivers. We can identify the key drivers by conducting what is traditionally referred to as Driver Analysis. The results of this analysis appears in Figure 3. As you can see in figure 3, for each of the 10 touch points, I have plotted both the satisfaction rating as well as its correlation with our metric of customer loyalty (aggregate of overall sat, recommend and buy again).

Figure 3. Driver Analysis showing the key drivers of loyalty. (click image to enlarge)

The results of the driver analysis show that the key drivers were represented by two customer touch points: purchasing / invoicing / delivery and the sales process. These two areas represent the touch points that are both important to customers (as is indicated by the high correlation with loyalty) and in which the customers are not as satisfied as they are with the other touch points.

New Survey (2 Questions)

Figure 4. Customer satisfaction metric based on the responses to the question, “What one word best describes this company?” (click image to enlarge)

1. Customer Metric. The one-word question is used to calculate a sentiment (satisfaction) score. Using a sentiment lexicon (see here and here for the development of this lexicon), each word was transformed into a numeric value ranging from 0 (negative sentiment) to 10 (positive sentiment). This metric can be used to track customer satisfaction over time (see Figure 4) as well as test other hypotheses you might have (e.g., test differences across customer segments).

This method of calculating customer satisfaction (derived from word) appears comparable to the traditional method using ratings. Satisfaction remained very stable over the same time period. In fact, the correlation between the two customer satisfaction ratings for this survey was r = .52, showing there is some agreement between the two methods of measuring customer satisfaction.

Figure 5. Understanding context of satisfaction – Word cloud based on responses to question, “What one word best describes this company?” (click image to enlarge)

2. Context. We can understand the context of the satisfaction metrics by simply examining the words that the customers used to describe the company. You can graph the frequencies of the words in a Pareto chart to highlight the different rates at which words are used. I also like to present the results in a word cloud to highlight the popular words that are used to describe the brand (see Figure 5).

This summary provides a high-level picture of how customers describe your company using their own words. Note that many of the popular words found using this question were not found by using the traditional approach of asking the customers to rate standardized CX questions. This approach revealed potentially useful words like, “Reliable,” “Innovative,” “Expensive” and “Leader.” Not only do these words provide you context behind what is driving customer satisfaction, they can also be used to improve how to better communicate your company’s value in sales and marketing collateral.

Figure 6. Understanding context of satisfaction – What improvements would you make?

We can also understand context of satisfaction by examining the responses to the second question. For this question, the respondents indicated what they would improve if they were in charge of the company. A content analysis of the responses showed that improvement areas (touch points) reflected a wide range of topics (see Figure 6). The most frequently mentioned touch points needing improvement were product, price and purchasing / invoicing / delivery. The remaining touch points were mentioned far less (all in single digits).

Figure 7. Satisfaction drops in the presence of specific touch points.

3. Key Drivers. To understand the drivers of satisfaction, I calculated sentiment values (from the one word question) for each of the touch points mentioned in the second question (see Figure 7). I wanted to understand if satisfaction dropped for customers who mentioned a particular touch points. As you can see in Figure 7, there were four touch points that had a big impact on customer satisfaction. Satisfaction dropped substantially when the following touch points were mentioned: customer service, sales, communication and purchasing / invoicing / delivery.

Figure 8. “Driver Matrix” for two-question survey to help prioritize CX improvement opportunities. (click image to enlarge)

To identify the CX touch points that were the key drivers, I combined the two pieces of information for each CX touch point into a single chart (see Figure 8). This is somewhat similar to the earlier driver matrix with the exception that, instead of looking at the impact of each CX touch point, we are focused on the magnitude of problem for each CX touch point.

Based on this analysis, we see that:

• Red touch points indicate that customers mention these touch points as needing improvement more than other touch points and they are less satisfied when they mention these touch points. Touch points with these characteristics are key drivers. In this analysis, we found two key drivers: purchasing / ordering / delivery and customer service.
• Dark green touch points indicate your customers are not mentioning them much and, when they do, they are still satisfied; these are not high priority to improve.
• Light green touch points indicate many customer are mentioning them but, when they do, they are still satisfied. Improvements in these areas won’t have much of an impact on satisfaction.
• Yellow touch points indicate few customers mention these areas but, when they do, they are less satisfied. Watch these areas carefully to ensure you are not underestimating the magnitude of this problem. For example, it’s conceivable that only a small portion of customers recently went through the sales process, artificially lowering the percentage of customers who said sales needs improvement. To be on the safe side, you might consider the yellow touch points as “next areas to improve” after addressing the red touch points.

Summary

I wanted to compare two survey approaches to determine if a short survey (two questions) would provide comparable results to a longer survey (13 questions). Using the three criteria for good customer surveys, I found that:

1. Customer satisfaction metrics behaved the same across both methods. The two survey approaches resulted in similar findings in that satisfaction was fairly stable over a few years.
2. The context of satisfaction could be determined by both methods. The one-word question provided additional information about the reasons behind the satisfaction which could not be gleaned from the longer survey.
3. Key drivers (areas that would result in greater ROI) could be identified using both methods. The key drivers overlapped considerably across both survey methods (traditional survey: sales, purchasing / invoicing / delivery; new survey: customer service and purchasing / invoicing / delivery and potentially sales and communications).

By using a short customer survey with the right questions, you can improve your customer experience improvement program. Here are five primary benefits to using the method described here:

1. Save money. While the analysis of text responses does require extra work, that process is typically automated. Shorter surveys are more cost-effective to manage than longer surveys, requiring less resources on the part of your CX team and other interested parties.
2. Obtain reliable, valid and useful information about your customers. Insights gleaned from the shorter survey are comparable to longer surveys.
3. Hear from more of your customers. Customers are more willing to complete shorter surveys than longer surveys.
4. Minimize customer effort. Short surveys provides customers a quick and easy way to tell you what you need to do to win their business. And short surveys show you respect your customers’ time.
5. Identify operational best practices, track satisfaction and understand customer segments. The customer satisfaction metric can be used to track changes over time as well as test hypotheses about your customers.

Using two specific questions, I found that you are able to glean much useful information about your customers’ attitudes that can help improve your customer programs. Coupled with all the other information businesses have about their customers (think Salesforce, Google Analytics, Mixpanel, Marketo), a short survey can provide a comprehensive picture about the health of your customer relationship and help you determine an optimal improvement approach. Are you asking too many questions of your customers? It may be time to simplify your survey.

Article originally appeared on CustomerThink.