Are Customers’ Own Words Better than a Rating Scale?

Buffer

I recently developed a methodology that combines both a structured and unstructured measurement approach.The methodology is based on a single open-ended survey question in which respondents are asked to provide one word that best describes the brand of interest. From this one word, you learn two things about your customers: their sentiment and the reasons behind their sentiment. In today’s post, I will illustrate the use of this methodology by comparing wireless service providers.

Sample

The sample of respondents were solicited from Amazon Turk to complete a customer survey about their current wireless service provider. I received 460 completed survey response. I was able to compare five service providers that had ample sample size (n > 15). The customer survey contained several questions, including a standard customer relationship survey (see Customer Relationship Diagnostic) and an additional question, “What one word best describes your wireless service provider?”

Metrics

Two measures were used to compare the wireless service providers: 1) overall rating of customer satisfaction in which customer provide a rating to indicate their level of satisfaction from 0 (extremely dissatisfied) to 10 (extremely satisfied) and 2) customer sentiment in which customer sentiment is calculated by applying a sentiment score to each word. The sentiment scores for particular words were derived using existing data from several large annotated corpora. This mapping of sentiment values to words constitutes the Customer Sentiment Index. The CSI can range from 0 (negative sentiment) to 10 (positive sentiment).

Results

Figure 1. Words used to describe wireless service providers. Words in larger fonts indicate word was used more frequently across respondents compared to words in smaller fonts.

First, I examined the overall ratings across all five wireless service providers. The average satisfaction rating was 7.3 and the average CSI was 6.9. The correlation between overall satisfaction and CSI was r = .71 (p < .001). This makes sense. Customers who reported higher levels of satisfaction also used more positive words to describe their provider.

The frequency of words that were used by the all respondents can be seen in the word cloud in Figure 1. Words in large font indicate the word was used by more respondents than words in small font. Overall, the most popular word used to describe service providers was “Good,” “Reliable,” “Service,” “Best,” and Expensive.”

Figure 2. Comparing wireless service providers on sentiment and overall satisfaction

When comparing the five service providers, we see statistically significant differences across the providers (see Figure 2). Cellular One and Virgin Mobile received the highest sentiment scores (i.e., CSI) while T-Mobile and Verizon received the lowest sentiment scores. The results were about the same when the comparison was made using the satisfaction ratings.

Understanding Context

Figure 3. Comparing Cellular One and Verizon Wireless on the best words used to describe them (by their own customers).

While the “one word” methodology allows us to quantify differences across different entities (in this case, providers) via the CSI, I wanted to examine the actual words that were used by providers to better understand why the differences might occur. To illustrate this difference, I compared two providers, Cellular One and Verizon. I chose these two providers because they represented the two providers with the biggest difference in sentiment (diff = 1.5). Upon examination of the list of words (see Figure 3), we see that Cellular One customers frequently used words like “Good,” “Best,” and “Service.” Verizon Wireless customers, on the other hand, frequently used the word, “Expensive.” Based on this examination, I suspect that service quality is responsible for Cellular One’s higher sentiment marks and that high cost (e.g., “Expensive”) is responsible for Verizon Wireless’ lower sentiment marks.

Summary

I illustrated how a single word response in a customer survey can be used to both quantify customer sentiment toward a brand (via Customer Sentiment Index) and identify the drivers of customer sentiment toward a brand (via word content). The results of this analysis suggest a couple of things: First, the CSI, able to distinguish among a set of providers, has the necessary precision (e.g., reliability) to detect differences across customer segments. Second, the overall satisfaction question that uses a rating scale to measure attitudes may be redundant to the “one word” question that measures customer sentiment. The results using the overall satisfaction rating were comparable to the algorithm-derived score of the CSI. It appears that allowing customers to use their own words to describe a brand gives you more information than requiring them to use a rating scale.

Article originally appeared on CustomerThink.