The Reliability and Validity of the Consumer Financial Protection Bureau (CFPB) Complaint Database

For the present analysis, I downloaded the credit card complaint data from June 1 through August 29, 2012. The total sample size for this data set was 3978 (June = 1671; July = 1494; August = 813).

Criteria for a Good Customer Metric

Consumer complaints, a specific kind of customer metric, need to be evaluated along four criteria. A customer metric is good when:

1. The metric is supported with a clear definition of what it measures and what is does not measure.
2. There is a clear method of how the metric is calculated, including all items and how they are combined.
3. The metric has good measurement properties; that is, there is good reliability and validity evidence regarding how well the customer metric measures what it is supposed to measure.
4. The metrics are useful in helping drive real internal changes (e.g., improved marketing, sales, service) that lead to measurable business growth (e.g., increased revenue, decreased churn).

For a more, in-depth discussion about these four criteria, please see Four Things you Need to Know about Your Customer Metrics.

Consumer Financial Protection Bureau Credit Card Complaints

Let's evaluate the CFPB complaint metric using the above four criteria:

1. Definition

The definition of a CFPB complaint is clear and straightforward. A CFPB credit card complaint is an expression of dissatisfaction or discontent with the credit card experience, a negative customer experience. While a complaint likely impacts customer loyalty (e.g., likely to recommend, buy again), a complaint is distinct from customer loyalty. When submitting complaints, consumers can categorize their complaints into one of 33 different categories. As such, each credit card issuer can have 33 different complaint metrics, one for each type of issue/complaint (e.g., Billing disputes, APR or interest rate,  Closing / Cancelling accounts, Credit reporting).

2. Calculation of Complaint Scores

The calculation of complaint scores involves summing the number of complaints associated with each variable. Summed scores can range from a low of 0 (no complaints) to a high of N (the total number of complaints).

Comparing Banks

Because total number of complaints is impacted by the sheer number of cards in circulation, normative complaint scores were calculated for each bank by dividing the complaint scores by the total number of card holders for that specific bank / card issuer (based on the Nilson Report, a card-focused trade publication). These data were from February 2012 and include numbers for only large card issuers.

3. Measurement Properties

Reliability

I examined the relationship of the frequency of complaints for specific issues across time 1 (first three months of CFPB complaint data - see Table 4) and time 2 (June 1 through August 29, 2012).  The data are in located in Table 1. The correlations between the frequency of different complaints across time was .89, suggesting that the relative frequency of complaints remains stable over time. Top complaints in time 1 (e.g.,  Billing disputes, APR or interest rate, Identify theft / Fraud / Embezzlement) are top complaints in time 2 and less frequent complaints in time 1 (e.g., Bankruptcy, Balance transfer fee, Overdraft fee) are less frequent in time 2.

Validity

There are three different lines of validity evidence that help show what the customer metric is measuring, I will present criterion-related validity evidence regarding the complaint data. For top US banks / card issuers, I examined the correlation of their normative scores with an independent ranking of banks / card issuers. JD Power and Associates conducted a credit card customer satisfaction survey (data collected in June 2012) on large card issuers.  I correlated the JD Power and Associates satisfaction ratings with the CFPB normative complaint metrics across 10 US banks / card issuers (for all complaints, Billing disputes and APR or interest rate). Table 2 contains these data.  These 10 banks were selected because they had ample data from both the CFPB and the JD Power and Associates credit card customer satisfaction survey.

The correlations between the satisfaction ratings and the three normative complaint scores are -.44 (all complaints), -.57 (Billing disputes) and -.35 (APR or interest rate).  That is, banks with higher complaint scores also received lower satisfaction ratings. To illustrate these relationships, I plotted the banks on their ranking of frequency of complaints (1 - low complaints; 10 - high complaints) against the ranking of banks based on their satisfaction ratings (1 - high satisfaction; 10 - low satisfaction). These figures appear in Figures 1 through 3. Based on these analyses, we see that billing dispute complaints, compared to APR interest rate complaints, appear to be more strongly associated with customer satisfaction.

4. Usefulness

I can see some practical applications of the CFPB complaint data. Specifically, these data can help banks track how they respond to different types of complaints and understand how their handling of complaints impacts customer satisfaction (see Figure 4). The data show that banks respond to late fee complaints primarily with some form of monetary relief to the consumer while other types of complaints are dealt with only an explanation.

Summary

The banking industry is opposed to publishing CFPB data on credit card issues; the American Banker Associations' spokesperson, Carol Kaplan stated, "Until somebody has had a chance to sort though it and figure out what's valid and invalid, it's a gossip column."

The current analyses, however, suggest that the CFPB credit card complaint database provides clear, reliable and valid information for banks about their credit card practices. The frequency of different types of complaints are fairly stable over time. Additionally, the normative CFPB complaint scores are related to credit card customer satisfaction ratings (from an independent source); banks with better complaint scores (lower number of complaints) receive higher satisfaction ratings compared to banks with poor complaint scores (higher number of complaints).

Additional Research Needed

While the current findings provide some evidence regarding the reliability and validity of the CFPB credit card complaint database, more research is surely needed to validate the current findings. Additionally, more research is needed to understand the usefulness of these complaint data.  Some potential areas of future research including the application of text analytics to CFPB verbatim comments. To be sure, while not provided in the public database, verbatim comments from customers are given to the specific bank who issued the credit card. Open-ended comments from customer surveys are commonly used to identify specific ways to improve the customer experience. Perhaps these CFPB verbatim comments could provide some additional insight about how banks can improve their practices around credit cards. Does customer sentiment behind the complaints provide useful information about the customer experience? Do improvements in the credit card experience correspond to improvement in sentiment scores?

Also, it would be interesting to study different types of complaints. The current analysis showed that billing disputes might be more closely linked to customer satisfaction compared to APR and interest rate complaints. As/If billing dispute complaints decrease over time, do other types of consumer complaints of credit cards become more important in predicting customer satisfaction?

Finally, based on a very informal polling of the audience, I found that a few bank representatives indicated that they incorporate their CFPB complaint data into their company's formal customer feedback/experience management program. I would be interested in understanding how banks are using these CFPB complaint data as part of their larger program to improve the customer experience.

I would love to hear what other research ideas people have regarding CFPB complaint data.