In the Absence of Data, Everyone is Right

Buffer

I wrote a post last week that compared two ways to make decisions/predictions: 1) opinion-driven and 2) data-driven. I am a big believer of using data to help make decisions/predictions. Many pundits/analysts made predictions about who would win the US presidential elections. Now that the elections are over, we can compare the success rate for predicting the election. Comparing the pundits with Nate Silver, Mr. Silver is clearly the winner, predicting the winner of the presidential election for each state perfectly (yes, 50 out of 50 states) and the winner of the popular vote.

Summary of polling results from fivethirtyeight.com published on 11/6/2012, one day before the 2012 presidential election. Click image to read entire post.

Let’s compare how each party made their predictions. While both used publicly available polling data, political pundits appeared to make their predictions based on the results from specific polls. Nate Silver, on the other hand, applied his algorithm to many publicly available polling data at the state level. Because of sampling error, poll results varied across the different polls. So, even though the aggregated results of all polls painted a highly probable Obama win, the pundits could still find particular poll results to support their beliefs. (Here is a good summary of pundits who had predicted Romney would win the Electoral College and the popular vote).

Next, I want to present a psychological phenomenon to help explain how the situation above unfolded. How could the pundits make decisions that were counter to the preponderance of evidence available to them? Can we learn how to improve decision making when it comes to improving the customer experience?

Confirmation Bias and Decision Making

Confirmation bias is a psychological phenomenon where people tend to favor information that confirms or supports their existing beliefs and ignores or discounts information that contradicts their beliefs.

Here are three different forms of confirmation bias with some simple guidelines to help you minimize their impact on decision making. These guidelines are not meant to be comprehensive. Look at them as a starting point to help you think more critically about how you make decisions using customer data. If you have suggestions about how to minimize the impact of confirmation bias, please share what you know. I would love to hear your opinion.

1. People tend to seek out information that supports their beliefs or hypotheses. In our example, the pundits hand-picked specific poll results to make/support their predictions. What can you do? Specifically look for data to refute your beliefs. If you believe product quality is more important than service quality in predicting customer loyalty, be sure to collect evidence about the relative impact of service quality (compared to product quality).
2. People tend to remember information that supports their position and not remember information that does not support their position. Don’t rely on your memory. When making decisions based on any kind of data, cite the specific reports/studies in which those data appear. Referencing your information source can help other people verify the information and help them understand your decision and how you arrived at it. If they arrive at a different conclusion than you, understand the source of the difference (data quality? different metrics? different analysis?).
3. People tend to interpret information in a way that supports their opinion. There are a few things you can do to minimize the impact of confirmation bias. First, use inferential statistics to separate real, systematic, meaningful variance in the data from random noise. Place verbal descriptions of the interpretation next to the graph. A clear description ensures that the graph has little room for misinterpretation. Also, let multiple people interpret the information contained in customer reports. People from different perspectives (e.g., IT vs. Marketing) might provide highly different (and revealing) interpretations of the same data.

Summary

My good friend and colleague, Stephen King (CEO of TCELab) put it well when describing the problem of not using data in decision-making: “In the absence of data, everyone is right.” We tend to seek out information that supports our beliefs and disregard information that does not. This confirmation bias negatively impacts decisions by limiting what data we seek out and ignore and how we use those data. To minimize the impact of confirmation bias, act like a scientist. Test competing theories, cite your evidence and apply statistical rigor to your data.

Using Big Data integration principles to organize your disparate business data is one way to improve the quality of decision-making. Data integration around your customers facilitates open dialogue across different departments, improves hypothesis testing using different customer metrics across disparate data sources (e.g., operational, constituency, attitudinal), improving how you make decisions that will ultimately help you win customers or lose customers.