Data Science Skills and the Improbable Unicorn

Buffer

The role of data and analytics in business continues to grow. To make sense of their plethora of data, businesses are looking to data scientists for help. Job site, indeed.com, shows a continued growth in “data scientist” positions. To better understand the field of data science, we studied hundreds of data professionals.

In that study, we found that data scientists are not created equal. That is, data professionals differ with respect to the skills they possess. For example, some professionals are proficient in statistical and mathematical skills while others are proficient in computer science skills. Still others have a strong business acumen. In the current analysis, I want to determine the breadth of talent that data professionals possess to better understand the possibility of finding a single data scientist who is skilled in all areas. First, let’s review the study sample and the method of how we measured talent.

Assessing Proficiency in Data Skills

We surveyed hundreds of data professionals to tell us about their skills in five areas: Business, Technology, Math & Modeling, Programming and Statistics. Each skill area included five specific skills, totaling 25 different data skills in all.

For example, in the Business Skills area, data professionals were asked to rate their proficiency in such specific skills as “Business development,” and “Governance & Compliance (e.g., security).” In the Technology Skills area, they were asked to rate their proficiency in such skills as “Big and Distributed Data (e.g., Hadoop, Map/Reduce, Spark),” and “Managing unstructured data (e.g., noSQL).” In the Statistics Skills, they were asked to rate their proficiency in such skills as “Statistics and statistical modeling (e.g., general linear model, ANOVA, MANOVA, Spatio-temporal, Geographical Information System (GIS)),” and “Science/Scientific Method (e.g., experimental design, research design).”

For each of the 25 skills, respondents were asked to tell us their level proficiency using the following scale:

• Don’t know (0)
• Fundamental Knowledge (20)
• Novice (40)
• Intermediate (60)
• Advanced (80)
• Expert (100)

This rating scale is based on a proficiency rating scale used by NIH. Definitions for each proficiency level were fully defined in the instructions to the data professionals.

Standard of Performance

Figure 1. Proficiency in data skills varies by job role.

The different levels of proficiency are defined around the data scientists ability to give or need to receive help. In the instructions to the data professionals, the “Intermediate” level of proficiency was defined as the ability “to successfully complete tasks as requested.” We used that proficiency level (i.e., Intermediate) as the minimum acceptable level of proficiency for each data skill. The proficiency levels below the Intermediate level (i.e., Novice, Fundamental Awareness, Don’t Know) were defined by an increasing need for help on the part of the data professional. Proficiency levels above the Intermediate level (i.e., Advanced, Expert) were defined by the data professional’s increasing ability to give help or be known by others as “a person to ask.”

We looked at the level of proficiency for the 25 different data skills across four different job roles. As is seen in Figure 1, data professionals tend to be skilled in areas that are appropriate for their job role (see green-shaded areas in Figure 1). Specifically, Business Management data professionals show the most proficiency in Business Skills. Researchers, on the other hand, show lowest level of proficiency in Business Skills and the highest in Statistics Skills.

For many of the data skills, the typical data professional does not have the minimum level of proficiency to do be successful at work, no matter their role (see yellow- and red-shaded areas in Figure 1). These data skills include the following: Unstructured data, NLP, Machine Learning, Big and distributed data, Cloud management, Front-end programming, Optimization, Graphic models, Algorithms and Bayesian statistics.

In Search of the Elite Data Scientist

Figure 2. There are only a handful of data professionals who are proficient in all skill areas

There are a couple of ways an organization can build their data science capability. It can either hire a single individual who is skilled in all data science areas or it can hire a team of data professionals who have complementary skills. In both cases, the organization has all the skills necessary to use data intelligently. However, the likelihood of finding a data professional who is an expert in all five skill areas is quite low (see Figure 2). In our sample, we looked at three levels of proficiency: Intermediate, Advanced and Expert. We found that only 10% of the data professionals indicated they had, at least, an Intermediate level of proficiency in all five skill areas. The picture looks more bleak you look for data professionals who have advanced or expert proficiencies in data skills. The chance of finding a data professional with Advanced skills or better in all five skill areas drops to less than 1%. There were no data professionals who were considered as Experts in all five skill areas.

Figure 3. Proficiency levels by industry

We looked at proficiency differences across five industries: Consulting (n = 52), Education / Science (n = 50), Financial, (n = 52), Healthcare (n = 50) and IT (n = 95). We identified data professionals who had an advanced level of proficiency across the different skills. We found that data professionals in the Education / Science industry have more advanced skills (54% of data professionals have at least an advanced level of proficiency in at least one skill area) compared to data professionals in the Financial (37%) and IT (34%) industries.

Summary

The term “data scientist” is ambiguous. There are different types of data scientists, each defined by their level of proficiency in one of five skill areas: Business, Technology, Programming, Math & Modeling and Statistics. Data scientists can be defined by the skills they possess. So, when somebody tells you they are a data scientist, be sure you know what type they are.

Finding a data professional who is proficient in all data science skill areas is extremely difficult. As our study shows, data professionals rarely possess proficiency in all five skill areas at the level needed to be successful at work. The chance of finding a data professional with Expert skills in all five areas (even in 3 or 4 skill areas) is akin to finding a unicorn; they just don’t exist. There were very few data professionals who even had the basic minimum level of proficiency (i.e., Intermediate level of proficiency) in all five skill areas. Additionally, our initial findings on industry differences in skill proficiency suggest that skilled data professionals might be easier to find in specific industries. These industry differences could impact recruitment and management of data professionals. An under-supply of data science talent in one industry could require companies to use more dramatic recruitment efforts to attract data professionals from outside the industry. In industries where there are plenty of skilled data professionals, companies can be more selective in their hiring efforts.

Optimizing the value of business data is dependent on the skills of the data professionals who process the data. We took a skills-based approach to understanding how organizations can extract value from their data. Based on our findings, we recommend that organizations avoid trying to find a single data professional who has the skills that span the entire spectrum of data science. Rather, a better approach is to consider building up your data science capability through the formation of teams of data professionals who have complementary skills.