The Big Data Industry Has a Skill Problem. Here's How They Can Solve it.

In the world of big business, data is the new currency. Companies like Amazon, Netflix, and Facebook have used it to become the dominant players in their respective industries. That has prompted other businesses of all stripes to follow suit. What's followed has been a tech sector boom the likes of which we haven't seen in some time.

As a share of the overall tech market, big data is a big deal. Industry analysts expect that revenues in the space will reach $260 billion by 2022 – an estimate that may prove conservative in hindsight. As you might expect, however, all that growth is producing some challenges. The largest among them is a large (and growing) shortage of individuals with the right skills to fuel the industry's growth. Here's an overview of the situation and what needs to be done to correct it.

A Dearth of Talent

One of the biggest threats to the big data industry is a lack of big data skills in the labor market. It's a problem that's come to be known as the big data skills gap. The reason it's happening is the fact that the world's education systems haven't been able to keep up with the astronomical demand the big data industry's growth has created. Although there's been no shortage of efforts to solve the problem, it has persisted and could put the brakes on the industry's growth in the near future.

It's an enormous problem, too. Last year alone, the demand for data scientists soared by 29%. That was just a part of the 344% overall increase since 2013, a figure that has no precedent in almost any other industry. That reality translates to over 150,000 data scientist positions remaining open at any given time. Predictably, the problem is at its worst in major technology hubs like New York, San Francisco, and Los Angeles.

Fixing the Problem

To overcome the problem, the big data industry has to deploy a variety of solutions. Some must aim at the short-term and others at the long-term. The first step is for businesses in need of data scientists and other big data employees to consider building remote data science teams. It's a useful strategy because, as noted earlier, the shortage is worse in specific parts of the country. That means companies willing to work with remote teams can draw from a larger candidate pool, and sidestep shortages in their local market that may be more acute. Alternately, businesses can opt to fund a relocation program, to lure talent from across the country to come to them, instead.

On top of making use of remote workers, companies can also build training programs to up-skill existing workers. It's a tactic already being used by Amazon, who is spending over $700 million to retrain up to 100,000 existing employees for hard-to-staff positions. Smaller firms can adopt this method by subsidizing continuing education in data science for existing employees. Today, it's possible to earn a Master of Data Science degree without stepping foot in a classroom – so creating a fully-featured big data training program has never been easier.

Over the longer term, the big data industry has to support efforts to increase STEM education in schools throughout the country. Right now, the major tech companies are already doing this by increasing funding to public/private partnerships in the education sector. The efforts, though, aren't nearly enough. In particular, more must be done to fund the hiring of teachers in the field. Without doing so, the education sector won't be able to create a more stable flow of talent to feed the big data industry of tomorrow.

Mitigating Factors

The good news is that the big data industry is about to get some serious assistance from another, related branch of the technology industry: artificial intelligence (AI). Advances in the field of AI are making it possible to automate many of the critical tasks that currently occupy data scientists' time. Things like data standardization and cleaning will soon be tasks relegated to machines. Even the development of machine learning models will become automated, which will eliminate some of the most labor-intensive parts of today's big data applications.

That won't eliminate the skills shortage but should reduce its rate of growth to keep things from getting worse. If the timing works out the right way, the growth of automation in big data should give the other kinds of efforts mentioned here a good chance of solving the skills shortage once and for all. That would be good news for everyone involved, but it depends on the big data industry making efforts right now before the problem worsens. With the statistics reflecting the issue piling up – it wouldn't make much sense for a data-centric industry to ignore such an obvious insight.

This article does not necessarily reflect the opinions of the editors or management of EconoTimes.