Hottest of the Hot Jobs: Big Data Engineers

One of Randstad Technologies' hot jobs is big data engineering. There can be confusion about the specific roles people with this title fill. According to the big data news portal, Datanami.com, data engineers are the specialists who are the bridge between the data analysts and the data scientists. They are the people charged with providing the data analytics applications, and help obtain quality data from different sources in the appropriate format(s) to make sure that users have access to that data quickly. They are adept at manipulating enormous quantities of data and making sure that the organization's big data software stack can scale to support the workload.

As with all other Randstad Technologies' Hot Jobs (full stack developers, IoT architects, VR engineers and security engineers), the demand for big data engineers is strong and will only get stronger. According to a report written by Stitch, an ETL service built for developers, there are fewer than 7,000 self-reported data engineers in the U.S. The same report notes that at the time it was written, there were 6,600 job openings for data engineers - in the San Francisco Bay Area alone! In the past 30 days there have been more than 1,000 jobs posted on Monster.com with either "data engineer” or "big data engineer” in the title. IBM, Cisco, and Oracle posted a combined total of nearly 27,000 open jobs that required big data expertise in the last 12 months. Finding the people to fill all these jobs will be challenging.


Profile of Big Data Engineers

Overview of data engineer characteristics/positions:

• Around 40% of data engineers graduated from a software engineering role. Other positions from which data engineers come include analyst, consultant, business analyst, data architect, data analyst, database administrator, and data scientist.


• Some of the most important technical skills for data engineers to possess are SQL, Java, Hadoop, Python, Linux, Spark, and Hive.


• Annual salaries for data engineers have been trending upward of late, with the average ranging between $100,000 and $200,000 and the median at $141,000.


• Of those, nearly 7,000 people who classify themselves as data engineers, 40% have a bachelor's degree, 27% a master's degree, 18% an associate's degree, with around 4% having a Ph.D. Sixty-one percent of big data engineer openings require a bachelor's degree.


• There are a number of certification programs, some of which are offered by universities (e.g., Certification of Professional Achievement in Data Sciences from Columbia University) and some by private firms (e.g., Cloudera Certified Data Engineer, Galvanize Spark workshops, etc.). Price and duration of each program can vary.


• There are various estimates, but it appears that men outnumber women as data engineers from around five to one, to as much as seven to one.

Who Hires Big Data Engineers?

According to Forbes, the top five industries hiring big data-related expertise include professional, scientific, and technical services 30%, information technologies 19%, manufacturing 18%, finance and insurance 10%, and retail trade 8%. Big employers include Google, Uber, Facebook, Amazon, Verizon, Capital One, AirBnB, IBM, Cisco, Oracle, EMC, and Spotify, among others. Given that the magnitude and value of data is going to continue to grow at an accelerating pace, even smaller companies are going to have to employ data engineers to find ways to effectively acquire, organize, and analyze their data if they are to remain competitive.

Soft Skills a Plus

As with almost all technical positions, having soft skills enhances a candidate's attractiveness and ability to fashion their own future. Because data engineers often have to explain highly technical concepts and findings to non-tech personnel, strong communication skills are highly attractive.

The demand for data engineers is going to outstrip supply for some time. Competition for individuals with the necessary skills will remain intense for the foreseeable future. It will remain one of the hottest of our hot jobs.

Looking for a new career opportunity as a big data engineer? Search our website today for big data engineer jobs in your area.

Originally posted on Randstad.

Opinions expressed by the author are not necessarily those of WITI.