Data Engineering with David Drummond and Austin Ouyang

“We want people to be able to pick up whatever tool it is and really push themselves to get something done with it in a short amount of time, because that’s ultimately what they need to do as a data engineer in the industry.”

Data engineering is a newly emerging field that exists at the intersection of software engineering and data science. Data engineers are increasingly in demand for their abilities to use the various open source big data frameworks to build efficient and robust data pipelines.

David Drummond and Austin Ouyang are program directors for the Insight Data Engineering Fellows Program, where they help train data engineers that can immediately start contributing in the industry.

Questions

• What is a data engineer?
• What are the tools and workflows of a data engineer?
• Does a data engineer’s job overflow with devops?
• What are the problems with Hadoop MapReduce?
• What is the lambda architecture, and what problems does it solve?
• What are the storage technologies and principles that data engineers need to know about?

Links

• Insight Data Engineering
• Lambda Architecture
• Questioning the Lambda Architecture
• David on Twitter
• Austin on Twitter

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Pranay