Apache Airflow with Maxime Beauchemin, Vikram Koka, and Ash Berlin-Taylor

Apache Airflow was released in 2015, introducing the first popular open source solution to data pipeline orchestration. Since that time, Airflow has been widely adopted for dependency-based data workflows. A developer might orchestrate a pipeline with hundreds of tasks, with dependencies between jobs in Spark, Hadoop, and Snowflake.

Since Airflow’s creation, it has powered the data infrastructure at companies like Airbnb, Netflix, and Lyft. It has also been at the center of Astronomer, a startup that helps enterprises build infrastructure around Airflow. Airflow is used to construct DAGs–directed acyclic graphs for managing data workflows.

Maxime Beauchemin is the creator of Airflow. Vikram Koka and Ash Berlin-Taylor work at Astronomer. They join the show to talk about the state of Airflow–the purpose of the project, its use cases, and open source ecosystem.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

JFrog swampUP is an online user conference with 30+ sessions from cross-industry experts, including Google, Microsoft, Capital One, Adobe, and more. There will be tracks for Cloud-Native DevOps, Enterprise DevOps, DevSecOps, and Digital Transformation. Participate in expert DevOps training classes across DevOps tools, security, CI/CD, and more. Join from two time zones – June 23-24 for the Americas and June 30-July 1 for EMEA and APAC to suit daylight hours across the globe. Go to softwareengineeringdaily.com/swampUP to learn more and sign up. 

G2i is a hiring platform run by engineers that matches you with React, React Native, GraphQL, and mobile engineers who you can trust. Whether you are a new company building your first product or an established company that wants additional engineering help, G2i has the talent you need to accomplish your goals. Go to softwareengineeringdaily.com/g2i

For more than two years and nearly 700 episodes, The Techmeme Ride Home has been Silicon Valley’s favorite tech news podcast. The Techmeme Ride Home is a daily podcast, only fifteen to twenty minutes long, and every day by 5pm EST its ALL the latest tech news. Search your podcast app now for Ride Home and subscribe to the Techmeme Ride Home podcast.

From their recent report on serverless adoption and trends, Datadog found half of their customer base using EC2s have now adopted AWS Lambda. You can easily monitor all your serverless functions in one place and generate serverless metrics straight from Datadog. Check it out yourself by signing up for a free 14-day trial and get a free t-shirt at softwareengineeringdaily.com/datadog

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.