Prefect Dataflow Scheduler with Jeremiah Lowin

A data workflow scheduler is a tool used for connecting multiple systems together in order to build pipelines for processing data. A data pipeline might include a Hadoop task for ETL, a Spark task for stream processing, and a TensorFlow task to train a machine learning model. 

The workflow scheduler manages the tasks in that data pipeline and the logical flow between them. Airflow is a popular data workflow scheduler that was originally created at Airbnb. Since then, the project has been adopted by numerous companies that need workflow orchestration for their data pipelines. Jeremiah Lowin was a core committer to Airflow for several years before he identified several features of Airflow that he wanted to change.

Prefect is a dataflow scheduler that was born out of Jeremiah’s experience working with Airflow. Prefect’s features include data sharing between tasks, task parameterization, and a different API than Airflow. Jeremiah joins the show to discuss Prefect, and how his experience with Airflow led to his current work in dataflow scheduling.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

strongDM lets you manage and audit access to servers, databases, and Kubernetes clusters, no matter where your employees are. With StrongDM, you can easily extend your identity provider to manage infrastructure access. You can automate onboarding, offboarding, and moving people within roles. strongDM. Manage and audit remote access to infrastructure. Start your free 14 day trial today at: strongdm.com/SEDaily

Logi Analytics’ developer-grade embedded analytics solutions make it easy to create branded dashboards and reports that scale within your own application. So stop wasting your time piecing analytics together and allow yourself to focus on your core application. Visit logianalytics.com/sedaily for a demo and to see what’s possible with Logi, today.

G2i is a hiring platform run by engineers that matches you with React, React Native, GraphQL, and mobile engineers who you can trust. Whether you are a new company building your first product or an established company that wants additional engineering help, G2i has the talent you need to accomplish your goals. Go to softwareengineeringdaily.com/g2i

ExtraHop is a modern security company that provides complete visibility into your infrastructure. ExtraHop lets you detect threats and respond quickly, giving you the deep context you need, as well as the ability to set up automated response workflows. At extrahop.com, you can see a demo for how ExtraHop works.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.