Flyte: Lyft Data Processing Platform with Allyson Gale and Ketan Umare

Lyft is a ridesharing company that generates a high volume of data every day. 

This data includes ride history, pricing information, mapping, routing, and financial transactions. The data is stored across a variety of different databases, data lakes, and queueing systems, and is processed at scale in order to generate machine learning models, reports, and data applications.

Data workflows involve a set of interconnected systems such as Kubernetes, Spark, Tensorflow, and Flink. In order for these systems to work together harmoniously, a workflow manager is often used to orchestrate them together. A workflow platform lets a data engineer have a high-level view into how data moves through the system, and can be used to reason about retries, resource utilization, and scalability.

Flyte is a data processing system built and open-sourced at Lyft. Allyson Gale and Ketan Umare work at Lyft, and they join the show to talk about how Flyte works, and why they needed to build a new workflow processing system when there are already tools available such as Airflow.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

SAP Data Intelligence connects and transforms data to extract value from the distributed data landscape. SAP Data Intelligence brings together data orchestration, metadata management, and powerful data pipelines with advanced machine learning, enabling close collaboration between data scientists and IT. To learn more about SAP Data Intelligence, visit sap.com/sedaily

X-Team is a company that can help you scale your team with new engineers. X-Team has thousands of proven developers in over 50 countries ready to join your team. X-Team is able and ready to support a full range of team/project needs. If your development team could use some firepower via some of the top engineering talent in the world, visit x-team.com/sedaily.

MongoDB made MongoDB Atlas, a global cloud database service that runs on AWS, GCP, and Azure.  You can deploy a fully managed MongoDB database in minutes with just a few clicks or API calls. You can get started free at mongodb.com/atlas

G2i is a hiring platform run by engineers that matches you with React, React Native, GraphQL, and mobile engineers who you can trust. Whether you are a new company building your first product or an established company that wants additional engineering help, G2i has the talent you need to accomplish your goals. Go to softwareengineeringdaily.com/g2i

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.