Metaflow: Netflix Machine Learning Platform with Savin Goyal

Netflix runs all of its infrastructure on Amazon Web Services. This includes business logic, data infrastructure, and machine learning. By tightly coupling itself to AWS, Netflix has been able to move faster and have strong defaults about engineering decisions. And today, AWS has such an expanse of services that it can be used as a platform to build custom tools.

Metaflow is an open source machine learning platform built on top of AWS that allows engineers at Netflix to build directed acyclic graphs for training models. These DAGs get deployed to AWS as Step Functions, a serverless orchestration platform.

Savin Goyal is a machine learning engineer with Netflix, and he joins the show to talk about the machine learning challenges within Netflix, and his experience working on Metaflow. We also talk about DAG systems such as AWS Step Functions and Airflow.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

Sentry.io can quickly triage and resolve issues in real-time. Sentry works for every major language and framework, from Rails, C#, Java, or React Native for example–install the SDKs, and integrate it with your application. If you are building an application and want to monitor errors and performance, try Sentry by visiting Sentry.io/signup. New users can use code SEDAILY for 2 free months.

With Triplebyte, you do one online interview, and then you get to go straight to final interviews at hundreds of companies (from tech giants like Dropbox to exciting startups). It’s like the Common App for software engineers. No resume needed. Apply now at triplebyte.com/sedaily. If you take a job through Triplebyte, you’ll get a $1000 signing bonus.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.