Dagster with Nick Schrock

Data engineering is difficult. 

Companies want to be able to maximize the value they get from their large data sets, but there are so many steps required for deriving that value that most companies feel like they are always far behind the ideal state of where they could be. 

The cloud makes it cheap to save data. Tools like Spark and Snowflake give us usable APIs for simplifying the data processing. Workflow engines like Airflow help us visualize a complex execution path of a data pipeline. With all of this tooling, why is it so hard to make use of our data?

Nick Schrock is the creator of Dagster, an open source system for building modern data applications. Nick is also the CEO of Elementl, a company that he is building around Dagster. Before creating Dagster, Nick worked at Facebook, where he co-created GraphQL. 

Nick returns to the show to discuss modern data engineering, and why it continues to be so difficult for engineers to be productive with their data.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

PagerDuty helps your company’s digital operations run more smoothly. PagerDuty helps you intelligently pinpoint issues like outages, as well as capitalize on opportunities, empowering teams to take the right, real time action. To see how companies like GE, Vodafone, Box and American Eagle Outfitters rely on PagerDuty to continuously improve their digital operations visit PagerDuty.com.

Monday.com is a team management platform that brings all your work, external tools and communication into one place, making cross-team collaboration easy. You can try Monday.com and get a 14 day trial by going to monday.com/sedaily. And if you decide to become a customer, you will get 10% off by using coupon code SEDAILY.

Datadog unites metrics, traces, and logs in one platform so you can get full visibility into your infrastructure and applications. Check out new features like Trace Search & Analytics for rapid insights into high-cardinality data, and Watchdog, an auto-detection engine that alerts you to performance anomalies across your applications. Datadog makes it easy for teams to monitor every layer of their stack in one place, but don’t take our word for it—start a free trial today & Datadog will send you a T-shirt! softwareengineeringdaily.com/datadog

Triplebyte just launched their brand-new Machine Learning track! They’ll now be helping machine learning engineers find jobs in the same way that they’ve already helped generalist, front-end, and mobile engineers. See how you stack up against the industry. Go to triplebyte.com/sedaily.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.