Kafka Data Pipelines with Robin Moffatt

A new software product usually starts with a single database. That database manages the tables for user accounts and basic transactions.

When a product becomes popular, the database grows in size. There are more transactions and more users. A company grows around that product, and the company starts to accumulate more data in different sources. Analytics systems, time series databases, and logging tools start to generate data.

Moving this data around between systems starts to become complicated. Apache Kafka is often used as a system for moving data between these different systems, performing transformations, and generating aggregations and summaries of these large quantities of data.

Robin Moffatt works at Confluent, and has written numerous articles about how to move data between systems and design effective workflows for data pipelines. Robin joins the show to talk about modern data platforms and databases, and the patterns for using Kafka to connect those systems to each other.

If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th – October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Check out our active companies and projects:

  • FindCollabs is a place to find collaborators and build projects. Find a project to work on
  • Podsheets is an open source podcast hosting platform built with the learnings from Software Engineering Daily. Our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
  • The SEDaily app for iOS and Android includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. Subscribe for ad-free episodes.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

Cox Automotive is looking for software engineers, data engineers, scrum masters, and a variety of other positions to help push the technology forward. If you want to innovate in the world of car buying, selling, and ownership, check out COXAUTOTECH.COM to find out more about career opportunities and what it’s like working at Cox Automotive. Cox Automotive isn’t a car company. They are a technology company that’s transforming the automotive industry.

Vettery is an online hiring marketplace that connects highly qualified workers with top companies. Vettery keeps the quality of workers and companies on the platform high, because they vet both workers and companies. Check out vettery.com/sedaily, and get a $300 sign-up bonus if you accept a job through Vettery.

MongoDB is the most popular document-based database built for modern application developers and the cloud era. Try MongoDB today with Atlas, the global cloud database service that runs on AWS, Azure, and Google Cloud. Configure, deploy, and connect to your database in just a few minutes. Check it out at mongodb.com/atlas.

Rehack emphasizes humane design, usability, and positive mental health. Rehack is being hosted by Princeton University this November, and they are looking for sponsors who are interested in supporting their mission. For $500 to $3000, your company can support Rehack with resources for the hackathon. Event specifics can be found on rehack.co.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.