Managed Kafka with Tom Crayford

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software to manage.

Tom Crayford is an engineer at Heroku, where he helped engineer the recent Heroku Kafka product, which is a managed version of Apache Kafka. Our conversation explored the use cases of Kafka and how to build Kafka as a cloud service at scale. For more information about Heroku Kafka, check out an upcoming webinar.

Full disclosure: Heroku is a sponsor of Software Engineering Daily. That said, this is a topic I am genuinely interested in–it is often difficult to get cloud providers to talk in detail about how they are architecting their services.