FluentD with Eduardo Silva

A backend application can have hundreds of services written in different programming frameworks and languages.

Across these different languages, log messages are produced in different formats. Some logging is produced in XML, some are produced in JSON, some is in other formats. These logs need to be unified into a common format and centralized for any developer who wants to debug.

The popularity of Kubernetes is making it easier for companies to build this kind of distributed application, where different services of different languages are communicating over a network, with a variety of log message types.

Fluentd is a tool for solving this problem of log collection and unification. In today’s episode, Eduardo Silva joins the show to describe how Fluentd is deployed to Kubernetes, and what the role of Fluentd is within a Kubernetes logging pipeline.

We also discuss the company where Eduardo works–Treasure Data. The story of Treasure Data is unusual. The team started out doing log management but has found itself moving up the stack, into marketing analytics, sales analytics, and customer data management. This story might be useful for anyone who is an open-source developer thinking about how to evolve your project into a business.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.