Scaling Log Management with Renaud Boutet
Log management requires the processing and indexing of high volumes of semi-structured data. A log management service takes log data and puts it in a cloud-hosted application so that application operators can access those logs to troubleshoot issues.
A large tech company will produce terabytes of logs. Those logs are produced on the host where a service is running. A logging agent on that host will transfer the logs to the log management service in the cloud. Once the logs are in the cloud, they are parsed, indexed, and stored in a way that is easy to query.
In 2014, Renaud Boutet co-founded Logmatic, a log management service that eventually became a leading provider. Logmatic was acquired by Datadog, and Renaud now works as a vice president at Datadog. In today’s episode, Renaud joins the show to talk about the architecture of a log management service. We talk about storage tiers, scalability requirements, failover strategies, and logging for serverless functions.
Full disclosure: Datadog is a sponsor of Software Engineering Daily.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Find a job you love on Hired. Stop job searching and get matched with 10,000+ companies looking for you. Hired combines intelligent matching technology with personalized career coaching to match you with opportunities based on your skills, industry, interests and desired salary. So stop searching and create a profile at Hired.com/sedaily and find a job you truly love.
With MongoDB Atlas, you can take advantage of MongoDB’s flexible document data model as a fully automated cloud service. MongoDB Atlas handles all the costly database operations and admin tasks that you’d rather not spend time on, like security, high availability, data recovery, monitoring, and elastic scaling.Try MongoDB Atlas for free today! Visit mongdb.com/se to learn more.
Datadog is a cloud monitoring service that unifies logs, distributed traces, and metrics in one place. Datadog’s approach of Logging without Limits enables you to cost-effectively collect, process, and archive all your logs. Explore and query a Live Tail of all your processed, enriched logs, and use the Log Patterns view to group logs for seamless investigation and troubleshooting. Try Datadog Logging for yourself with a free, 14-day trial: softwareengineeringdaily.com/datadog
GoCD is a continuous delivery tool created by ThoughtWorks. It’s great to see the continued progress on GoCD with the new Kubernetes integrations–and you can check it out for yourself at gocd.org/sedaily.