Cloud Log Analysis with Jack Naglieri

Large software companies have lots of users, and the activity from those users results in high volumes of traffic. These companies also have a large surface area across the enterprise. 

There are hundreds of services and databases that are fulfilling user requests. As these requests enter the infrastructure of the enterprise, the requests travel through the different services and result in database queries, payments, and other transactions. These transactions result in the generation of log messages. The log messages tell the story of what is happening across the entire company. 

Log messages can provide valuable data for security and site reliability engineering. But analyzing a high volume of log data requires a scalable system that can account for that high volume. 

Jack Naglieri is the CEO of Panther Labs. He previously worked at Airbnb, where he helped develop a system called StreamAlert. At Airbnb, log messages are buffered into distributed queueing systems like Kafka or Kinesis, and they are written to bucket storage systems like S3. Those logs are processed by AWS Lambda functions that test the log messages for rules defined by a system operator.

Jack left Airbnb and started Panther Labs to generalize the tools he built within Airbnb and build a company around the same ideas. Jack joins the show to discuss modern logging infrastructure, his work at Airbnb, and his experience building Panther.

Contact Panther Labs at…

 

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

 

 

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

Seen by Indeed is a tech-focused matching platform. Every Seen candidate also gets free access to technical career coaching, resume reviews, mock interviews, and even salary negotiation tips to seal the deal. If you are ready for a new job, you are ready for Seen by Indeed. Join today and get a free resume review when you go to beseen.com/dailypodcast.

VictorOps is a collaborative incident response tool. VictorOps brings your monitoring data and your collaboration tools into one place–so that you can fix issues more quickly, and reduce the pain of on-call. If you want to hear about how VictorOps works, you can listen to our episode with Chris Riley. Learn more about it as well as get a free t-shirt when you check it out at victorops.com/sedaily.

DataStax provides DataStax Enterprise, a powerful distribution of Cassandra, created by the team that has contributed the most to Cassandra. DataStax Enterprise enables teams to develop faster, scale further, achieve operational simplicity, ensure enterprise security, and run mixed workloads that work with latest Graph, Search, and Analytics technology—all running across the hybrid and multi-cloud. To learn more about Apache Cassandra and DataStax Enterprise, go to datastax.com/sedaily

With over 9 million apps created on Heroku, over 2 million managed data services and serving over 26 million request per day,  Heroku has earned the trust of developers–and it is as easy to start today as it always has been. Try Heroku for free today. Visit softwareengineeringdaily.com/heroku to get started today.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.