Time Series Databases with Rob Skillington

A time series database is optimized for the storage of high volumes of sequential data across time.

Time series databases are often organized as columnar data stores that can write large volumes of data quickly. These systems can sometimes tolerate data loss, because the data they are gathering is used for monitoring and other applications that require aggregated data sets rather than highly important individual transactions.

The demand for time series databases has grown over the last decade with the rise of mobile devices and the decreasing cost of cloud storage. There has been an increase in the number of systems that require monitoring, and some of those systems produce an incredibly large amount of data, requiring compression, downsampling, and garbage collection.

Rob Skillington is an engineer at Uber, where he helped create M3DB, a time series database. In a previous show, Rob described the basics of M3DB and how it helps Uber with storing data from Prometheus, a monitoring system. In today’s show we discuss the field of time series databases, and Rob’s approach to building M3.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

ANNOUNCEMENTS

  • FindCollabs is a place to find collaborators and build projects. We recently launched GitHub integrations. It’s easier than ever to find collaborators for your open source projects. And if you are looking for some people to start a project with, FindCollabs we have topic rooms that allow you to find other people who are interested in a particular technology, so that you can find people who are curious about React, or cryptocurrencies, or Kubernetes, or whatever you want to build with.
  • Podsheets is an open source podcast hosting platform that we recently launched. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com.
  • New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.   

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

Kafka Summit is the place to learn about the new design patterns and engineering practices in the world of Kafka. Kafka Summit returns to San Francisco September 30-October 1, 2019. Listeners of Software Engineering Daily can get 25% off their ticket to Kafka Summit by entering promo code “SED”. 

Buildkite is a CI/CD platform for running scalable and secure continuous integration pipelines. Buildkite helps you keep your builds fast and reliable, even as they grow large. Visit buildkite.com/sedaily to learn more, and see how Shopify used Buildkite as they scaled from 300 to 1200 engineers, migrated between cloud providers, and kept their build times under 5 minutes.

Cruise is a San Francisco-based company building a fully electric self-driving car service. Cruise is a place where you can build on your existing skills while developing new skills and experiences that are pioneering the future of industry. There are opportunities for backend engineers, frontend developers, machine learning programmers, and many more positions. At Cruise you will be surrounded by talented, driven engineers-all while helping make cities safer and cleaner. Apply to work at Cruise, by going to getcruise.com/careers.

Podsheets is an open source podcast hosting platform. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.