Prometheus Scalability with Bryan Boreham
Podcast: Play in new window | Download
Subscribe: RSS
Prometheus is an open source monitoring system and time series database. Prometheus includes a multi-dimensional data model, a query language called PromQL, and a pull model for gathering metrics from your different services. Prometheus is widely used by large distributed systems deployments such as Kubernetes and Cloud Foundry.
Prometheus gathers metrics from your services by periodically scraping those services. Those metrics get gathered, compressed, and stored onto disk for querying. But Prometheus is designed to store all of its records on one host in one set of files–which limits the scalability and availability of those metrics.
Cortex is an open source project built to scale Prometheus. Cortex effectively shards Prometheus by parallelizing the “ingestion” and storage of Prometheus metrics. Cortex can take metrics from multiple Prometheus instances and store them across a distributed NoSQL database like DynamoDB, BigTable, or Cassandra.
Bryan Boreham is an engineer at Weaveworks, where he works on deployment, observability, and monitoring tools for containers and microservices. He wrote much of the code for Cortex, and we met up at KubeCon North America to talk about the motivation for creating Cortex, the broader landscape of Kubernetes monitoring, and other approaches to scaling Prometheus.
Transcript
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Sponsors
Triplebyte is a company that connects engineers with top tech companies. We’re running an experiment and our hypothesis is that Software Engineering Daily listeners will do well above average on the quiz. Go to triplebyte.com/sedaily.
Digital Ocean is the easiest cloud platform to run and scale your application. Try it out today and get a free $100 credit–go to do.co/sedaily. Digital Ocean is a complete cloud platform to help developers and teams save time when running and scaling their applications.
OpenShift is a Kubernetes platform from Red Hat. OpenShift takes the Kubernetes container orchestration system and adds features that let you build software more quickly. OpenShift includes service discovery, CI/CD, built-in monitoring and health management, and scalability. With OpenShift, you avoid getting locked into any particular cloud provider. Check out OpenShift from RedHat, by going to softwareengineeringdaily.com/redhat.
Deploy infrastructure faster; simplify life cycle maintenance for your servers; give IT the ability to deliver infrastructure to developers as a service like the public cloud. Go to softwareengineeringdaily.com/hpe and learn about how HPE OneView can improve your infrastructure operations.
