Cluster Schedulers with Ben Hindman

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction.

Ben Hindman co-founded Mesosphere to bring the Mesos project to market. Large enterprises like Uber, Netflix, and Yelp use Mesosphere for resource management. Before he started the company, Ben worked in the Berkeley AMP Lab, a research program where the Spark and Tachyon projects were also born.

At this point, he has spent significant time in both academia and industry. This conversation spans distributed systems theory, history, and practice. Ben and I spoke at KubeCon 2018 in Copenhagen–which was an amazing conference. We were both amazed at how big the audience for Kubernetes has gotten, and the pace at which the technology is advancing.

Today, Kubernetes is mostly used for scheduling containerized applications that engineers have built themselves. But there will be higher level tools that use Kubernetes as a building block. Much like Zookeeper was used as a building block for Hadoop, Kubernetes will be used to build serverless applications and distributed databases.

Once you are using a distributed database built on Kubernetes, you don’t want to think about the container orchestration–you want to think about the raw storage and CPU requirements for that database. This is one reason why Mesos is so compelling. Since Kubernetes creates an increased cardinality of distributed systems, it’s good to know that there is a framework built to manage those higher level applications.


Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Today’s podcast is sponsored by Datadog, a cloud-scale monitoring platform for infrastructure and applications. In Datadog’s new container orchestration report, Kubernetes holds a 41-percent share of Docker environments, a number that’s rising fast. As more companies adopt containers, and turn to Kubernetes to manage their containers, they need a comprehensive monitoring platform that’s built for dynamic, modern infrastructure. Datadog integrates seamlessly with more than 200 technologies, including Kubernetes and Docker, so you can monitor your entire container infrastructure in one place. And with Datadog’s new Live Container view, you can see every container’s health, resource consumption, and running processes in real time. See for yourself by starting a free trial and get a free Datadog T-shirt!

Casper combines supportive memory foams for a sleep surface that’s got just the right sink and just the right bounce. Plus, its breathable design sleeps cool to help you regulate your temperature through the night. And, buying a Casper mattress is completely risk free. Casper offers free delivery and free returns with a 100-night home trial. If you don’t love it, they’ll pick it up and give you a full refund. As a special offer to Software Engineering Daily listeners, get $50 toward select mattresses by visiting and using code SEDAILY at checkout. Terms and conditions apply.

Segment allows us to gather customer data from anywhere and send that data to any analytics tool. Segment is the customer data infrastructure that has saved us from writing duplicate code across all of the different platforms that we want to analyze. And if you’re using cloud apps such as – Mailchimp, Marketo, Intercom, AppNexus, Zendesk–you can integrate with all of these different tools and centralize your customer data in one place–with Segment. To get a free 90-day trial, signup for Segment at and enter SEDaily in the “How did you hear about us box?” during signup.

Azure Container Service simplifies the deployment, management and operations of Kubernetes. You can continue to work with the tools you already know, such as Helm, and move applications to any Kubernetes deployment. Integrate with your choice of container registry, including Azure Container Registry. Also, quickly and efficiently scale to maximize your resource utilization without having to take your applications offline. Isolate your application from infrastructure failures and transparently scale the underlying infrastructure to meet growing demands—all while increasing the security, reliability, and availability of critical business workloads with Azure. Check out the Azure Container Service at

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.