Kubernetes Vision with Brendan Burns

Kubernetes has become the standard system for deploying and managing clusters of containers. But the vision of the project goes beyond managing containers. The long-term goal is to democratize the ability to build distributed systems.

Brendan Burns is a co-founder of the Kubernetes project. He recently announced an open-source project called Metaparticle, a standard library for cloud-native development:

Metaparticle builds on top of Kubernetes primitives to make distributed synchronization easier… It supplies language-independent modules for locking and leader election as easy-to-use abstractions in familiar programming languages.

After decades of distributed systems research and application, patterns have emerged about how we build these systems. We need a way to lock a variable so that two nodes will not be able to write to that variable in a non-deterministic fashion. We need a way to do a master election so that if the master node dies, the other nodes can pick a new node to orchestrate the system.

We know that just about every distributed application needs locking and leader election–so how can we build these features directly into our programming tools, rather than bolting them on?

With Kubernetes providing a standard operating system for distributed applications, we can start to build standard libraries that assume we have access to underlying Kubernetes primitives. Instead of calling out to external tools like Zookeeper and etcd, a standard library like Metaparticle will abstract them away.

An example: if I am writing a system to do distributed mapreduce, I would like to avoid thinking about node failures and race conditions. Brendan’s idea is to push those problems down into a standard library–so the next developer who comes along with a new idea for a multi-node application has an easier time.

Brendan Burns currently works as a distinguished engineer at Microsoft, and he joins the show to discuss why it is still hard to build distributed systems and what can be done to make it easier. This is the second time we have had Brendan on the show. The first time he came on, he discussed the history of Kubernetes and some of the design decisions of the system. This episode was more about the future. Full disclosure: Microsoft (where Brendan is employed) is a sponsor of Software Engineering Daily.

Show Notes

Get a free digital copy of Brendan’s new O’Reilly book, Designing Distributed Systems: http://aka.ms/sedaily

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.