YouTube runs a large MySQL database to hold the metadata about its videos. As YouTube scaled, the database was sharded, and applications within YouTube had to write queries that were aware of the sharding layout of that database.
This is problematic, because it pushes complexity to the application developer. An application developer shouldn’t have to be aware of how a database is laid out among different nodes. The developer should be able to issue a query, and have the cluster simply return the data.
Vitess is an open source system for scaling large MySQL databases. Sugu Sougoumarane co-created Vitess at YouTube. Since YouTube is owned by Google, Vitess was able to leverage the Borg cluster manager developed at Google. Once Kubernetes came to market, it became more viable to make Vitess accessible to open source developers.
Sugu joins the show to talk about the scalability problems that YouTube’s database infrastructure encountered and the motivations for building Vitess.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Every team has its own software, and every team has specific questions about that internal software stack. Stack Overflow for Teams is a private, secure home for your team’s questions and answers. No more digging through stale wikis and lost emails—give your team back the time it needs to build better products. Your engineering team already knows and loves Stack Overflow. They don’t need another tool they won’t use. Get everything that 50 million people already love about Stack Overflow in a private, secure environment with Stack Overflow for Teams. Try it today, with your first 14 days free. Go to s.tk/daily Stack Overflow for teams gives your team the answers they need to be productive–with the same interface that Stack Overflow users are familiar with. Go to s.tk/daily to try it today, with your first 14 days free.
Azure Container Service simplifies the deployment, management and operations of Kubernetes. You can continue to work with the tools you already know, such as Helm, and move applications to any Kubernetes deployment. Integrate with your choice of container registry, including Azure Container Registry. Also, quickly and efficiently scale to maximize your resource utilization without having to take your applications offline. Isolate your application from infrastructure failures and transparently scale the underlying infrastructure to meet growing demands—all while increasing the security, reliability, and availability of critical business workloads with Azure. Check out the Azure Container Service at aka.ms/sedaily.
GoCD is a continuous delivery tool created by ThoughtWorks. GoCD agents use Kubernetes to scale as needed. Check out gocd.org/sedaily and learn about how you can get started. GoCD was built with the learnings of the ThoughtWorks engineering team, who have talked about building the product in previous episodes of Software Engineering Daily. It’s great to see the continued progress on GoCD with the new Kubernetes integrations–and you can check it out for yourself at gocd.org/sedaily.