Workload Scheduling with Brian Grant
Google has been building large-scale scheduling systems for more than fifteen years.
Google Borg was started around 2003, giving engineers at Google a unified platform to issue long-lived service workloads as well as short-lived batch workloads onto a pool of servers. Since the early days of Borg, the scheduler systems built by Google have matured through several iterations. Omega was an effort to improve the internal Borg system, and Kubernetes is an open source container orchestrator built with the learnings of Borg and Omega.
A scheduling system needs to be able to accept a wide variety of workload types and find compute resources within a cluster to schedule those workloads onto.
There is a wide variety of potential workloads that could be scheduled–batch jobs, stateful services, stateless services, and daemon services. Different workloads can have different priority levels. A high priority workload should be able to find compute resources quickly, and a low priority workload can wait longer to find resources.
Brian Grant is a principal engineer at Google. He joins the show to talk about his experience building workload schedulers and designing APIs for engineers to interface with those schedulers.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Instantly deploy and manage an SSD server in the Linode Cloud. Get a server running in seconds with your choice of Linux distro, resources, and node location. Now offering Dedicated CPU instances, designed for consistent, high performance computing needs like video encoding, game servers, and busy application servers. Get a $20 credit when you use promocode: sedaily2019 at linode.com/sedaily.
Find a job you love on Hired. Stop job searching and get matched with 10,000+ companies looking for you. Hired combines intelligent matching technology with personalized career coaching to match you with opportunities based on your skills, industry, interests and desired salary. So stop searching and create a profile at Hired.com/sedaily and find a job you truly love.
Bitbar tests your app on real devices–no emulators or virtual environments. Bitbar has real Android and iOS devices, and the Bitbar testing tools integrate with Jenkins, TravisCI, and other continuous integration tools. Check out bitbar.com/sedaily and get a free month of unlimited mobile app testing.
GoCD is a continuous delivery tool created by ThoughtWorks. It’s great to see the continued progress on GoCD with the new Kubernetes integrations–and you can check it out for yourself at gocd.org/sedaily.