Slack Data Platform with Josh Wills

Slack is a messaging platform for organizations. Since its creation in 2013, Slack has quickly become a core piece of technology used by a wide variety of technology companies, groups, and small teams. 

The messages that are sent on Slack are generated at a very high volume, and are extremely sensitive. These messages must be stored on Slack’s servers in a way that does not risk a message from one company accidentally being accessible to another company. The messages must be highly available, and they also must be indexed for search.

When Slack was scaling, the company started to encounter limitations in its data infrastructure that the company was unsure how to solve. During this time, Josh Wills was the director of data engineering at Slack, and he joins the show to retell the history of his time at Slack, and why the problem of searching messages was so hard. 

Josh also provides a great deal of industry context around how engineers from Facebook and Google differ from one another. When Slack was starting to become popular, the company quickly began to attract engineers from both of those companies. Facebook and Google have distinct solutions for how they have tackled the problems of data engineering.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Announcements

• We are hiring a software engineer who can work across both mobile and web. This role will work on SoftwareDaily.com, our iOS app, and our Android application. We are looking for someone who learns very quickly and can produce high quality code at a fast pace. If you are interested in working with us, send an email to jeff@softwareengineeringdaily.com.
• If you are planning a hackathon, check out FindCollabs Hackathons. Whether you are running an internal hackathon for your company, or you are running an open hackathon so that users can try out your product, FindCollabs Hackathons are a tool for people to build projects and collaborate with each other. FindCollabs is a company I started to allow people to find collaborators for their software projects, and our new hackathon product allows you to organize your hackathon participants to make your hackathon as productive as possible. Check it out at FindCollabs.com.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.