Slack Data Platform with Josh Wills

Slack is a messaging platform for organizations. Since its creation in 2013, Slack has quickly become a core piece of technology used by a wide variety of technology companies, groups, and small teams. 

The messages that are sent on Slack are generated at a very high volume, and are extremely sensitive. These messages must be stored on Slack’s servers in a way that does not risk a message from one company accidentally being accessible to another company. The messages must be highly available, and they also must be indexed for search.

When Slack was scaling, the company started to encounter limitations in its data infrastructure that the company was unsure how to solve. During this time, Josh Wills was the director of data engineering at Slack, and he joins the show to retell the history of his time at Slack, and why the problem of searching messages was so hard. 

Josh also provides a great deal of industry context around how engineers from Facebook and Google differ from one another. When Slack was starting to become popular, the company quickly began to attract engineers from both of those companies. Facebook and Google have distinct solutions for how they have tackled the problems of data engineering.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Announcements

  • We are hiring a software engineer who can work across both mobile and web. This role will work on SoftwareDaily.com, our iOS app, and our Android application. We are looking for someone who learns very quickly and can produce high quality code at a fast pace. If you are interested in working with us, send an email to jeff@softwareengineeringdaily.com.
  • If you are planning a hackathon, check out FindCollabs Hackathons. Whether you are running an internal hackathon for your company, or you are running an open hackathon so that users can try out your product, FindCollabs Hackathons are a tool for people to build projects and collaborate with each other. FindCollabs is a company I started to allow people to find collaborators for their software projects, and our new hackathon product allows you to organize your hackathon participants to make your hackathon as productive as possible. Check it out at FindCollabs.com.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

Datadog unites metrics, traces, and logs in one platform so you can get full visibility into your infrastructure and applications. Check out new features like Trace Search & Analytics for rapid insights into high-cardinality data, and Watchdog, an auto-detection engine that alerts you to performance anomalies across your applications. Datadog makes it easy for teams to monitor every layer of their stack in one place, but don’t take our word for it—start a free trial today & Datadog will send you a T-shirt! softwareengineeringdaily.com/datadog

Redis can be used as your operational data store, for queuing, streaming, and other data applications. Redis Labs is the company that makes Redis Enterprise, which offers performance, reliability, and professional assistance with your Redis instances. If Redis is on the critical path of your application, go to redislabs.com/SEDaily, and learn about Redis Labs, as well as some design patterns for Redis that you might not have seen before.

G2i is a hiring platform run by engineers that matches you with React, React Native, GraphQL, and mobile engineers who you can trust. Whether you are a new company building your first product or an established company that wants additional engineering help, G2i has the talent you need to accomplish your goals. Go to softwareengineeringdaily.com/g2i

Logi Analytics is the leading development platform for embedded dashboards and reports. Unlike other solutions, Logi gives you complete control to create your own unique analytics experience. You don’t have to be a UI/UX designer to create impressive dashboards for your application. Discover 6 basic principles that can transform your embedded analytics UX by visiting logianalytics.com/sedaily.