Dashboarding and Query Latency with Tom O’Neill

A dashboard is a data visualization that aggregates metrics in a way that we can quickly understand. In a modern software company, everyone uses dashboards–from salespeople to DevOps to HR.

Each dashboard represents a query that must be updated frequently, so that anyone looking at it is getting up-to-date information. The data set being queried might be getting updated quickly in the case of time series or log data. Some queries require joins between disparate data sources.

How do you keep dashboards accurate? How do you keep query latency down?

Tom O’Neill is the CTO of Periscope Data, a company that makes popular dashboarding tools. In this episode, Tom explains the data engineering that underlies Periscope data. We explore topics such as caching, columnar data, and Redshift.

We have done many other shows about data engineering, including including shows about how data engineering works at companies like Airbnb and Giphy. Download the Software Engineering Daily app for iOS to hear all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that.

The iOS app is the first project to come out of the Software Engineering Daily Open Source Project. There are more projects on the way, and we are looking for contributors–if you want to help build a better SE Daily experience, check out github.com/softwareengineeringdaily. We are working on an Android app, the iOS app, a recommendation system, and a web frontend. Help us build a new way to consume software engineering content at github.com/softwareengineeringdaily.