Data Engineering Automation with Mike Kim

Every company has the idea of the “nightly report.” A business analyst comes into the office, sits down in front of their inbox, and looks at yesterday’s data. Did sales go up? Did the marketing campaigns bring in the expected number of customers? Was there an increase in helpdesk tickets? The statistics that these reports deliver to human analysts can change the direction of the business.

Everyone within a company could use a regular report that documents how the business is changing over time. Outlier.ai is a company that processes the data sets within a business and generates automated reports that are relevant to different people within the organization.

If you are an email marketing analyst, your data from MailChimp campaigns will be analyzed. If you manage a customer success team, your Zendesk tickets will be analyzed. If you are a technical support analyst, the crash reports and error messages from your users will be analyzed. In all of these cases, the data gets processed automatically, and a story is sent to you, so that you can have the information in your inbox waiting for you, instead of having to go ask a data scientist to generate it for you.

Mike Kim is the CTO of Outlier.ai, and in this show, he describes the engineering challenges of integrating with all the different data sets of an organization–and why there is so much value in the idea of the automated “report” or “story” for analysts.

In past shows, we have explored how data engineering has progressed over the last twenty years–from database administration to Hadoop cluster management to the emergence of “data breadlines” where analysts wait for a data scientist to process the job they asked for. Outlier represents a step towards a world where the data science reports are delivered to us before we even ask, rather than us having to query the system.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.