Data Engineering Automation with Mike Kim

Every company has the idea of the “nightly report.” A business analyst comes into the office, sits down in front their inbox, and looks at yesterday’s data. Did sales go up? Did the marketing campaigns bring in the expected number of customers? Was there an increase in helpdesk tickets? The statistics that these reports deliver to human analysts can change the direction of the business.

Everyone within a company could use a regular report that documents how the business is changing over time. Outlier.ai is a company that processes the data sets within a business and generates automated reports that are relevant to different people within the organization.

If you are an email marketing analyst, your data from MailChimp campaigns will be analyzed. If you manage a customer success team, your Zendesk tickets will be analyzed. If you are a technical support analyst, the crash reports and error messages from your users will be analyzed. In all of these cases, the data gets processed automatically, and a story is sent to you, so that you can have the information in your inbox waiting for you, instead of having to go ask a data scientist to generate it for you.

Mike Kim is the CTO of Outlier.ai, and in this show he describes the engineering challenges of integrating with all the different data sets of an organization–and why there is so much value in the idea of the automated “report” or “story” for analysts.

In past shows, we have explored how data engineering has progressed over the last twenty years–from database administration to Hadoop cluster management to the emergence of “data breadlines” where analysts wait for a data scientist to process the job they asked for. Outlier represents a step towards a world where the data science reports are delivered to us before we even ask, rather than us having to query the system.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Sponsors

Flatiron School is an outcomes-focused coding bootcamp, offering transformative education in person and online. Start learning for free at flatironschool.com/sedaily and get $500 off your first month of Flatiron’s Online Data Science Bootcamp or Online Web Developer Program.

Hired is a career marketplace that intelligently matches tech talent with the world’s most innovative companies. We are on a mission to find everyone a job they love. Go to hired.com/sedaily, and get $600 free, if you find a job through Hired.

Jamf Now helps you set up, manage and protect your Apple devices on demand. Software Engineering Daily listeners can start securing their business today setting up the first 3 devices for free, forever. Add more for just two dollars a month, per device. Create your free account today at jamf.com/sedaily.