Dremio Data Engineering with Tomer Shiran

Twenty years ago, all of the data in an organization could fit inside relational databases.

Imagine a company like Proctor and Gamble. P&G is a consumer packaged goods company with hundreds of business sectors–shaving products, toothpaste, shampoo, laundry detergent.

Twenty years ago, if the chief financial officer of P&G wanted to answer a question about the revenue projections within the enterprise, that CFO would ask a VP to find the answer. The VP would contact the business analysts in all the different departments within Procter and Gamble, and those business analysts would all work with database administrators to answer questions for their business sector. In that world, it might have taken weeks or months for the CFO to get the answer about revenue projections.

Today, data engineering has improved dramatically. Data sets within an enterprise are updated more rapidly. The tooling has advanced thanks to the Hadoop project leading to a wide range of open source projects that feed into one another.

But data problems across an enterprise still exist. Business analysts, data scientists, and data engineers struggle to communicate with each other. The CFO still can’t get a question about revenue projections answered instantly. Instead of instant answers, we live in a world of friction, batch processing, and monthly reports.

And this is not just true of old enterprises like P&G. It is true of newer startups like Uber, Airbnb, and Netflix. It seems that no amount of engineers and financial windfall can completely cure the frictions of the modern data platform.

Tomer Shiran started Dremio to address the long-lived problems of data management, data access, and data governance within an enterprise. Dremio connects databases, storage systems, and business intelligence tools together, and uses intelligent caching to make commonly used queries within an organization more readily accessible.

Dremio is an ambitious project that spent several years in stealth before launching. In today’s episode, Tomer gives a history of data engineering and provides his perspective on how the data problems within an organization can be diminished. Full disclosure: Dremio is a sponsor of Software Engineering Daily.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.