Data Mesh with Zhamak Deghani

Data engineering involves numerous tools–a data lake, databases, data warehouses, numerous APIs, streaming systems, and microservices. There is no shortage of ways to interact with data and manage data, but many companies are struggling to figure out design patterns and best practices for how to manage data and build data infrastructure.

Zhamak Dehgani is a principal consultant and portfolio director with ThoughtWorks, where she works with enterprises to improve their software systems and workflows. She is the author of an article called How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh and she joins the show to discuss her perspective on data infrastructure, as well as the “data mesh” concept that she has coined.

Data mesh represents the movement away from having a centralized data lake that all teams interact with, and towards having different data products and individual data management systems for individual domain teams.

ANNOUNCEMENTS

  • FindCollabs is a place to find collaborators and build projects. We recently launched GitHub integrations. It’s easier than ever to find collaborators for your open source projects. And if you are looking for some people to start a project with, FindCollabs we have topic rooms that allow you to find other people who are interested in a particular technology, so that you can find people who are curious about React, or cryptocurrencies, or Kubernetes, or whatever you want to build with.
  • Podsheets is an open source podcast hosting platform that we recently launched. We are building Podsheets with the learnings from Software Engineering Daily, and our goal is to be the best place to host and monetize your podcast. If you have been thinking about starting a podcast, check out podsheets.com. 
  • New SEDaily app for iOS and for Android. It includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. You can comment on episodes and have discussions with other members of the community. I’ll be commenting on each episode, so if you hear an episode that you have some commentary on, jump onto the app, or on SoftwareDaily.com to share your thoughts. And you can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe. Altalogy is the company who has been developing much of the software for the newest app, and if you are looking for a company to help you with your mobile and web development, I recommend checking them out.    

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

With MongoDB Atlas, you can take advantage of MongoDB’s flexible document data model as a fully automated cloud service. MongoDB Atlas handles all the costly database operations and admin tasks that you’d rather not spend time on, like security, high availability, data recovery, monitoring, and elastic scaling.Try MongoDB Atlas for free today! Visit mongdb.com/se to learn more.

Datadog unites metrics, traces, and logs in one platform so you can get full visibility into your infrastructure and applications. Check out new features like Trace Search & Analytics for rapid insights into high-cardinality data, and Watchdog, an auto-detection engine that alerts you to performance anomalies across your applications. Datadog makes it easy for teams to monitor every layer of their stack in one place, but don’t take our word for it—start a free trial today & Datadog will send you a T-shirt! softwareengineeringdaily.com/datadog

SpringOne Platform is a conference for learning about building scalable web applications. SpringOne Platform is organized by Pivotal, the company that has helped build open source technologies such as Spring and Cloud Foundry. SpringOne Platform 2019 is in Austin, Texas October 7-10, and you can get $200 off your pass by going to softwareengineeringdaily.com/spring, and using promo code S1P200_SED.

GoCD is a continuous delivery tool from ThoughtWorks. If you have heard about continuous delivery, but you don’t know what it looks like in action, try the GoCD test drive at gocd.org/sedaily. GoCD’s test drive will set up example pipelines for you to see how GoCD manages your continuous delivery workflows. Visualize your deployment pipelines and understand which tests are passing and which tests are failing.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.