StreamSets: DataOps and Smart Pipelines with Arvind Prabhakar

Jun 17, 2021 by SED

The company StreamSets is enabling DataOps practices in today’s enterprises. StreamSets is a data engineering platform designed to help engineers design, deploy, and operate smart data pipelines. StreamSets Data Collector is a codeless solution for designing pipelines, triggering CDC operations, and monitoring data in flight. StreamSets Transformer uses Apache Spark to generate insights about your data across multiple different platforms. Their Control Hub is the single hub for managing all of your data pipelines, data processing jobs, and execution engines.

In this episode we talk to Arvind Prabhakar, CTO at StreamSets. Arvind is also an Official Member of the Forbes Technology Council, and a Member, PMC Chair/Member, Committer, Mentor, and Contributor to multiple projects with the Apache Software Foundation. He was previously a Director of Engineering at Cloudera, and a Software Architect at Informatica before that.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com to get 15% off the first three months of audio editing and transcription services with code: SED. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Thanks to our sponsors

This episode is brought to you by:

cloud.google.com/devops
Today’s podcast is brought to you by Google Cloud and DORA research team. The team recently launched a survey to collect insights for the 2021 State of DevOps report and would love your input! The State of DevOps report is the largest and longest running research of its kind, providing insight into how we can improve software delivery performance with DevOps. By completing the survey, you get to shape the conversation on DevOps along with over 30 thousand software professionals who took the survey over the past six years. So what are you waiting for? Take the survey at cloud.google.com/devops!
softwareengineeringdaily.com/airflowsummit
The Apache Airflow community would like to invite our listeners to join Google, Astronomer, AWS, Electronic Arts, BBC, Pinterest and more leading companies on July 8th at the Airflow Summit 2021—a virtual conference designed for data engineers, data scientists and anyone with a need to author, schedule and monitor data pipelines using Python. The conference runs July 8th-16th and will be held in multiple time zones around the world. To discover what’s driving this excitement, check out the full agenda at softwareengineeringdaily.com/airflowsummit and register now to reserve your spot!
try TeamCity Cloud
TeamCity Cloud is a new continuous integration service that is completely hosted and managed by JetBrains. It is based on the original on-premises version of TeamCity, and shares most of its functionality. Multiplatform development, integration with popular build and test frameworks, real-time feedback, test history and test analysis – these are just a few of the many powerful features that can take your team to a new level of productivity. You can try TeamCity Cloud free of charge for 14 days. The trial period gives you 12,000 build credits (equivalent of 20 build hours on the Linux Small build agent), unlimited parallel builds, 120 GB of storage, and up to 3 self-hosted build agents. Get started with cloud CI/CD today!
softwareengineeringdaily.com/algolia
Algolia is a search API that gives engineering teams a complete toolkit for adding search to your products with minimal development time. Our SaaS platform removes complexities of building and integrating search while focusing on great developer experience. More than 10000 companies, including Under Armour, Fanatics, Medium, Lacoste, and NPR rely on Algolia to deliver relevant results in under 50ms. softwareengineeringdaily.com/algolia
getstream.io/SED
Stream provides an easy-to-integrate chat solution for any application. With robust SDKs and an API built for ease of use, scalability, reliability, and security, product teams can focus on what makes their app unique, rather than spending months on building a chat infrastructure. Stream’s feature-rich products include robust client-side SDKs for iOS, Android, React, React Native, Flutter, and support for the most commonly used server-side languages; scalable and secure APIs; and a beautiful UI kit. Check it out at getstream.io/SED.