Stream Processing with Satish Mittal

stream processing frameworks

satish-mittal

“We still need to see in the long run how much of community and industry adoption is there. Because at the end of the day, these are the single two most important things which define and determine the success of any platform.”

Real-time stream processing of data is becoming widely adopted in the efforts to manage and process “big data”. Some of the top frameworks for processing data streams include Storm, Spark, Samza and Flink.

Satish Mittal is an architect at InMobi, an ad platform that needs to deal with processing large volumes of ad data with tight time demands. He recently conducted an analysis of streaming frameworks to determine which of the tools was right for the application at InMobi.

Questions

  • What is real-time stream processing?
  • What is micro-batching, and what are the problems with it?
  • What were the streaming platforms you were trying to decide between at InMobi?
  • Did you look at the Lambda architecture when you were building your system?
  • How do Spark and Storm compare when it comes to message delivery guarantees and fault tolerance?
  • What platform did you finally end up choosing?
  • What do you see as the future of the big data stack?

Links

Sponsors

Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired.

Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Software Daily

Software Daily

 
Subscribe to Software Daily, a curated newsletter featuring the best and newest from the software engineering community.