Pulsar Messaging with Lewis Kaneshiro

Message broker systems decouple the consumers and producers of a message channel. In previous shows, we have explored ZeroMQ, PubNub, Apache Kafka, and NATS. In this episode, we talk about another message broker: Apache Pulsar.

Pulsar is an open source distributed pub-sub message system originally created at Yahoo. It was used to scale products with high volumes of users–such as Yahoo Mail.

There are three components of a Pulsar deployment: the Pulsar broker (which handles the message brokering), Apache Bookkeeper (which handles the durable storage of the messages), and Apache Zookeeper, which manages the distributed coordination.

Lewis Kaneshiro joins the show to describe how Apache Pulsar works, and how it compares to other messaging systems like Apache Kafka. Lewis is the CEO of Streamlio, a company that builds messaging and stream processing systems for enterprises, and uses Pulsar in its core product.


Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Segment allows us to gather customer data from anywhere and send that data to any analytics tool. Segment is the customer data infrastructure that has saved us from writing duplicate code across all of the different platforms that we want to analyze. And if you’re using cloud apps such as – Mailchimp, Marketo, Intercom, AppNexus, Zendesk–you can integrate with all of these different tools and centralize your customer data in one place–with Segment. To get a free 90-day trial, signup for Segment at segment.com and enter SEDaily in the “How did you hear about us box?” during signup.

The GCP Podcast covers the technologies that Google Cloud is building–through interviews with the people building them. And these are often unique, Google cloud services–like BigQuery, AutoML, and Firebase. So if you want to stay on the leading edge of what is being released at Google, and how these new technologies are built, check out gcppodcast.com. I’ve been a listener for a few years now, and the content is consistently good–a few of my favorite recent episodes are the interview with Vint Cerf and the show about BigQuery. You can find those episodes and more by going to gcppodcast.com.

Sumo Logic is a cloud-native, machine data analytics service that helps you Run and Secure your Modern Application. If you are feeling the pain of managing your own log, event, and performance metrics data, check out sumologic.com/sedaily. Even if you have tools already, it’s worth checking out Sumo Logic and seeing if you can leverage your data even more effectively, with real-time dashboards and monitoring, and improved observability – to improve the uptime of your application and keep your day-to-day runtime more secure. Check out sumologic.com/sedaily for a free 30-day Trial of Sumo Logic, to find out how Sumo Logic can improve your productivity and your application observability–wherever you run your applications.

GoCD is a continuous delivery tool created by ThoughtWorks. GoCD agents use Kubernetes to scale as needed. Check out gocd.org/sedaily and learn about how you can get started. GoCD was built with the learnings of the ThoughtWorks engineering team, who have talked about building the product in previous episodes of Software Engineering Daily. It’s great to see the continued progress on GoCD with the new Kubernetes integrations–and you can check it out for yourself at gocd.org/sedaily.

Software Weekly

Software Weekly

Subscribe to Software Weekly, a curated weekly newsletter featuring the best and newest from the software engineering community.