Cloud Search with Liam Cavanagh

Search is part of almost every application. Users search for movies to watch. Engineers search through terabytes of log messages to find exceptions. Drivers search through maps to find a destination. Search remains an unsolved problem, with lots of room for optimization.

Many search applications have been built Elasticsearch, an open source distributed search engine. Elasticsearch is the code that powers some search-as-a-service products offered by major cloud providers. After eight years of open source development, Elasticsearch is excellent at core search functionalities, such as indexing data, sharding, and serving queries.

With improved access to machine learning tools, search applications can advance in new and interesting ways. For example, an incoming search query can be sent to an API for natural language processing before being served by the search engine. A natural language processing API can derive additional meaning from the query, adding metadata to a search query. Machine learning can also be applied to better understand how people are searching across your search index, and to optimize the search index to incorporate those user preferences.

Liam Cavanagh is the principal program manager on Azure Search. He joins the show to talk about the architecture of a search index, how search queries are served by an index, and how machine learning APIs can be used to improve queries.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.


Sponsors

Triplebyte is a company that connects engineers with top tech companies. We’re running an experiment and our hypothesis is that Software Engineering Daily listeners will do well above average on the quiz. Go to triplebyte.com/sedaily.

Digital Ocean is the easiest cloud platform to run and scale your application. Try it out today and get a free $100 credit–go to do.co/sedaily. Digital Ocean is a complete cloud platform to help developers and teams save time when running and scaling their applications.

Datadog is a cloud-scale monitoring platform for infrastructure and applications. And with Datadog’s new Live Container view, you can see every container’s health, resource consumption, and running processes in real time. See for yourself by starting a free trial and get a free Datadog T-shirt! softwareengineeringdaily.com/datadog.

Azure Container Service simplifies the deployment, management and operations of Kubernetes. Check out the Azure Container Service at aka.ms/sedaily.