Visual Search with Neel Vadoothker

If I have a picture of a dog, and I want to search the Internet for pictures that look like that dog, how can I do that?

I need to make an algorithm to build an index of all the pictures on the Internet. That index can define the different features of my images. I can find mathematical features in each image that describe that image. The mathematical features can be represented by a matrix of numbers. Then I can run the same algorithm on the picture of my dog, which will make another matrix of numbers. I can compare the matrix representing my dog picture to the matrices of all the pictures on the internet.

This is what Google and Facebook do–and we covered this topic in our episode about similarity search a few weeks ago. Today, we evaluate a similar problem: searching images within Squarespace. Squarespace is a platform where users can easily build their own website for blogging, e-commerce, or anything else.

Neel Vadoothker is a machine learning engineer at Squarespace, and he joins the show to talk about how and why he built a visual similarity search engine.

If you like this episode, we have done many other shows about machine learning. You can check out our back catalog by going to or by downloading the Software Engineering Daily app for iOS, where you can listen to all of our old episodes, and easily discover new topics that might interest you. You can upvote the episodes you like and get recommendations based on your listening history. With 600 episodes, it is hard to find the episodes that appeal to you, and we hope the app helps with that.


Have you been thinking you’d be happier at a new job? If you’re dreaming about a new job and have been waiting for the right time to make a move, go to Hired makes finding work enjoyable. Hired uses an algorithmic job-matching tool in combination with a talent advocate who will walk you through the process of finding a better job. Check out to get a special offer for Software Engineering Daily listeners–a $600 signing bonus from Hired when you find that great job that gives you the respect and salary that you deserve as a talented engineer. 


GrammaTech CodeSonar helps development teams improve code quality with static analysis. It helps flag issues early in the development process, allowing developers to release better code faster. CodeSonar can easily be integrated into any development process. CodeSonar performs advanced static analysis of C, C++, Java, and even raw binary code. CodeSonar performs unique dataflow and symbolic execution analysis to aggressively scan for problems in your code. Just like battleships use sonar to detect objects deep underwater, engineers use CodeSonar to detect subtle problems deep within their code. Go to to get your free 30-day trial, exclusively for Software Engineering Daily listeners and unleash the power of advanced static analysis.

Who do you use for log management? I want to tell you about Scalyr, the first purpose built log management tool on the market. Most tools on the market utilize text indexing search, which is great… for indexing a book. But if you want to search logs, at scale, fast… it breaks down. Scalyr built their own database from scratch: the system is fast. Most searches take less than 1 second. In fact, 99% of their queries execute in <1 second.  Companies like OKCupid, Giphy and CareerBuilder use Scalyr. It was built by one of the founders of Writely (aka Google Docs). Scalyr has consumer grade UI, that scales infinitely. You can monitor key metrics, trigger alerts, and integrate with PagerDuty. It’s easy to use and did we mention: lightning fast. Give it a try today. It’s free for 90 days at

Thanks to Symphono for sponsoring Software Engineering Daily. Symphono is a custom engineering shop where senior engineers tackle big tech challenges while learning from each other. Check it out at Thanks to Symphono for being a sponsor of Software Engineering Daily for almost a year now. Your continued support allows us to deliver content to the listeners on a regular basis.