Transcript: When to use vector search (and when NOT to)

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

• Video URL: https://www.youtube.com/watch?v=Ryme6rUbhX8
• Video ID: Ryme6rUbhX8
• Word Count: 1,161 words

Full Transcript

Semantic search with vector databases, commonly called vector search, it's all the rage. It's tempting to treat it as a golden hammer, and use it to search all the things. But how do you cut through the hype and know when you should use the cool new tech? Tech that you really, really, really want to use and when to use more traditional search mechanisms. Well, that's what we're going to talk about today. Fundamentally, there are two types of data. Structured data and unstructured data. Structured data is data that has a schema. It's the strings, numbers, and booleans we've been using our entire careers. It lives in tables and JSON documents and key value databases like Reddus. Unstructured data, it's a little bit trickier. It lives in files. It's the emails and chat logs and images that we use every day. And it's everywhere. Traditionally, there was only one way to search this data. Structured search. Odds are you're familiar with structured search, although you probably don't call it that. Structured search is, as the name suggests, ideal for searching structured data. It's the SQL you're using to query tables, the JSON path to find elements in your documents, or the Reddus query to find the keys containing your data. The main thing about these queries is that they match values exactly. Query the customer table given an email address. Search the JSON document for a phone number. Fls in the catalog to those costing between $50 and $100. But there's a new kid in town. Semantic search. Semantic search uses AI models to turn unstructured data like images or natural language into embeddings. Embeddings are a big topic. There are links below to videos explaining them in greater detail, but at their simplest, they are vectors that encode the meaning or semantics of data. And since they're vectors, you can compare them. The more similar the vectors, the more similar the data. The less similar the vector, the less similar the data. This is huge because it allows you to query your unstructured data on its meaning instead of an exact match. So, when should you use the traditional approach of structure search and when should you use the cool new semantic search stuff? Well, structured search is great for structured data, so you'd probably want to use that for well, structured data. And semantic search lets you search unstructured things like natural language and images. So, you should probably use semantic search for unstructured data. Easy enough. Thanks for watching the video. Wait, it's actually a little more nuanced than that. See, we have two types of data, structured and unstructured, and two ways of searching, structured and semantic. So, what we have here is a matrix. Let's fill in the quadrants and see what we learn. First, we have structured data with structured queries. Match and filter data with a schema using exact queries. The most common example of this is the humble relational database and the SQL query. But there are plenty of other ways to store and query structured data. Next, unstructured data with structured queries. Find matches and data that doesn't have a schema like natural language using exact or almost exact queries. The best example of this is full text search with or without the stemming and fuzzy matching. But there are also image analysis techniques that can compare things like color, brightness or symmetry. Unstructured data with semantic search. Take images or natural language and convert them into embeddings. Then compare the embeddings. The star here is of course vector databases which store embeddings and match them using metrics like cosine similarity or uklidian distance. The last one structured data with semantic search is one that you might not have heard of. Entity embeddings. An entity embedding is structured data like JSON i.e. the entity that has been run through an AI model to encode the semantics of that data into an embedding. This isn't that common, but certainly interesting. It's best accomplished by using custom models that have been trained on your data in your domain. However, you can also get decent results by simply making the data into a sentence and using a sentence embed to create the embedding or sometimes just using the JSON as is. Then of course, you load these embeddings into a vector database and look for matches. So, which technique should you use? Should you embed all the things or should you do simple structured filtering? Well, as always, it depends. Reality is messy and there's often a messy mix of structured and unstructured data in your real world database. And individual pieces of data could even be thought of as structured or unstructured depending on how you want to use them. As an example, let's look at the things you might have in a product catalog. Images and descriptions. These are unstructured items that could be searched semantically or in the case of the description structurally with full text search. Prices, colors, and categories. These could be simple structured queries. You filter on simple structured data or they could be semantically searched by converting them to entity embeddings. Product names. These could be either structured or unstructured depending on how you want to find the specific product. Do you want to find it by matching the name or to find products with similar names? And you often want to use these techniques together. searching for, for example, products that are semantically similar but under a particular price. By the way, this is called hybrid search and we have a video on it. Links in the description below. So, I didn't really answer the question we started with because, well, there isn't one answer. It all depends on what you need and the needs of your users. That said, here are a couple of rules of thumb. Favor structured search over semantic search. Semantic search enables all sorts of amazing use cases that weren't even possible a decade ago, but can be overused outside of those use cases. Structured search is tried, true, and frankly faster. Semantic search is usually the way to go for image searches, but there are other statistical options that search on color, brightness, or symmetry. When searching text, if meaning matters more than precision, go with semantic search. If precision matters more than semantics, go with full text search. Entity embeddings are often more than you need, but can be useful for use cases like recommendations, fraud detection, and data dduplication. So, I've given you more of an exploration of search options and what they do than a simplistic answer or a list of best practices. And this is exactly what you need when deciding if you should use the shiny golden hammer that is semantic search or a more traditional tool like structured search. So, use this knowledge, go forth and make amazing software and thanks for watching.

Timestamped Segments

• [00:00] Semantic search with vector databases, commonly called vector search, it's all the rage. It's tempting to treat it as a golden hammer, and use it to search all the things. But how do you cut through the hype and know when you should use the cool new tech? Tech that you really, really, really want to use and when to use more traditional search mechanisms. Well, that's what we're going to talk about today.
• [00:21] Fundamentally, there are two types of data. Structured data and unstructured data. Structured data is data that has a schema. It's the strings, numbers, and booleans we've been using our entire careers. It lives in tables and JSON documents and key value databases like Reddus. Unstructured data, it's a little bit trickier. It lives in files. It's the emails and chat logs and images that we use every day. And it's everywhere.
• [00:45] Traditionally, there was only one way to search this data. Structured search. Odds are you're familiar with structured search, although you probably don't call it that. Structured search is, as the name suggests, ideal for searching structured data. It's the SQL you're using to query tables, the JSON path to find elements in your documents, or the Reddus query to find the keys containing your data. The main thing about these queries is that they match values exactly. Query the customer table given an email address. Search the JSON document for a phone number. Fls in the catalog to those costing between $50 and $100. But there's a new kid in town.
• [01:16] Semantic search. Semantic search uses AI models to turn unstructured data like images or natural language into embeddings. Embeddings are a big topic. There are links below to videos explaining them in greater detail, but at their simplest, they are vectors that encode the meaning or semantics of data. And since they're vectors, you can compare them. The more similar the vectors, the more similar the data. The less similar the vector, the less similar the data. This is huge because it allows you to query your unstructured data on its meaning instead of an exact match. So, when should you use the traditional approach of structure search and when should you use the cool new
• [01:48] semantic search stuff? Well, structured search is great for structured data, so you'd probably want to use that for well, structured data. And semantic search lets you search unstructured things like natural language and images. So, you should probably use semantic search for unstructured data. Easy enough. Thanks for watching the video. Wait, it's actually a little more nuanced than that. See, we have two types of data, structured and unstructured, and two ways of searching, structured and semantic.
• [02:16] So, what we have here is a matrix. Let's fill in the quadrants and see what we learn. First, we have structured data with structured queries. Match and filter data with a schema using exact queries. The most common example of this is the humble relational database and the SQL query. But there are plenty of other ways to store and query structured data. Next, unstructured data with structured queries. Find matches and data that doesn't have a schema like natural language using exact or almost exact queries. The best example of this is full text search with or without the stemming and fuzzy matching. But there are also image analysis techniques that can compare things like color,
• [02:48] brightness or symmetry. Unstructured data with semantic search. Take images or natural language and convert them into embeddings. Then compare the embeddings. The star here is of course vector databases which store embeddings and match them using metrics like cosine similarity or uklidian distance. The last one structured data with semantic search is one that you might not have heard of. Entity embeddings. An entity embedding is structured data like JSON i.e. the entity that has been run through an AI model to encode the semantics of that data into an embedding. This isn't that common, but certainly interesting. It's best
• [03:22] accomplished by using custom models that have been trained on your data in your domain. However, you can also get decent results by simply making the data into a sentence and using a sentence embed to create the embedding or sometimes just using the JSON as is. Then of course, you load these embeddings into a vector database and look for matches. So, which technique should you use? Should you embed all the things or should you do simple structured filtering? Well, as always, it depends. Reality is messy and there's often a messy mix of structured and unstructured data in your real world database. And individual pieces of data
• [03:55] could even be thought of as structured or unstructured depending on how you want to use them. As an example, let's look at the things you might have in a product catalog. Images and descriptions. These are unstructured items that could be searched semantically or in the case of the description structurally with full text search. Prices, colors, and categories.
• [04:15] These could be simple structured queries. You filter on simple structured data or they could be semantically searched by converting them to entity embeddings. Product names. These could be either structured or unstructured depending on how you want to find the specific product. Do you want to find it by matching the name or to find products with similar names? And you often want to use these techniques together.
• [04:33] searching for, for example, products that are semantically similar but under a particular price. By the way, this is called hybrid search and we have a video on it. Links in the description below. So, I didn't really answer the question we started with because, well, there isn't one answer. It all depends on what you need and the needs of your users. That said, here are a couple of rules of thumb. Favor structured search over semantic search. Semantic search enables all sorts of amazing use cases that weren't even possible a decade ago, but can be overused outside of those use cases. Structured search is tried, true, and frankly faster. Semantic search is
• [05:06] usually the way to go for image searches, but there are other statistical options that search on color, brightness, or symmetry. When searching text, if meaning matters more than precision, go with semantic search. If precision matters more than semantics, go with full text search. Entity embeddings are often more than you need, but can be useful for use cases like recommendations, fraud detection, and data dduplication.
• [05:27] So, I've given you more of an exploration of search options and what they do than a simplistic answer or a list of best practices. And this is exactly what you need when deciding if you should use the shiny golden hammer that is semantic search or a more traditional tool like structured search. So, use this knowledge, go forth and make amazing software and thanks for watching.