Transcript: What is semantic search？

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

• Video URL: https://www.youtube.com/watch?v=o3XN4dImESE
• Video ID: o3XN4dImESE
• Word Count: 574 words

Full Transcript

A traditional searching system works by matching the words a user types with the words stored in a database or document collection. It usually looks for exact or partial matches without understanding the meaning behind the words. In a movie streaming service, for example, if a user doesn't know exactly what they're looking for, they might have a hard time finding the movie they want to watch. The movie synopsis is stored in a database as a cowboy doll feels threatened when a new space toy becomes his owner's favorite, but the user searches for jealous toy struggles with new rival, a traditional search system might not find the movie because the exact words don't line up. And this is exactly where semantic searching shines because instead of just matching words, it tries to understand the meaning behind what the user is asking. So even if the words are different like jealous toy and cowboy doll, a semantic search system can still connect the two ideas and bring up the right movie. Behind the scenes, this is possible thanks to vector similarity search. Vector similarity search is a technique that combines embedding models with vector databases. Embedding models are machine learning algorithms specializ in turning unstructured data like text, images, or audio into vectors. or in other words meaningful numerical representation of this data. These vectors or embeddings are then stored in a vector database that is a type of database used for efficiently storing and retrieving these vectors based on their proximity in the vector space model inside of the database. When the user types a new query, this query is also vectorzed using the same embedding model and then the resulting vectors compared to all the other vectors inside of the database. the most similar ones are retrieved and returned to the user. This way, even if the search words don't match the keywords in the database exactly, the system can still find similar results by using vector similarity search, focusing on the meaning of the words instead of just matching the text. On top of that, an ideal vector database lets you do hybrid search, too. This means you can combine vector similarity with traditional filtering to get even better results. For example, instead of just searching by the meaning of a movie synopsis, users could also narrow it down by release year, actors, or Jenners all in one query. This way, the system has a much better chance of finding the right movie. In this case, Toy Story. However, unlike traditional search, semantic search usually needs to search across a large number of actors in the database to find similar results. This makes it very important to have a vector database that is both fast and that is able to scale well for the system to stay efficient. And according to our benchmarks, Reddis is the fastest and most scalable vector database available today because we deliver the fastest queries and can scale to 1 billion vectors without adding latency. The links to these studies are in the description of this video. If you want to start experimenting with semantic search, vector similarity search, or many other use cases for vector databases today, you should check our Rady's AI resources GitHub repository that is packed with recipes and demos to help you start building fast. My name is Rafael Deio. Thank you for watching and as always, stay curious. [Music]

Timestamped Segments

• [00:00] A traditional searching system works by matching the words a user types with the words stored in a database or document collection. It usually looks for exact or partial matches without understanding the meaning behind the words. In a movie streaming service, for example, if a user doesn't know exactly what they're looking for, they might have a hard time finding the movie they want to watch.
• [00:21] The movie synopsis is stored in a database as a cowboy doll feels threatened when a new space toy becomes his owner's favorite, but the user searches for jealous toy struggles with new rival, a traditional search system might not find the movie because the exact words don't line up. And this is exactly where semantic searching shines because instead of just matching words, it tries to understand the meaning behind what the user is asking. So even if the words are different like jealous toy and cowboy doll, a semantic search
• [00:54] system can still connect the two ideas and bring up the right movie. Behind the scenes, this is possible thanks to vector similarity search. Vector similarity search is a technique that combines embedding models with vector databases. Embedding models are machine learning algorithms specializ in turning unstructured data like text, images, or audio into vectors. or in other words meaningful numerical representation of this data. These vectors or embeddings are then stored in a vector database that is a type of database used for
• [01:27] efficiently storing and retrieving these vectors based on their proximity in the vector space model inside of the database. When the user types a new query, this query is also vectorzed using the same embedding model and then the resulting vectors compared to all the other vectors inside of the database. the most similar ones are retrieved and returned to the user. This way, even if the search words don't match the keywords in the database exactly, the system can still find similar results by using vector similarity search, focusing on the
• [02:00] meaning of the words instead of just matching the text. On top of that, an ideal vector database lets you do hybrid search, too. This means you can combine vector similarity with traditional filtering to get even better results. For example, instead of just searching by the meaning of a movie synopsis, users could also narrow it down by release year, actors, or Jenners all in one query. This way, the system has a much better chance of finding the right movie. In this case, Toy Story. However, unlike traditional search, semantic
• [02:34] search usually needs to search across a large number of actors in the database to find similar results. This makes it very important to have a vector database that is both fast and that is able to scale well for the system to stay efficient. And according to our benchmarks, Reddis is the fastest and most scalable vector database available today because we deliver the fastest queries and can scale to 1 billion vectors without adding latency. The links to these studies are in the description of this video. If you want to start experimenting with semantic
• [03:06] search, vector similarity search, or many other use cases for vector databases today, you should check our Rady's AI resources GitHub repository that is packed with recipes and demos to help you start building fast. My name is Rafael Deio. Thank you for watching and as always, stay curious.
• [03:27] [Music]