Transcript: What is a Vector Database？ Explained Simply with Real-World Examples

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

• Video URL: https://www.youtube.com/watch?v=He4Ft2lzNwc
• Video ID: He4Ft2lzNwc
• Word Count: 2,215 words

Full Transcript

Hello, I'm Brian Samboden and today I want to tell you about vector databases. Let's delve into the world of vector databases, the motivations for their existence, what they enable us to accomplish, and how the everexpanding world of artificial intelligence and machine learning has created renewed interest in the technology. Imagine it's the late 1990s and you created an online e-commerce retailer to complement your brickandmortar store. The data comes from a set of published paper cataloges. The catalog data lends itself perfectly to being stored in a relational database. New products are added ahead of inventory availability. Customers search for products using your brand new search page and everybody's happy. The type of data you were dealing with is structured data which is already in tabular form or lends itself to being transformed into tabular form. Tabular data is the perfect fit for a relational database and therefore manipulating and searching over that data became a matter of knowing how to talk effectively to a relational database. Most early e-commerce search experiences were just a few SQL queries string together with some blue code. The popularity of the web kicked off an exponential increase in digital content, marking the beginning of the big data era. In the mid 1990s, online shopping platforms like Amazon and eBay started gaining popularity over their brickandmortar equivalents. Vast amounts of transactional data were being generated by the minute. By the early 2000s, the web 2.0 era, the popularity of interactive and social web platforms led to a surge in userenerated content, including blogs, reviews, and comments, adding to the pool of structured data. Then in the mid200s, social media took root in society with platforms like Facebook, Twitter, and Instagram, radically changing how people create and share information contributed significantly to the volume of unstructured data. All of these events were compounded by the rapid adoption of cellular telephones worldwide and the internet of things IoT which suddenly turned many everyday devices like thermostats and doorbell cameras into sources of data. That brings us to today a world in which the balance of data has changed radically. It is estimated that at the beginning of the decade 80% of all data generated by organizations was unstructured data. This percentage has grown at a compound annual rate of more than 35%. Today, structured data is in the minority, while semistructure, quasi structure, and fully unstructured data make up the vast majority. More data typically means more insights, but only if we can act on it effectively and on time. We need ways to effectively store, manage, and search over large amounts of primarily unstructured data. But what makes unstructured data like images, documents, and videos hard to deal with in the traditional sense? First, unstructured data comes in many different flavors, each requiring different handling and storage methods, complicating the underlying implementations. Relational databases introduce blob and club fields to store and structure data. Still, these fields are not optimized for any data type in particular. They are known for degrading the performance of relational databases. But without the ability to search over set data, your unstructured data is only of value if associated with searchable structured data. Before the advent of powerful machine learning models, making unstructured data searchable require a significant effort to pre-process the data. Full text search engines help with text content. But for other data types, techniques that can be categorized as feature engineering had to be employed to extract structured data from the unstructured data. Techniques like tagging, keyword extraction, manual annotation, and regular expressions were used to pre-process and extract features from the data. One place where the impact of handling unstructured data is felt heavily is in search interfaces. User expectations about what is possible have risen to new levels. Visual search has become commonplace. In visual search interfaces, users take a photo and the search interface returns similar products in the company's catalog. Similarly, the usage of natural language to search for products is expected. This concept of finding similar things is directly applied to product recommendations where your past purchases or browsing history, it's used to find similar items. Some systems have extended this to find similar users and then use those users purchases to recommend products. Modern search interfaces need to capture meaning and context. The richest sources for this information might be buried in your unstructured data. But as we learn unstructured data, it's highdimensional and extracting this valuable information used to be a painstaking experience. Unstructured data such as a picture can pack a multitude of features. Extracted features allow us to create a compact form of our data. This representation is a vector in ndimensional space. Here we pick only three dimensions so that we can visualize the vector. This dress can be represented by a vector made from the collar pattern and the material. To work effectively with this multi-dimensional vectors, they are mapped to numeric values and normalized to a fixed range typically between minus1 and one. The advantages of unstructured data being represented as vectors are many. First, vectors provide a quantitative representation of the features of an object. And second, they put our data in the language of linear algebra, which gives us the tools to manipulate and compare them. Once a collection of vectors is created, you can employ various techniques to compare them. In a multi-dimensional space, vectors representing similar data points are expected to be clustered together or at least be closer to each other as opposed to vectors representing dissimilar data. This principle is behind many machine learning and data analysis techniques, especially those used in pattern recognition, clustering, and nearest neighbor searches. Here are some of the common similarity and distance metrics that measure the closeness or alikeness between vectors in a multi-dimensional space. For example, with cosine similarity, we use the cosine of the angle between our multi-dimensional vectors, effectively gauging how similar their orientations are. In particular, values closer to one indicate the vectors are similar. Therefore, the items those vectors represent are also similar. based on the features present in the vectors. Vectorization is the secret sauce that allow us to standardize, store and operate on unstructured data. But before the advent of powerful machine learning models, creating efficient vectors was at best a semi-automated process and at worst depending on an army of humans to label and tag raw data. The magic of ML models as vectorization engines is that they transform complex highdimensional unstructured data into a lowerdimensional numerical form that captures the essence of the data. This process enables efficient and effective comparisons, searches, and analysis in a way that traditional data processing methods cannot. By distilling intricate patterns, textures, and relationships into concise numerical represented vectors, these models facilitate the handling, retrieval, and understanding of diverse data types from images and text to audio and beyond, thus unlocking the potential for advanced applications and insights. Now that we understand the advantages of vectorizing unstructured data, we need a suitable system to keep and manipulate our vectors. Generating the vector embeddings is referred to as vectorization typically performed outside the database. Once we have a collection of vectors, a vector database becomes the system of record. Store vectors are indexed and each collection of vectors gets a corresponding index. Each vector collection represents an embedding space, a highdimensional space where each point or vector represents a piece of data. When it comes to querying the database, the query data must also be vectorized using the same technique used to vectorize the collection, which results in a query vector that can now be used by the query engine to find nearby vectors. Vector databases must be designed to handle million scale embedding spaces and beyond with searches that return results in subsecond scale without sacrificing accuracy. Quering on a vector database aka vector search entails identifying data points similar to a given sample. As we learn the sample data has to be converted into a query vector which gives a location in our multi-dimensional embedding space. The rest of the process entails finding a number of existing vectors nearest the query vector. Simple, right? With small embedding spaces, this is not an issue. In a simple flat indexing scheme, we calculate the distance between our input vector and all the vectors in the database and return the k nearest neighbors. The issue arises when the space of vectors to search is in the millions or billions. When we perform a vector search, we use the index built for the target collection. There are many schemes to build an index for a vector collection. And as we learned, the most straightforward approach is a flat index. A flat index is a brute force approach that returns an exact set of neighbors to the query vector. They are flat because the vectors in the index are unchanged and provide the most accurate results. Imagine we have a query vector and we want to find the n most similar matches in database. With a flat index, we first calculate the distance using a distance metric such as uklidian or manhattan distance. However, this brute force approach is computationally intensive, especially for highdimensional vectors and it is not scalable for large data sets. In practice, vector databases use approximations to get a set of nearest neighbors at an affordable computational price. We refer to these techniques collectively asn approximate nearest neighbor. There are several techniques and algorithms to precomputee distances between vectors and create groupings or clusters of similar vectors, including locality sensitive hashing and hierarchical navigable small word. Such techniques morph the problem from finding individual neighbors to finding closed neighborhoods first, reducing the number of computations by orders of magnitude. Let's look at how HNSW creates an efficient indexing structure for a vector database. This technique organizes the data into layers with each layer representing a different granularity of the search pace. Here layer 2 the topmost or entry layer receives two vectors just added to the database. Nodes in each layer are interconnected creating an undirected graph. The connecting edge representing the proximity or similarity between the vectors. In higher up layers, nodes are sparssely connected, representing longer jumps in the graph. As we add more nodes, we populate subsequent layers more densely. The lower the layer, the more refined the search step becomes. At query time, given a query vector, the navigation starts at the top small world. In our example, the entry layer, layer two, using the entry point closest to the query vector. In each layer, we greedily move closer to the query vector. Once we can get any closer, we move down to the next layer. The search ends when we no longer get closer to the query vector. The traverse path through the layers efficiently narrows down the candidate vectors leading to a refined set of nearest neighbors by the end of the search process. What does it take for a system to be called a vector database? It's not just about storing vectors. It's about optimizing data management for performance and efficiency. We can divide the features of vector databases into two groups. Vector capabilities, those essential to managing and extracting value out of our unstructured data and those operational features that all database systems must possess to be trusted with our production data. First, a vector database must have efficient scalable storage for highdimensional vector data. Advanced indexing and search capabilities are a must. The system should support fast, accurate querying, but allow for fine-tuning the trade-off between speed and accuracy. In the real world, vector queries don't stand alone. There's a need to be able to search across a combination of both structured data and unstructured data in a single pass. The ability to prefilter vector searches, it's crucial to finding specific items by feature and then narrowing down the results with vector similarity. There are wellestablished expectations from the operational side of a database management system. Multi-tenency enables a single server instance to serve multiple users or tenants with each tenants's data isolated from the others. High availability provides automated database resilience and mitigates hardware failures. Modern databases are expected to ensure a high availability of 59s. the ability to scale to handle large highdimensional data sets without degradation by scaling up scaling out and actively rebalancing and redistributing the data. Security is paramount. A vector database should offer robust features like encryption and access control aligning with data privacy standards. Whether dealing with long passages of text, videos, images, or even sounds, a vector database is optimized to store vector representations of unstructured data, allowing us to efficiently organize, search, and analyze this complex information in ways traditional databases can't. Vector databases enable rapid and accurate retrieval of similar items, transforming vast, unstructured data sets into actionable insights. They are indispensable for powering various applications from recommendation systems and personal assistance to advanced analytics in fields like genomics and linguistics. As we continue to generate and interact with ever growing volumes of unstructured data, the role of vector databases becomes increasingly vital, acting as a cornerstone in the ongoing journey of harnessing the full potential of big data and artificial intelligence.

Timestamped Segments

• [00:00] Hello, I'm Brian Samboden and today I want to tell you about vector databases. Let's delve into the world of vector databases, the motivations for their existence, what they enable us to accomplish, and how the everexpanding world of artificial intelligence and machine learning has created renewed interest in the technology. Imagine it's the late 1990s and you created an online e-commerce retailer to complement your brickandmortar store. The data comes from a set of published paper cataloges.
• [00:30] The catalog data lends itself perfectly to being stored in a relational database. New products are added ahead of inventory availability. Customers search for products using your brand new search page and everybody's happy. The type of data you were dealing with is structured data which is already in tabular form or lends itself to being transformed into tabular form. Tabular data is the perfect fit for a relational database and therefore manipulating and searching over that data became a matter of knowing how to talk effectively to a
• [01:02] relational database. Most early e-commerce search experiences were just a few SQL queries string together with some blue code. The popularity of the web kicked off an exponential increase in digital content, marking the beginning of the big data era. In the mid 1990s, online shopping platforms like Amazon and eBay started gaining popularity over their brickandmortar equivalents. Vast amounts of transactional data were being generated by the minute. By the early 2000s, the web 2.0 era, the popularity of
• [01:36] interactive and social web platforms led to a surge in userenerated content, including blogs, reviews, and comments, adding to the pool of structured data. Then in the mid200s, social media took root in society with platforms like Facebook, Twitter, and Instagram, radically changing how people create and share information contributed significantly to the volume of unstructured data. All of these events were compounded by the rapid adoption of cellular telephones worldwide and the internet of things IoT which suddenly
• [02:10] turned many everyday devices like thermostats and doorbell cameras into sources of data. That brings us to today a world in which the balance of data has changed radically. It is estimated that at the beginning of the decade 80% of all data generated by organizations was unstructured data. This percentage has grown at a compound annual rate of more than 35%.
• [02:34] Today, structured data is in the minority, while semistructure, quasi structure, and fully unstructured data make up the vast majority. More data typically means more insights, but only if we can act on it effectively and on time. We need ways to effectively store, manage, and search over large amounts of primarily unstructured data. But what makes unstructured data like images, documents, and videos hard to deal with in the traditional sense? First, unstructured data comes in many different flavors, each requiring
• [03:07] different handling and storage methods, complicating the underlying implementations. Relational databases introduce blob and club fields to store and structure data. Still, these fields are not optimized for any data type in particular. They are known for degrading the performance of relational databases. But without the ability to search over set data, your unstructured data is only of value if associated with searchable structured data. Before the advent of powerful machine learning models, making unstructured data searchable require a
• [03:41] significant effort to pre-process the data. Full text search engines help with text content. But for other data types, techniques that can be categorized as feature engineering had to be employed to extract structured data from the unstructured data. Techniques like tagging, keyword extraction, manual annotation, and regular expressions were used to pre-process and extract features from the data. One place where the impact of handling unstructured data is felt heavily is in search interfaces.
• [04:11] User expectations about what is possible have risen to new levels. Visual search has become commonplace. In visual search interfaces, users take a photo and the search interface returns similar products in the company's catalog. Similarly, the usage of natural language to search for products is expected. This concept of finding similar things is directly applied to product recommendations where your past purchases or browsing history, it's used to find similar items. Some systems have extended this to find similar users and
• [04:44] then use those users purchases to recommend products. Modern search interfaces need to capture meaning and context. The richest sources for this information might be buried in your unstructured data. But as we learn unstructured data, it's highdimensional and extracting this valuable information used to be a painstaking experience.
• [05:06] Unstructured data such as a picture can pack a multitude of features. Extracted features allow us to create a compact form of our data. This representation is a vector in ndimensional space. Here we pick only three dimensions so that we can visualize the vector. This dress can be represented by a vector made from the collar pattern and the material. To work effectively with this multi-dimensional vectors, they are mapped to numeric values and normalized to a fixed range typically between minus1 and one. The
• [05:39] advantages of unstructured data being represented as vectors are many. First, vectors provide a quantitative representation of the features of an object. And second, they put our data in the language of linear algebra, which gives us the tools to manipulate and compare them. Once a collection of vectors is created, you can employ various techniques to compare them. In a multi-dimensional space, vectors representing similar data points are expected to be clustered together or at least be closer to each other as opposed to vectors representing dissimilar data.
• [06:12] This principle is behind many machine learning and data analysis techniques, especially those used in pattern recognition, clustering, and nearest neighbor searches. Here are some of the common similarity and distance metrics that measure the closeness or alikeness between vectors in a multi-dimensional space. For example, with cosine similarity, we use the cosine of the angle between our multi-dimensional vectors, effectively gauging how similar their orientations are. In particular, values closer to one indicate the
• [06:45] vectors are similar. Therefore, the items those vectors represent are also similar. based on the features present in the vectors. Vectorization is the secret sauce that allow us to standardize, store and operate on unstructured data. But before the advent of powerful machine learning models, creating efficient vectors was at best a semi-automated process and at worst depending on an army of humans to label and tag raw data.
• [07:21] The magic of ML models as vectorization engines is that they transform complex highdimensional unstructured data into a lowerdimensional numerical form that captures the essence of the data. This process enables efficient and effective comparisons, searches, and analysis in a way that traditional data processing methods cannot. By distilling intricate patterns, textures, and relationships into concise numerical represented vectors, these models facilitate the handling, retrieval, and understanding of diverse data types from images and
• [07:56] text to audio and beyond, thus unlocking the potential for advanced applications and insights. Now that we understand the advantages of vectorizing unstructured data, we need a suitable system to keep and manipulate our vectors. Generating the vector embeddings is referred to as vectorization typically performed outside the database. Once we have a collection of vectors, a vector database becomes the system of record. Store vectors are indexed and each collection of vectors gets a corresponding index.
• [08:28] Each vector collection represents an embedding space, a highdimensional space where each point or vector represents a piece of data. When it comes to querying the database, the query data must also be vectorized using the same technique used to vectorize the collection, which results in a query vector that can now be used by the query engine to find nearby vectors. Vector databases must be designed to handle million scale embedding spaces and beyond with searches that return results in subsecond scale without sacrificing accuracy. Quering on a vector database
• [09:02] aka vector search entails identifying data points similar to a given sample. As we learn the sample data has to be converted into a query vector which gives a location in our multi-dimensional embedding space. The rest of the process entails finding a number of existing vectors nearest the query vector. Simple, right? With small embedding spaces, this is not an issue.
• [09:27] In a simple flat indexing scheme, we calculate the distance between our input vector and all the vectors in the database and return the k nearest neighbors. The issue arises when the space of vectors to search is in the millions or billions. When we perform a vector search, we use the index built for the target collection. There are many schemes to build an index for a vector collection. And as we learned, the most straightforward approach is a flat index. A flat index is a brute force approach that returns an exact set
• [10:00] of neighbors to the query vector. They are flat because the vectors in the index are unchanged and provide the most accurate results. Imagine we have a query vector and we want to find the n most similar matches in database. With a flat index, we first calculate the distance using a distance metric such as uklidian or manhattan distance. However, this brute force approach is computationally intensive, especially for highdimensional vectors and it is not scalable for large data sets. In practice, vector databases use
• [10:34] approximations to get a set of nearest neighbors at an affordable computational price. We refer to these techniques collectively asn approximate nearest neighbor. There are several techniques and algorithms to precomputee distances between vectors and create groupings or clusters of similar vectors, including locality sensitive hashing and hierarchical navigable small word. Such techniques morph the problem from finding individual neighbors to finding closed neighborhoods first, reducing the number of computations by orders of magnitude.
• [11:09] Let's look at how HNSW creates an efficient indexing structure for a vector database. This technique organizes the data into layers with each layer representing a different granularity of the search pace. Here layer 2 the topmost or entry layer receives two vectors just added to the database. Nodes in each layer are interconnected creating an undirected graph. The connecting edge representing the proximity or similarity between the vectors. In higher up layers, nodes are sparssely connected, representing longer
• [11:43] jumps in the graph. As we add more nodes, we populate subsequent layers more densely. The lower the layer, the more refined the search step becomes. At query time, given a query vector, the navigation starts at the top small world. In our example, the entry layer, layer two, using the entry point closest to the query vector. In each layer, we greedily move closer to the query vector. Once we can get any closer, we move down to the next layer. The search ends when we no longer get closer to the query vector. The traverse path through
• [12:17] the layers efficiently narrows down the candidate vectors leading to a refined set of nearest neighbors by the end of the search process. What does it take for a system to be called a vector database? It's not just about storing vectors. It's about optimizing data management for performance and efficiency. We can divide the features of vector databases into two groups.
• [12:39] Vector capabilities, those essential to managing and extracting value out of our unstructured data and those operational features that all database systems must possess to be trusted with our production data. First, a vector database must have efficient scalable storage for highdimensional vector data.
• [12:59] Advanced indexing and search capabilities are a must. The system should support fast, accurate querying, but allow for fine-tuning the trade-off between speed and accuracy. In the real world, vector queries don't stand alone. There's a need to be able to search across a combination of both structured data and unstructured data in a single pass. The ability to prefilter vector searches, it's crucial to finding specific items by feature and then narrowing down the results with vector
• [13:31] similarity. There are wellestablished expectations from the operational side of a database management system. Multi-tenency enables a single server instance to serve multiple users or tenants with each tenants's data isolated from the others. High availability provides automated database resilience and mitigates hardware failures. Modern databases are expected to ensure a high availability of 59s.
• [13:56] the ability to scale to handle large highdimensional data sets without degradation by scaling up scaling out and actively rebalancing and redistributing the data. Security is paramount. A vector database should offer robust features like encryption and access control aligning with data privacy standards. Whether dealing with long passages of text, videos, images, or even sounds, a vector database is optimized to store vector representations of unstructured data, allowing us to efficiently organize, search, and analyze this complex
• [14:30] information in ways traditional databases can't. Vector databases enable rapid and accurate retrieval of similar items, transforming vast, unstructured data sets into actionable insights. They are indispensable for powering various applications from recommendation systems and personal assistance to advanced analytics in fields like genomics and linguistics. As we continue to generate and interact with ever growing volumes of unstructured data, the role of vector databases becomes increasingly vital, acting as a cornerstone in the ongoing
• [15:02] journey of harnessing the full potential of big data and artificial intelligence.