Skip to main content
Version: v1.0 (Raw Transcripts)

Transcript: What is a vector database？

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

Full Transcript​

Vector databases have recently been popping all around and have constantly been defined as a gen tool but what if I told you that Vector databases have been around for longer than Ai and that it's much easier to understand how they work by taking AI out of the [Music] equation to understand V to databases I want you to First Look above and think about all the planets in the solar system what's the most similar Planet worth well it depends on how you're comparing all of them doesn't it I mean think with me if we're talking about mass then Venus is the most similar to Earth on the other hand if we're talking about temperature then Mars is the most similar one and you can actually see it mathematically but don't worry right here I won't get deep in mathematics or show you complicated equations here instead I want you to draw a straight line and divide this line into 1,000 equal parts where each part represents temperature degrees ranging fromus 500 to positive 500 and then let's place all the planets of the solar system on this line based on their average temperatures Now by looking at them you can already estimate which are the most similar Ones based on their proximity however if you want to be precise or you want to compare millions of points just by looking at them wouldn't be enough mathematically though we can determ how close these points are by calculating the actual distance between them on the line This is called the ukian distance and by calculating it computers can then determine how similar two different data are based on their numerical distance on this line but if if we were to decide which planet to colonize next we couldn't rely only one factor we'd have to consider more characteristics or features or let's say dimensions of these planets so let's add another dimension here let's talk about meas if we draw another Line crossing our first line now then also divide this line into equal parts and place each planet on that other line based on its mass we would now have a two dimensional coordinate system where one axis represents temperature and the other axis represents Mass now instead of just comparing Planet space on temperature alone we can also consider their Mass when determining similarity if we map these two points in this two dimensional space we now have a vector and if we calculate thean distance between them we can still find out how similar they are based on their proximity the ukan distance isn't the only way of calculating how close your points are in the multi-dimensional coordinate system in fact as we start adding many more vectors to it there are other algorithms that can perform better to determine what are the closest vectors to each other or in other words their nearest neighbors another way of doing that is by comparing the angle between them that is traced from the origin to each Vector in the multi-dimensional space this method known as cosign similarity measures how similar true vectors are based on their Direction rather than their magnitude all right let's up for a second here what I've seen so far is that Vector databases represent our data vectors or in other words numerical data points and that they determine similarity between these vectors mathematically based on algorithms such as the pleting distance and cosine similarity so a vector database is a database optimized for storing Vector representations of data and efficiently creating that data the distance among vectors is determined by how close vectors are to each other in this multi-dimensional Vector space where each Dimension represents a feature of our data all right but where there's a ey fit in this whole picture if we pull back the coordinate system we're talking about the Minal we can see that we're talking about only two dimensions and only nine different data points as humans we're completely capable of determining all of those however when we're talking about hundreds or even thousands of dimensions in millions or billions of data points this task becomes extremely complex and this is where Ai and the famous embedding models we've been hearing so much about come into play embedding models are machine learning algorithms that convert unstructured data like text images and sound into meaningful Vector representations they take complex non- numerical data like text images audio or even user behavior and convert them into vectors that capture the most important features unlike simple attributes like mass or temperature Concepts like meaning in text or styling images don't have a straightforward numerical representation AI embedded model solve this by learning how to map similar items closer together in a vector space for example in a text search instead of just looking for exact keyword matches and in model can place words like cat and fine closer together even though they're different words this means that when you search for cat we might still retri documents containing fine because their vectors are mathematically close this is how AI powered search engines recommendation systems and chat Bots find relevant information even when the exact words don't match by using embedded models to convert data into high dimensional vectors and then storing and searching these vectors efficiently in a vector database AI enables fast and accurate similarity searches whether it's only for text images or any other kind of data and that's why Vector data bases and AI embedding models go hand inand in modern search and recommendation systems to wrap up while Vector search itself is not new and has been applied for decades choosing the right Vector database is crucial especially when working at scale you need a database that can handle high performance similarity searches support billions of vectors and provide real time indexing and ing this is where reddis 8 stands out redis has long been known for its blazing fast performance due to its submillisecond operations and with the latest release it has introduced major performance improvements in Vector search benchmarks show that bres can handle billions of vectors with real-time latency making it the fastest Vector database available today with enhancements in latency reduction query engine scaling and realtime Vector insertions red is now provides a production ready solution for large scale Ai and search applications if you're looking for a vector database that delivers speed scalability and efficiency red is 8 is the way to go try it out today and experience the fastest Rish ever my name is Rafael Deo thank you for watching and as always stay curious

Timestamped Segments​

  • [00:00] Vector databases have recently been popping all around and have constantly been defined as a gen tool but what if I told you that Vector databases have been around for longer than Ai and that it's much easier to understand how they work by taking AI out of the [Music] equation to understand V to databases I want you to First Look above and think about all the planets in the solar system what's the most similar Planet worth well it depends on how you're comparing all of them doesn't it I mean think with me if we're talking about
  • [00:32] mass then Venus is the most similar to Earth on the other hand if we're talking about temperature then Mars is the most similar one and you can actually see it mathematically but don't worry right here I won't get deep in mathematics or show you complicated equations here instead I want you to draw a straight line and divide this line into 1,000 equal parts where each part represents temperature degrees ranging fromus 500 to positive 500 and then let's place all the planets of the solar system on this line based on their average temperatures Now by looking at them you can already estimate which are the most
  • [01:05] similar Ones based on their proximity however if you want to be precise or you want to compare millions of points just by looking at them wouldn't be enough mathematically though we can determ how close these points are by calculating the actual distance between them on the line This is called the ukian distance and by calculating it computers can then determine how similar two different data are based on their numerical distance on this line but if if we were to decide which planet to colonize next we couldn't rely only one factor we'd have to consider more characteristics or features or let's say dimensions of
  • [01:38] these planets so let's add another dimension here let's talk about meas if we draw another Line crossing our first line now then also divide this line into equal parts and place each planet on that other line based on its mass we would now have a two dimensional coordinate system where one axis represents temperature and the other axis represents Mass now instead of just comparing Planet space on temperature alone we can also consider their Mass when determining similarity if we map these two points in this two dimensional space we now have a vector and if we calculate thean distance between them we
  • [02:12] can still find out how similar they are based on their proximity the ukan distance isn't the only way of calculating how close your points are in the multi-dimensional coordinate system in fact as we start adding many more vectors to it there are other algorithms that can perform better to determine what are the closest vectors to each other or in other words their nearest neighbors another way of doing that is by comparing the angle between them that is traced from the origin to each Vector in the multi-dimensional space this method known as cosign similarity measures how similar true vectors are based on their
  • [02:45] Direction rather than their magnitude all right let's up for a second here what I've seen so far is that Vector databases represent our data vectors or in other words numerical data points and that they determine similarity between these vectors mathematically based on algorithms such as the pleting distance and cosine similarity so a vector database is a database optimized for storing Vector representations of data and efficiently creating that data the distance among vectors is determined by how close vectors are to each other in this
  • [03:17] multi-dimensional Vector space where each Dimension represents a feature of our data all right but where there's a ey fit in this whole picture if we pull back the coordinate system we're talking about the Minal we can see that we're talking about only two dimensions and only nine different data points as humans we're completely capable of determining all of those however when we're talking about hundreds or even thousands of dimensions in millions or billions of data points this task becomes extremely complex and this is where Ai and the famous embedding models we've been hearing so much about come
  • [03:49] into play embedding models are machine learning algorithms that convert unstructured data like text images and sound into meaningful Vector representations they take complex non- numerical data like text images audio or even user behavior and convert them into vectors that capture the most important features unlike simple attributes like mass or temperature Concepts like meaning in text or styling images don't have a straightforward numerical representation AI embedded model solve this by learning how to map similar
  • [04:21] items closer together in a vector space for example in a text search instead of just looking for exact keyword matches and in model can place words like cat and fine closer together even though they're different words this means that when you search for cat we might still retri documents containing fine because their vectors are mathematically close this is how AI powered search engines recommendation systems and chat Bots find relevant information even when the exact words don't match by using embedded models to convert data into high dimensional vectors and then
  • [04:54] storing and searching these vectors efficiently in a vector database AI enables fast and accurate similarity searches whether it's only for text images or any other kind of data and that's why Vector data bases and AI embedding models go hand inand in modern search and recommendation systems to wrap up while Vector search itself is not new and has been applied for decades choosing the right Vector database is crucial especially when working at scale you need a database that can handle high performance similarity searches support billions of vectors and provide real
  • [05:28] time indexing and ing this is where reddis 8 stands out redis has long been known for its blazing fast performance due to its submillisecond operations and with the latest release it has introduced major performance improvements in Vector search benchmarks show that bres can handle billions of vectors with real-time latency making it the fastest Vector database available today with enhancements in latency reduction query engine scaling and realtime Vector insertions red is now provides a production ready solution for large scale Ai and search applications if you're looking for a vector database
  • [06:03] that delivers speed scalability and efficiency red is 8 is the way to go try it out today and experience the fastest Rish ever my name is Rafael Deo thank you for watching and as always stay curious