Transcript: What Is RAG？ Retrieval-Augmented Generation Explained Simply

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

• Video URL: https://www.youtube.com/watch?v=xPMQ2cVbUTI
• Video ID: xPMQ2cVbUTI
• Word Count: 2,666 words

Full Transcript

Hello, I am Brian Samboden and today I will be your guide through the world of RAG or retrieval augmented generation which is the implementation pattern that's enabling us to put the power of large language models or LLMs to practical use. Let's delve into the world of Gen AI LLM's information retrieval and conversational user interfaces. To understand why we need architectures like Rag, we should start at the beginning and learn about the forces that got us here. Starting with the systems at the core of this new computing revolution, language models. Language models have been around in one form or another since the early days of computational linguistics and artificial intelligence. We can think of them as systems designed to parse, understand, generate or manipulate human language. A way to make machines and humans interact in the language of humans. Since the beginning, one of the main goals of a language model is to power conversational chatlike user interfaces where the system is prompted with some natural language text input and the language model generates some natural language text output. One of the first notable examples of such a system was ELISA developed in the mid 1960s which could simulate conversations using pattern matching and substitution techniques. Systems like TLISA worked with hard-coded rules. The language model underneath was nothing more than a careful crafted program that could parse the incoming sentences and construct responses from templates and parts of the input. Fast forward to the 1990s when we saw the rise of statistical methods for machine translation that focused on learning from data rather than relying only on linguistic rules. And by the mid 2000s, several methods, including statistical models and early neural network approaches, were being used to power language models. But it wasn't until the late 2000s and early 2010s, with the advent of deep learning techniques and computational power improvements that neural networks truly began to take center stage in language processing. Advances in computer vision in the mid210s brought deep learning to the forefront with convolutional neural networks or CNN's. And it wasn't long before language models caught up to the deep learning renaissance. Sequencetose sequence models and the emergence of attention mechanisms evolved to improve the ability to understand context in language. 2017 marks a milestone in natural language processing with Google's attention it's all you need a paper introducing the transformer model. Soon after in 2018, the first GPT model generative pre-trained transformer was released by OpenAI. GPT was trained on the web text data set consisting of text extracted from millions of web pages likely amounting to several tens of gigabytes of text data. This training data set included pages from various websites which provided a broad spectrum of language use styles and topics and aimed to capture a wide variety of human language. GPT had a capacity of 117 million parameters. Around this time, the term large language model was popularized to denote the model's capacity, architecture, and training data set size. In the following years, LLMs have observed an exponential growth trend. GPT2 released in 2019 had 1.5 billion parameters. GPT3 one year later had 175 billion parameters. And in 2023, GPT4 is estimated to have 1 trillion parameters. An LLM is trained to calculate the probability of a word following a given sequence of text. All words are given a weight from the known vocabulary and the word with the highest weight it's taken as the best candidate to be the next word to appear in the sequence. LLM capture world knowledge within their parameters demonstrating an advanced ability to mimic language understanding. This capability allows them to be adapted or fine-tuned to achieve state-of-the-art performance on a wide range of language tasks. LLMs are central to the emerging field of generative AI. So what is RAG? RAG stands for retrieval augmented generation for knowledge intensive NLP tasks originated with the seminal paper published in 2020 by Louisal under the sponsorship of Facebook AI research. Let's sparse rag into its constituting components. Starting at the end of the acronym, the part that we have already touched upon, generation. The generation entails assembling a prompt containing the user's question or query, submitting the prompt to the LLM, and receiving the generated answer. So, why is this mode of operation not enough when building knowledge intensive applications? If LLMs capture the world's knowledge in their parameters, shouldn't they always be able to generate a precise answer to a query? LLM responses sometimes contain information grounded on facts mixed with speculative and sometimes create minations. These types of responses have been labeled hallucinations. Instances when the model generates somewhat plausible but incorrect information. LLM hallucinations arise from several known and theorized causes reflecting the complex interplay between model architecture, training data, and generation techniques. Also, the accuracy of an LLM's generated responses heavily depends on the quality of the input data it was trained on. If the training data contains inaccuracies, biases, or is otherwise flawed, the model's output can reflect these. the well-known garbage in garbage out principle. An LLM may also overgeneralize from the training data. Similar to how humans sometimes incorrectly apply a learn pattern to a new context. What appears to be a clear chain of reasoning, it's rooted in nothing more than learned statistical language patterns. The freshness of the knowledge is also a big issue. By the nature of their lengthy and costly static training process, LLM possess knowledge that it's frozen in time. If discoveries, events, or retractions have occur that could affect an answer, the LLM would not know about them. Even if the generated answer is factually correct, there is no traceability to the source. Finding what set of documents, web pages, or papers the LLM truth is grounded on becomes a task for the user. LLMs are trained on a large variety of data. For a given topic, the LLM might have learned a combination of expert information mixed with naive comments, misinformation, and outright lies. To improve answers in a specific domain, for example, a company's inner workings. Context must be provided to guide and ground the LLM's response. And finally, the way a question is worded for an LLM. It's incredibly important. LLMs rely on the provided context to generate relevant responses. If the input prompt is ambiguous, lacks specificity, or the model's input size limits essential contextual information, the model may hallucinate details to fill in the gaps. Crafting an effective prompt has become both science and art to the point that the term prompt engineering it's common place and prompt engineer has at least temporarily become a job title. And this is where the retrieval part of rag comes into the picture. The R in rag, it's part of the larger field of information retrieval, which is the science of finding relevant information in one or more data sources in response to a user query. In the context of rag and conversational AI interfaces, the queries are natural language queries. the relevant data. It's typically in the form of unstructured data such as PDFs, word documents, web pages on your company's internet, industry journals behind a gated wall, or company confidential information. This sounds simple enough, right? But it is not. Accurate and effective information retrieval is a very important and complex component of a rag implementation. So, let's dig deeper into the common implementation of the retrieval component. A very popular implementation uses a vector database. A vector database is a system that enables the storage, management, and querying of unstructured data. The first step to using a vector database in a rack system is to generate vector representations of the data in our knowledge base. Vectorization is the secret sauce that allow us to standardize, store, and operate on unstructured data. Previously, we learned that language models have the power to guess the next word or token based on some input text. Another feature that LLMs in many machine learning models possess, it's the ability to generate a numeric representation of the input data. In machine learning, we call this numeric representation a vector embedding. These vector embeddings compress and encode the semantic information of the input and place them in a latent space, a multi-dimensional space where a data point's location reflects the meaning of the content. When these embeddings are stored in a capable vector database, we are provided with the ability to efficiently search for related content given an input example. We call this kind of retrieval dense retrieval. And in the rack case, it's a form of semantic search. Now that we know the potential pitfalls of throwing an LLM into a knowledge inensive application and we had a high-level overview of how modern dance retrieval systems can be implemented, let's connect the two halves and tackle the A in rag augmentation. The first step in a typical RAG interaction is translating the user's natural language query into a form the system can efficiently process. We must first vectorize the user's query to search for relevant data in a vector database. The vectorzed input query it's used to find semantically similar vectors. The k nearest vectors identified in the vector database correspond to specific documents or passages most relevant to the user's query. These documents are then retrieved and the relevant content which could be entire documents, paragraphs or sentences. It's extracted for each of the documents. The extracted content and possibly the original query are used to construct a prompt for the LLM. This prompt is carefully designed to encapsulate both the query and the context provided by the retrieve documents, guiding the LLM to produce a relevant and informed response. The LLM generates a response that may undergo further refinement to ensure its relevance and accuracy or to meet specific criteria set by the application. This could sometimes involve additional processing steps such as reranking generated answers, post-processing for grammar and coherence, or even human in the loop review. Finally, the generated and refined answer is presented to the user as the response to their query. Let's focus on the setup of the information retrieval portion of a rack system. More specifically, the indexing process. In order to efficiently perform dense retrieval based on our user queries, we need to populate our data store with information that will provide the accurate up-to-date context to our queries. As we previously learned, this data needs to be vectorized and stored in a vector database. Creating a proper data ingestion pipeline for rag. It's a complex process that so far we have abstracted and simplified. Let's dive deeper into this aspect of rag. Let's focus on the typical documents found in an enterprise environment. These documents can vary widely from structured reports to informal chat transcripts each of variable length and with a possible combination of topics. Some documents might be well structured, maybe following a template. Others might be completely organic chat transcripts from meetings. Some might be emails. Think about the dynamics of a corporate meeting. For example, we all have been in a meeting where the topic and focus of the conversation drifts. The transcript of such a meeting might contain the specific answer to a user's query, but it is now buried in a semantic soup of topics representing the whole document. That transcript if vectorized will place that document in the latent space somewhere between all those topics potentially eliminating it as a match for the user's query. The level of complexity of storing such a document in a vector database depends on how precise and efficient you want the dense retrieval process to be. There are a couple of immediate issues at hand. The first one is a potential size of the content. LLMs have a fixed context window length which puts a limit to how much contextual information we can pass as part of the prompt. The second and more important issue when it comes to the relevancy of the information retrieved is the concept of context drift. To deal with both of these issues, one of the simplest strategies is to break the document into chunks and vectorize the chunks individually. The component in charge of generating appropriate chunks is aptly named the chunker. These chunks are more likely to capture a focused view of the document allowing for a finer grain match against user queries. How we select the chunk size is very important. If chunks are too small, certain questions cannot be answered. If the chunks are too large, then the answers will include a level of noise. The vector database, the vector for a chunk gets stored along some metadata that points back to the document of origin and the location of the chunk in the document. There are several chunking techniques in use including fixed length chunking, semantic chunking, and querybased chunking. Each offering unique benefits in processing and retrieving information. Now, let's focus on the querying process and break down the commonly found components and their functions in a rag implementation. Rag is an emerging architecture and advancements are being made every day. So, today we'll cover only the most commonly found components in current rag implementations. The querying process starts with the rag system receiving a user's query. Then a component called the rewriter modifies the original query to improve retrieval results, potentially by expanding it with synonyms or rephrasing it for clarity or even breaking it into multiple subqueries. The retriever component fetches relevant documents or chunks from the dense retrieval system or other sources in order to provide context. The retriever might contain or employ a router to manage multiple data sources. A reranker can be used to further assess the relevance of the retrieve documents, perform compression, and better align the context with the user query. The consolidator aggregates and synthesizes information from the top documents, possibly dduplicating or summarizing information to prepare for final processing. At this state, the prompt has been grounded with context and instructions to guide the LLM generation process. Finally, the reader prompts the LLM with the engineered prompt and interprets the LLM's response to create a coherent and contextually relevant answer for the user. In some RAG implementations, especially those dealing with complex or multi-turn interactions, a contextualizer might be necessary to integrate the current query with the context of a long conversation. With the understanding gained, let's run through a simulated Ragnar action in a fictitious cinema expert rack power chatbot. Let's say that we want to ask our chatbot whether any of the characters the actor Pedro Pascal ever played had an animal nickname, but our user's wording of the query it's less than perfect. The rack query rewriter might rephrase the query to make it less ambiguous, including keywords like movie, film, TV show, and ro to guide the search process. The retriever component searches our context store using the constructed query and returns a few articles. Imagine the top match is an article about the Mandalorian TV show and the second one it's about the 2019 movie Triple Frontier. The re-ranker then evaluates the relevance of the retrieve articles based on the query specific focus and in our case it will prioritize a second article which directly addresses the query. The consolidator then takes the top rank articles provided by the re-ranker and creates a focus summary by extracting the key facts and presenting them in a manner that aligns with the query. Finally, the reader component crafts the prompt that will be submitted to the LLM. Upon receiving a response, it will perform any reformatting, sanitizing or enhancements required by the application so that we can return a proper response to the user. In closing, retrieval augmented generation or rag represents a sophisticated pipeline of processes that elevate the capabilities of large language models by infusing them with up-to-date, curated, and highly relevant contextual information from external sources.

Timestamped Segments

• [00:00] Hello, I am Brian Samboden and today I will be your guide through the world of RAG or retrieval augmented generation which is the implementation pattern that's enabling us to put the power of large language models or LLMs to practical use. Let's delve into the world of Gen AI LLM's information retrieval and conversational user interfaces. To understand why we need architectures like Rag, we should start at the beginning and learn about the forces that got us here. Starting with the systems at the core of this new computing revolution, language models.
• [00:34] Language models have been around in one form or another since the early days of computational linguistics and artificial intelligence. We can think of them as systems designed to parse, understand, generate or manipulate human language. A way to make machines and humans interact in the language of humans. Since the beginning, one of the main goals of a language model is to power conversational chatlike user interfaces where the system is prompted with some natural language text input and the language model generates some natural language text output. One of the first
• [01:07] notable examples of such a system was ELISA developed in the mid 1960s which could simulate conversations using pattern matching and substitution techniques. Systems like TLISA worked with hard-coded rules. The language model underneath was nothing more than a careful crafted program that could parse the incoming sentences and construct responses from templates and parts of the input. Fast forward to the 1990s when we saw the rise of statistical methods for machine translation that focused on learning from data rather
• [01:40] than relying only on linguistic rules. And by the mid 2000s, several methods, including statistical models and early neural network approaches, were being used to power language models. But it wasn't until the late 2000s and early 2010s, with the advent of deep learning techniques and computational power improvements that neural networks truly began to take center stage in language processing. Advances in computer vision in the mid210s brought deep learning to the forefront with convolutional neural networks or CNN's. And it wasn't long before language models caught up to the
• [02:15] deep learning renaissance. Sequencetose sequence models and the emergence of attention mechanisms evolved to improve the ability to understand context in language. 2017 marks a milestone in natural language processing with Google's attention it's all you need a paper introducing the transformer model.
• [02:35] Soon after in 2018, the first GPT model generative pre-trained transformer was released by OpenAI. GPT was trained on the web text data set consisting of text extracted from millions of web pages likely amounting to several tens of gigabytes of text data. This training data set included pages from various websites which provided a broad spectrum of language use styles and topics and aimed to capture a wide variety of human language. GPT had a capacity of 117 million parameters. Around this time,
• [03:10] the term large language model was popularized to denote the model's capacity, architecture, and training data set size. In the following years, LLMs have observed an exponential growth trend. GPT2 released in 2019 had 1.5 billion parameters. GPT3 one year later had 175 billion parameters. And in 2023, GPT4 is estimated to have 1 trillion parameters. An LLM is trained to calculate the probability of a word
• [03:44] following a given sequence of text. All words are given a weight from the known vocabulary and the word with the highest weight it's taken as the best candidate to be the next word to appear in the sequence. LLM capture world knowledge within their parameters demonstrating an advanced ability to mimic language understanding. This capability allows them to be adapted or fine-tuned to achieve state-of-the-art performance on a wide range of language tasks. LLMs are central to the emerging field of generative AI. So what is RAG? RAG
• [04:19] stands for retrieval augmented generation for knowledge intensive NLP tasks originated with the seminal paper published in 2020 by Louisal under the sponsorship of Facebook AI research. Let's sparse rag into its constituting components. Starting at the end of the acronym, the part that we have already touched upon, generation.
• [04:41] The generation entails assembling a prompt containing the user's question or query, submitting the prompt to the LLM, and receiving the generated answer. So, why is this mode of operation not enough when building knowledge intensive applications? If LLMs capture the world's knowledge in their parameters, shouldn't they always be able to generate a precise answer to a query?
• [05:04] LLM responses sometimes contain information grounded on facts mixed with speculative and sometimes create minations. These types of responses have been labeled hallucinations. Instances when the model generates somewhat plausible but incorrect information. LLM hallucinations arise from several known and theorized causes reflecting the complex interplay between model architecture, training data, and generation techniques. Also, the accuracy of an LLM's generated responses heavily depends on the quality of the
• [05:39] input data it was trained on. If the training data contains inaccuracies, biases, or is otherwise flawed, the model's output can reflect these. the well-known garbage in garbage out principle. An LLM may also overgeneralize from the training data.
• [05:56] Similar to how humans sometimes incorrectly apply a learn pattern to a new context. What appears to be a clear chain of reasoning, it's rooted in nothing more than learned statistical language patterns. The freshness of the knowledge is also a big issue. By the nature of their lengthy and costly static training process, LLM possess knowledge that it's frozen in time. If discoveries, events, or retractions have occur that could affect an answer, the LLM would not know about them. Even if the generated answer is factually
• [06:30] correct, there is no traceability to the source. Finding what set of documents, web pages, or papers the LLM truth is grounded on becomes a task for the user. LLMs are trained on a large variety of data. For a given topic, the LLM might have learned a combination of expert information mixed with naive comments, misinformation, and outright lies. To improve answers in a specific domain, for example, a company's inner workings.
• [06:59] Context must be provided to guide and ground the LLM's response. And finally, the way a question is worded for an LLM. It's incredibly important. LLMs rely on the provided context to generate relevant responses. If the input prompt is ambiguous, lacks specificity, or the model's input size limits essential contextual information, the model may hallucinate details to fill in the gaps.
• [07:24] Crafting an effective prompt has become both science and art to the point that the term prompt engineering it's common place and prompt engineer has at least temporarily become a job title. And this is where the retrieval part of rag comes into the picture. The R in rag, it's part of the larger field of information retrieval, which is the science of finding relevant information in one or more data sources in response to a user query. In the context of rag and conversational AI interfaces, the
• [07:56] queries are natural language queries. the relevant data. It's typically in the form of unstructured data such as PDFs, word documents, web pages on your company's internet, industry journals behind a gated wall, or company confidential information. This sounds simple enough, right? But it is not.
• [08:15] Accurate and effective information retrieval is a very important and complex component of a rag implementation. So, let's dig deeper into the common implementation of the retrieval component. A very popular implementation uses a vector database. A vector database is a system that enables the storage, management, and querying of unstructured data. The first step to using a vector database in a rack system is to generate vector representations of the data in our knowledge base.
• [08:44] Vectorization is the secret sauce that allow us to standardize, store, and operate on unstructured data. Previously, we learned that language models have the power to guess the next word or token based on some input text. Another feature that LLMs in many machine learning models possess, it's the ability to generate a numeric representation of the input data. In machine learning, we call this numeric representation a vector embedding. These vector embeddings compress and encode the semantic information of the input and place them in a latent space, a
• [09:18] multi-dimensional space where a data point's location reflects the meaning of the content. When these embeddings are stored in a capable vector database, we are provided with the ability to efficiently search for related content given an input example. We call this kind of retrieval dense retrieval. And in the rack case, it's a form of semantic search. Now that we know the potential pitfalls of throwing an LLM into a knowledge inensive application and we had a high-level overview of how modern dance retrieval systems can be implemented, let's connect the two
• [09:50] halves and tackle the A in rag augmentation. The first step in a typical RAG interaction is translating the user's natural language query into a form the system can efficiently process. We must first vectorize the user's query to search for relevant data in a vector database. The vectorzed input query it's used to find semantically similar vectors. The k nearest vectors identified in the vector database correspond to specific documents or passages most relevant to the user's query. These documents are then retrieved and the relevant content which
• [10:23] could be entire documents, paragraphs or sentences. It's extracted for each of the documents. The extracted content and possibly the original query are used to construct a prompt for the LLM. This prompt is carefully designed to encapsulate both the query and the context provided by the retrieve documents, guiding the LLM to produce a relevant and informed response. The LLM generates a response that may undergo further refinement to ensure its relevance and accuracy or to meet specific criteria set by the application. This could sometimes
• [10:55] involve additional processing steps such as reranking generated answers, post-processing for grammar and coherence, or even human in the loop review. Finally, the generated and refined answer is presented to the user as the response to their query. Let's focus on the setup of the information retrieval portion of a rack system. More specifically, the indexing process. In order to efficiently perform dense retrieval based on our user queries, we need to populate our data store with information that will provide the accurate up-to-date context to our queries. As we previously learned, this
• [11:29] data needs to be vectorized and stored in a vector database. Creating a proper data ingestion pipeline for rag. It's a complex process that so far we have abstracted and simplified. Let's dive deeper into this aspect of rag. Let's focus on the typical documents found in an enterprise environment. These documents can vary widely from structured reports to informal chat transcripts each of variable length and with a possible combination of topics.
• [11:56] Some documents might be well structured, maybe following a template. Others might be completely organic chat transcripts from meetings. Some might be emails. Think about the dynamics of a corporate meeting. For example, we all have been in a meeting where the topic and focus of the conversation drifts. The transcript of such a meeting might contain the specific answer to a user's query, but it is now buried in a semantic soup of topics representing the whole document. That transcript if vectorized will place that document in the latent space somewhere between all those topics potentially eliminating it as a match for the user's query. The
• [12:31] level of complexity of storing such a document in a vector database depends on how precise and efficient you want the dense retrieval process to be. There are a couple of immediate issues at hand. The first one is a potential size of the content. LLMs have a fixed context window length which puts a limit to how much contextual information we can pass as part of the prompt. The second and more important issue when it comes to the relevancy of the information retrieved is the concept of context drift. To deal with both of these issues, one of the simplest strategies
• [13:03] is to break the document into chunks and vectorize the chunks individually. The component in charge of generating appropriate chunks is aptly named the chunker. These chunks are more likely to capture a focused view of the document allowing for a finer grain match against user queries. How we select the chunk size is very important. If chunks are too small, certain questions cannot be answered. If the chunks are too large, then the answers will include a level of noise. The vector database, the vector for a chunk gets stored along some
• [13:35] metadata that points back to the document of origin and the location of the chunk in the document. There are several chunking techniques in use including fixed length chunking, semantic chunking, and querybased chunking. Each offering unique benefits in processing and retrieving information. Now, let's focus on the querying process and break down the commonly found components and their functions in a rag implementation. Rag is an emerging architecture and advancements are being made every day.
• [14:02] So, today we'll cover only the most commonly found components in current rag implementations. The querying process starts with the rag system receiving a user's query. Then a component called the rewriter modifies the original query to improve retrieval results, potentially by expanding it with synonyms or rephrasing it for clarity or even breaking it into multiple subqueries. The retriever component fetches relevant documents or chunks from the dense retrieval system or other sources in order to provide context. The retriever might contain or employ a router to manage multiple data sources.
• [14:37] A reranker can be used to further assess the relevance of the retrieve documents, perform compression, and better align the context with the user query. The consolidator aggregates and synthesizes information from the top documents, possibly dduplicating or summarizing information to prepare for final processing. At this state, the prompt has been grounded with context and instructions to guide the LLM generation process. Finally, the reader prompts the LLM with the engineered prompt and interprets the LLM's response to create a coherent and contextually relevant
• [15:10] answer for the user. In some RAG implementations, especially those dealing with complex or multi-turn interactions, a contextualizer might be necessary to integrate the current query with the context of a long conversation. With the understanding gained, let's run through a simulated Ragnar action in a fictitious cinema expert rack power chatbot. Let's say that we want to ask our chatbot whether any of the characters the actor Pedro Pascal ever played had an animal nickname, but our user's wording of the query it's less than perfect. The rack query rewriter might rephrase the query to make it less
• [15:44] ambiguous, including keywords like movie, film, TV show, and ro to guide the search process. The retriever component searches our context store using the constructed query and returns a few articles. Imagine the top match is an article about the Mandalorian TV show and the second one it's about the 2019 movie Triple Frontier. The re-ranker then evaluates the relevance of the retrieve articles based on the query specific focus and in our case it will prioritize a second article which directly addresses the query. The consolidator then takes the top rank
• [16:16] articles provided by the re-ranker and creates a focus summary by extracting the key facts and presenting them in a manner that aligns with the query. Finally, the reader component crafts the prompt that will be submitted to the LLM. Upon receiving a response, it will perform any reformatting, sanitizing or enhancements required by the application so that we can return a proper response to the user. In closing, retrieval augmented generation or rag represents a sophisticated pipeline of processes that elevate the capabilities of large language models by infusing them with up-to-date, curated, and highly relevant
• [16:50] contextual information from external sources.