Version: v1.0 (Raw Transcripts)

Transcript: Building the future Architecting AI Agents with AWS, LlamaIndex and Redis

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

Video URL: https://www.youtube.com/watch?v=SFWroqAbBM4
Video ID: SFWroqAbBM4
Word Count: 11,051 words

Full Transcript

all right uh thanks everybody for coming here today to this webinar I'm super excited to be here talking about how to architect AI agents with yma index and Technologies from AWS such as Amazon Titan and Amazon badrock uh this is a huge topic these days and probably you you are aware of this but we're going to do like this presentation that's going to be broken down into three parts we're going to talk about rag as an architect we're going to talk about Amazon badrock and its foundation mods this support it has for it and then we're also going to start talking about the yemma index so um this is the agenda for today right uh and I'm going to let the the presenters like introduce themselves in a minute but uh after this we're going to start talking about rag agents and then Red's AI for agents and how R fits on this entire architecture and obviously we're going to talk about Amazon badrock and also orchestration with yma index with me today we have and prad from AWS which is an amazing person and knows tons of AWS specifically Amazon badrock and he's going to be talking about this in a minute so prad if you want to look introduce yourself for a minute thank you thank you Ricardo uh pleasure to be here with everyone uh I'm Anthony prad deage senior partner sa here at AWS uh been with the company for close to nine years uh I still am very much a recovering data engineer uh which used to be uh the first part of my life at Amazon uh at the end of the day uh the important Essence is uh garbage in garbage out no matter how sophisticated your llm is no matter how great your application is if you don't have the right data set and the right the way you structure your data is actually going to power your gen applications so with that being said I'll hand it over to my good friend and Pier Ricardo take it away thank you pad um and also we have Lori which is the VP of developer relations of yma enx so Ari hi everybody yeah uh so I run developer relations at llama index you're going to hear more about what llama index is and why you should use it uh in the course of this presentation um as background on me uh a long time ago I was a co-founder of npm Inc so uh you may remember me from when I used to talk incessantly about javascripts uh now I talk incessantly about AI but the Common Thread is that I talk incessantly that's super cool I I remember that Larry so I remember that uh thank you everybody for coming here today uh and also we have Ricardo fer that happens to be this person here so I am the lead developer Advocate at rad is and um I've just joined a company I'm actually came in front aw so prad and I we were like co-workers uh a month ago actually that's quite new and uh I'm very excited with all the things that Rus has to offer and integration with the other technology so I'm going to start off like uh I would like to put everybody on the same page where about the understanding about rag right uh rag is a common term these days like it it's definitely one of the most like Hot Topics in the technology industry but uh it is always important for us to kind of uh make sure that the thms and the understanding of it is the same right whether if you are an expert whether you were uh hearing about rag for the first time so the building block of rag architectures is what we call edings right so when it comes to data right uh this is EXA the raw material that rag architect are going to use to do what it's supposed to do and Ricardo what is rag in the end of the day rag is the ability for you to augment uh request and answer scenario with facts right so the way you create and store those facts in the form of embeding right the retrieval part is actually the process of when you actually are responding to a question that has been sent to a llm or large language model or a foundation model in the jargon ad Bedrock Amazon badrock is it's going to make sure that the process of the answering is going to be as accurate as possible right and we call all of this process end to endend the amen and generation right so this is what uh ragged is all about um if we look at the typical architectur for rag uh architector this is usually what we're going to see out there whether you were imployment with uh radis or not whether you implment with uh Amazon Bedrock or some specific llms right you're always going to see what we call The Usual Suspects right so the knowledge basis one of them right the knowledge base is where you're going to set your context and those facts that were mentioned before right and also you're are going to have the the the Persona of the user quer is exactly with the the questions that your user or customers are asking to the LM component right and then the component that's going to tie the llm to the knowledge base and the embeding service is going to be your orchestration framework to which in this case we're going to be using yma index and there's a huge reason for you to start developing your agentic rag with yma index so Lor is going to explain a little bit more later but usually this is what you are going to see um uh when a typical rag implementation all right you're also going to use this component of the embeding service which is the service that's going to transform data into a set of V or embeddings right that's going to become your facts right and then here is what we are calling the evolution of a rag architectur evolution in a sense of whether you were implementing uh regardless of the technology that you are using to implement rag you are going to face some problems let's call them industry customer problems that are uh unipr presentent everywhere right one of them is for example the process of the need for event skew right for example all of this in the end of today isri is a descripted system that in creates a lot of network interaction right so let's say there's a your website just came live and it become a white success so a lot of customers are using the AI agents to ask questions about whatever the their business needs are and then you have to have some component that's going to throttle those interaction so you don't overload your backend systems right so this is something that we are fixing and soing with radis these days also there's the component of a santic cach right santic cach is the ability for you to surprise surprise use radis as a cache which is the most popular use case for radis but with a huge difference right and here comes the part we call santic cache it's a specialized type of cache that knows how to interpret a question and make sure that even though it's not the same wordss that are being asked but if some similar question has use the same answer it retrieves from the cash instead of going for it entire to the llm and the huge value for this because you're going to save a lot of a token interaction for the llm service and also the identic memory so you can create a more comprehensive context every time you are going to deal with questions so everything that has been discussed before is considered in the upcoming and forthcoming questions right uh and lastly we are also have this component called santic router which is uh it is a component where you can use for when you don't want to rely only one specific llm that specialized in aing you can create different llms or use different llms that has been training into specific domain models or domain expertises and then you can route accordingly depending of the context that you are dealing with so all of this create a more complete and more human friendly version of architector right and of course uh the usual using R is as a knowledge base to store the facts um another Trend that is probably kind of a very it's not so common as ragged itself but it's starting to kind of a uh keep up these days is the concept of agent supervising so agent supervising is this ability for you to create a much larger view of one specific agent or multiple agents so you create a more human friendly version of your the the human interactions that your customers are going to use with the agent AI so again radus is going to be using as a persistent layer and then there's going to be some buil-in integration with the agent supervising here that's can be also Implement with Y index um and lastly red is for AI which is kind of the the the component that we're using for this new generation of uh rag architecture is it has this foundation in this component that we call rag Square engine right this is something that we' build on top of radus for the radus that you know and love and then we have this kind of a radis quare engine that it it is was originally created to allow radus to perform Json document search on it but when we start adding support for vectorization it also the same engine allows this vectorization searches and centic cache and all the components that we've discussed so far to also work this is the foundation that actually gives birth to the flexible development that we have available including integration with yma index so there is some very interesting partnership that we did with um yma index to integrate the Frameworks and the SDK and the code level and also the components that we are calling then if you look the architecture from the architectur perspective and last that a code perspective those are the components that we support that are backed by the qu engines and all of this is what we are using to sustain the use cases that you can implment with this architecture right so this is the first step for you to understand the this new type of approach for us to implement red architectures but one very important concept that you have to master to understand this is exactly the how the llms are implemented and how this integrate with the radish technology and for this I am going to actually invite my friend prad that's going to give you an overview about how Amazon Bedrock fits into this thank you Ricardo I would still need your help to share the slides absolutely no worries my friend it's all it's almost we're almost getting there I swear uh but thank you um let's start with the previous slide so I just want to spend a moment uh you know help to help you guys understand uh why we're having this conversation today if you think about it in 2023 uh a lot of our customers were um still doing poc's right and a bunch of questions kept uh coming up hey um how do we go from po to production what am I supposed to do uh how am I supposed to prioritize my projects or um how can I lower my costs um you know stuff like that while Ricardo is trying to fix uh the slideshow but uh long story short customers have realized that uh we were actually able to build production ready systems and the reason we were able to do that is because uh we leveraged Amazon Redrock in combination with isv Solutions like redis right um Ricardo I think you can just share the slides I think there's a screen recording delay when you share slides like if you go to presenter viw or just present slideshow uh there might be yeah it's fine but hey what's without a imperfect demo that's how it is but having said that um in 2024 we've achieved massive success uh with our customers uh we've seen systems in production uh last year at Rin vent there was a presentation by NASDAQ would actually involved a fraud detection system that was leveraging database software solution uh and we were able to take that to production um and we also noticed uh for example uh uh doctors were able to diagnose the uh treatment that was supposed to be given to a patient based on historical data so we created production grade ready systems uh in 2024 and now we're taking it to the next step with that being said um I want to cover a few topics today on uh Amazon bedrock and what it is uh and how we've got here right uh Ricardo I think you're sharing this wrong slide maybe you want to go back a few slides um go back and for yeah uh we can see the Bedrock models slide now no we gotta go the other way um in Reverse I think you're moving forward Ricardo okay um yes let's start from here um what are the needs what is the service itself uh I'm going to talk about some of the key features and we're going to touch upon uh some of the customers the only slide that I want you guys to maybe take a photo of is the last slide where we'll share a QR code on everything that I've just talked about it's a Hands-On Workshop uh it's a 4-Hour Workshop that lets you actually get started on everything we've talked about uh let's move on to the next slide please yes okay the top three questions that we come across from all our customers today hey Anthony there's so many models out there what model do I use right where do I get started second one how quickly can I go to production now for this evolution of gen U during my days as a data engineer I remember when we had spent six months to productionize a click probability model this model all it was supposed to do was predict if this ad on amazon.com was going to get a click or not over the next seven days sounds like a simple ml problem sure but then getting data is uh is is is a problem right so that's where we took six months to go to production but today we don't have six months time last but not the least Anthony is my data secure is my data only accessible to me these are the top three questions that every customer had next slide please and to address all these questions is why we launched Amazon Bedrock what Amazon Bedrock does for you is think of it as a platform that can host multiple llms and Foundation models uh some of which Ricardo just talked about but think of it as a single API call you are able to Leverage the power of all these llms you're able to deploy AI agents you're able to deploy the entire rag flow that Ricardo just touched upon right right from retrieval augmentation and finally generating your output everything can be automated using bedrock in the console itself right and last but not the least on security I don't know how many of you guys are familiar with private link but AWS private link is a service that ensures your data coming from outside the internet into your a us bpc is absolutely secure and Bedrock is private link enabled by default so you don't have to worry about any data security consents let's move on Ricardo all right just to summarize with bedrock you can pick a bunch of models llms and Foundation models you can play with you can customize your model as well so let's say you say hey Anthony I want to tweak a few parameters to adhere to my use case yes you can L tweak that and customize that and guess what the copy is only in your control unless you specify permissions to your I am roles and policies the other people in the room don't get access to your model the next one is you can integrate seamlessly with other native AWS services like red shift for example if you have a data ering request you can store data in red shift and then you can feed it into Amazon Redrock you are not going to be charged for additional access to the service Bedrock will only charge you for the number of tokens you consume per minute to run your llm last but not the least we talked about the security and governance if your data lies in Europe we have all the local governing policies aduring to our standards and we are available in most regions across the world I would have a slide talking about that as well next please all right so this slide is uh very very outdated uh clearly outdated and it will be outdated next week as well as uh those of you who were following reinvent we launched something known as Amazon Nova which was our very own uh Foundation model as well it was based on or inspired from the work we did with uh amazon.com shopping experience if you guys have tried out Rufus uh and then Alexa of course the conversational style AI Hardware that has existed for years where you could place orders or you know even requests for things that you had purchased in the past um so Amazon Nova I don't I know it's not mention on the slide but I will share the link in the chat in a moment when I get through but the point of this slide is you have a ton of options we work with some of the top grade llm providers like anthropics of the world coheres meta Mr Lei and stability a all these guys have their LMS offered to us through the Bedrock console so that's the key takeaway of the slide next one please and I want to focus a little bit on Claude in particular right because uh anthropics cloud is actually uh one of our best performing models in terms of the sophistication that brings to the table in terms of the cost benefit it has in terms of the ability to scale as well so and the option I want to give to the customers here is we have something for everyone let's say you want to start small you don't have a very complex use case you don't require that level of intelligence and sophistication but you're focused on speed and cost alone then better off with claw 3 Haiku because it's the fastest performance and the low slowest cost but you get a compromise on the ability of the LM itself if you want something somewhere you know in between all these worlds the second best performing one is claw three office which gives you intelligence which gives you the speed and it also gives you the cost benefits but our most powerful one till date is Claude 3.5 son so if you really want to build a sophisticated intelligent application we recommend going there um and again you pay the cost but we have cost mechanisms to optimize your workload as well I'll talk about that in a little bit let's move on um okay again this is number one of the most s after features I'm glad we launched this a few months ago and the reason we did this is um you know you have a ton of models out there right so let's say you say Anthony how do I know which model performs better which model performs better than the other one well guess what Bedrock introduced something known as a model evaluation report and we have two options here so if you follow my theme so far we are going to have humans we're not going to replace humans we're going to enable and help humans build faster better geni applications so we have two options here automatic or human evaluation if you pick automatic you will let Bedrock decide based on the metrics and the kpi I know the slide might be a little convoluted but there's a ton of metrics and there's custom metrics you can Define which will specify what the performance means for the impact of the end output or if you pick human evaluation method you can specify your email Alias or your team's email Alias and you get a report saying hey user here's how the performance metrics looks model B performs better than Model A if you give a thumbs up it kicks back the workflow in your Bedrock console it's that easy next one please um I want to spend a little bit time on this one right so going back to C conversation on rag right like how do you structure data how do you store your embeddings right once you figured out all that right the next question is how do I improve the accuracy of my model response right because obviously if you're a customer that cares about the data itself the quality of data itself there's a combination of techniques we want to introduce to you to enable you to provide high quality data results for your end users prompt engineering as Ricardo said you it's a simple standard well-known technique you provide prompts the LM gives you your response you you narrow down the search results and then you get a better response right then the other one is the rag workflow I'm not going to spend time on rag because Ricardo talked about it but then the other two approaches I want to talk about is fine-tuning and pre-training now if you're a data scientist and if you know what good looks like from your model if you know the false positives if you know the false negatives you can Define that within your S3 bucket for example and then you let the model skip records for example that are not in line with what you're expecting and then you can only consume the data that is required for your model so that's one way of fine tuning pre-training is for people who don't know what their data looks like let's say you you come in and you say Anthony I got this raw data dump I don't know what to do with it I haven't checked it out then you can opt for continued pre-training because then what you do is you'll allow Bedrock to go through the data and give you what is in the bucket his3 bucket itself so it'll break it down into buckets and tell you this is exactly what your data looks like right and then it can also generate mock data sets so if you're a data scientist you want to productionize a model you want to go from po to production you need to test your end results you need to test the use cases you need to test every possible Edge case we can generate mock data sets with contined pre-training that's one of the feature I would highly recommend you guys to check it out and if you want to get a perfect result I would say play around with the combination of all these four techniques again depends on the cost you want to incure it depends the time you want to consume let's move on all right this is something that we introduced very recently which is known as bedrocks prom flow so if uh you know for those who are not familiar with this it's like a drag and drop experience right so you have a bunch of icons there and let's say you built your data source which is starting from a legacy system or it's an S3 and then you pick a combination of models that you want to use you can do this through the console itself so you can select options through a drop and drag uh Dropbox bar that's how easy it is so that's why we created uh promp flow and it also helps with collaboration within the team as well right so think of it as a playground uh for people to test it out you can move on uh and this is how it looks like right it's that easy drag and drop interface versioning is also enabled so if you make changes to one workflow you can revert back to the previous version and really has changed the lives of developers uh and it's made it very very uh user friendly next please all right uh here I want to spend a little bit of time right Bedrock has something within the console itself a feature an optional feature known as knowledge base and I want to take a step back to help you guys understand what knowledge base means so let's say um you know a pharmaceutical use case where the doctor is trying to figure out what is the right patient diagnosis for patient XYZ based on five years worth of clinical trial data now question how does your llm know what your patient went through no idea it's domain knowledge and that domain knowledge is going to be input to your knowledge base so what knowledge base is actually doing is is providing domain knowledge to llm in educating the llm about what exactly we're looking for in this patient based on the clinical trials so it's as good as an input to your uh Bedrock uh you know llm and this is uh something that we're working with partner like redis redis is one of the few isvs who is actually present as a vector database offering on the Bedrock console we are slowly trying to incorporate redis as a knowledgebase input where the reddis database itself can be an input to knowledge base so then we're trying to eliminate the need for moving data outside your database exporting to S3 and so on so we're working on that with Red's team as we go on but again in this use case the user files a query looks for data within the domain that is fed as an input to your knowledge base you like the data or not you're looking for more details you provide the prompts and then in this example we've selected a combination of these four models not just one and then eventually you get your final answer and that's how knowledge bases work let's move on all right there we are uh red is Enterprise cloud is uh actually available on uh your Bedrock console we have been available for over a year now um we were launch launched at last reinvent we've optimized and launched even further Innovations with red Spector database there's something known as llm rag evaluation that we launched recently at ROM rent um so if you're a reddish customer you have your data in redish uh reddish Cloud as a vectors then you can definitely check it out uh I would highly recommend you to explore this offering on the console next slide please all right we already talked about fine-tuning and pre-training I would highly recommend you guys to go through that with the Bedrock console the step-by-step tutorial that I'll share at the end will also provide you steps to do so let's move on Ricardo all right I do want to say that since I spent a lot of time on cloud 3 uh we have fine-tuning specifically for CLA 3 models and Amazon Bedrock is the only fully managed service that provides the ability to fine-tune uh Cloud models so that's something that you guys if you're working on I would highly recommend checking it out because the feature exists for you out of the box next slide please all right the conversation of the day right we talked about AI agents and you know how does Bedrock fit into this piece right now no solution can be built just by one service it is a combination of services like Ricardo said you have to use reddis as a vector store for the reasons mentioned and then what next so in this way you can Define bedrock agents to execute on your behalf so what does that mean every time there is an API call made to look for a specific service you don't have to write separate code facets to get that API to work as expected and pull the data for example or or send it back you can orchestrate that to the Bedrock agents it's as simple as providing basic instructions through English select your foundation model if you remember the previous slide for four llms you've selected select all of them provide those instructions in the Bedrock console the next one what am I doing break it down to steps look for this API if I'm checking for is this asset in stock or not I'm just giving a retail example let's say you're a retail company you're trying to find out if this product exists or is in inventory or not provide that orchestration piece in the instructions the next one select the right data sources where is my model going to look look for this data is it in an S3 bucket is it in the Legacy store select that leement data source and last but not the least as for those of you are familiar with AWS reference icons that's a Lambda function right so we given an example here is let's say you have all the things in place and you want your Lambda function to specifically execute on this piece of code with this API call Provide those instructions as well this entire thing can be automated so it's a onetime human intervention we're not taking the human out of the loop this second thing is we also have a Chain of Thought tracing enabled which allows you to diagnose and troubleshoot the issues at the lir for those of you you are familiar with cloud formation templates we're not going to give irrelevant error codes I was one of them who kept complaining I'm sure Ricardo can attest to that some of the error codes for gibberish didn't really provide you exact details but the Chain of Thought tracing actually gives you specifically where the problem is next slide please all right um for those of you are familiar with the ec2 consumption pricing similarly we have on demand for those of you who are not aware of what workload it is how many tokens you're going to consume we recommend starting with on demand if you are familiar with your workloads and if you're familiar with the number of tokens you're going to consume per minute I would recommend uh provisioning with ahead of Advance because you don't want your application to ever go down and then we also introduced batch inference recently because there are companies who actually want to process the data in chunks or micro batches that is one way to do it uh via Redrock next slide please all right we also introduced cross region inferencing what that means is if you're a European client you're running your llms within data reiding in Europe we probably don't have enough resources to run your models then we're going to go to another region fetch the resources temporarily and help you run your workflows and that is why we introduced cross region inferencing we don't have extra charges for you on that account let's move on all right as we continue to grow uh these are the regions that we exist in today we also exist in govcloud if you're a gocloud customer we highly recommend checking that page we also are in deance to all the Hippa and gdpr compliance across the globe and we're contining to expand as we go on next slide please all right we're almost there to the end uh I know I've had to rush through the glitches we've had in the technical presentation but here we are Adidas uh they built uh you know uh a Bedrock knowledge based repository with uh documents or wikis that would enable software developers to get started when they on board to Adidas for example there were a ton of irrelevant outdated documents created by people who had left the company years ago scrapped them all out and provided only the relevant links to their Engineers to get started that's why they use Bedrock next slide please uh booking.com for example you know they would figure out Anthony's travel patterns and they would send me suggestions and travel I and say hey Anthony maybe you should be exploring Hawaii this season summer well past summer anyways but um you know those are the kind of things that booking.com figured out they had a lot of customer data and they were able to create personalized recommendations for every customer of theirs next slide please um NASDAQ uh this is the example I talked about they presented a reinvent uh a reinvent ago and uh they built an anti-financial crime and surveillance application using uh vrock but for those who are con ered how do I get to know more about these These are public references there are customer reference architectures GitHub codes available on the Bedrock customer testimonial page all you need to do is go to the Bedrock customer testimonial page and look for these customers your customer might be one of these and maybe you can draw inspiration from what we've already done so you don't have to start from scratch last but not the least the next slide please uh Ricardo this is a slide I want you guys to take a photo of if you can this QR code actually helps you get started with bedrock everything I've talked about in the last 15 minutes if you want a Hands-On feel step-by-step tutorial process um you know go through it provide us feedback you can reach out to me or us uh I will share these details post the session with that being said I will hand it over to the folks on the other end thank you Ricardo thank you prad I appreciated that um so let's continue with our present ation so I'm going to hand it over to our friend Lori which going to present uh about LMA index which is the gro that connects everything so Lori thank you all right whoops I shared the wrong screen we practice this a million times and yet I click the wrong button there we go that's the right screen okay uh so thanks Anthony uh so far we've heard about Rag and what it is and how to use reddis to build it and how Services uh from AWS uh fit into the equation um now let's take a closer look at llama index which is uh as regar said the the glue layer that brings this stuff together um let's start with what llama index is uh at our at our core we are an open source framework we are available in python or typescript um and what we do is we help you build worldclass generative AI applications specifically we focus on Rag and making it work well and building agents on top of your rag pipeline to provide really great results um I'll go into how we do that in a bit um but in the service of the goal of making rag work well uh we have a paid service called llap parse llap par parses your documents like uh PDFs word power Word documents PowerPoint spreadsheets stuff like that and converts them into a format more easily understood by llms uh this turns out to be a critical feature of any llm based application uh making your data easy to understand is a superpower that leads to better apps if you're an Enterprise uh we also have lamac Cloud this is a turnkey service that lets you connect your de document Source at one end for instance an S3 bucket uh and we take care of parsing it indexing it storing it in reddis for instance uh and give you an API at the other end which you can then use with the rest of llama index to quickly and easily build your gen applications llama Cloud takes care of the hosting the scaling the security uh and helps you get to Market faster but a good question is why should I use llama index in fact why should I use a framework at all uh the answer is that we will help you go faster uh we have thoroughly explored the space of building generative AI applications so we've got built-ins and abstractions in place that will help you skip boilerplate under avoid early pitfalls in gen development uh you also get a lot of best practices for free we will help you go from prototype to production with your identic applications let's talk a little bit about some of those built-ins so you can understand the value that we're providing here um because they're really useful for getting agentic rag apps going um a really simple one is the router query engine this allows you to provide your agent with multiple data sources and have it intelligently route user requests to the most appropriate source for instance if you have multiple specialized rag databases about different topics or if you have tools that provide document summaries instead of rag it can detect when a user is requesting a summary instead of asking question specific questions about the document um the sub question query engine is another identic construct this one is focused on answering comp answering complicated questions by splitting them into several simpler questions it then asks the llm to answer each each of those simpler questions and once it has the answers the llm takes those answers and summarizes them into a single coherent answer to the complex question those two built-ins are agentic but not really full agents uh but we have full agents built and ready to go in the framework as well um in these ones you give them a goal and some tools and they solve the problem that you give them they exist on a spectrum of complexity so some require fewer fewer llm calls and therefore less time and money um and some require more so you have to make trade-offs for your application the simplest one that we offer is the react agent react in this case is not the web framework it means reasoning plus action this is a simple pattern that simply looks at the current state of the world and tries to figure out what the next action might be to get closer to the goal it then takes that action usually by using a tool for instance in that tool might be query this rag database or it might be make an API call uh and it either gathers more data or it changes something about the world it stays in a loop trying to get closer to its goal until it either achieves that goal or it gives up it's simple but it can be very effective The Next Step Up in terms of complexity is the structured planning agent and this agent instead of trying to get to the goal one step at a time it first constructs a step-by-step plan where each step depends on one or more of the previous steps in what's called a dag or directed A directed acyclic graph you can think of a dag as basically a flowchart bunch of boxes connected by arrows going from your starting point to your goal it follows the flowchart and every few steps it observes the world to see if things are going according to plan if not it can repan a new dag to try and solve the problem that ability called self-reflection uh is a key feature of true agents getting even more complex is language agent tree search this is one of several tree based approaches uh to agent reasoning instead of generating a single linear plan like like a dag laps produces a group of potential next steps and then from each of those next steps it predicts what the state of the world would be and then it produces another set of next steps from each of those States it repeats that process until it has this tree of possible steps and states it then looks at all of the leaf States the furthest away uh and looks for one that is closest to the goal it then picks that one collapses the tree and follows the steps that go down to that particular Leaf if it's not yet reached the goal it can then repeat the process creating a new tree until it does the built-ins that I just described to you are super powerful but you'll still want to instill your own flavor uh and encode your own processes into your agents and that is where workflows come in in llama index workflows are a powerful event-driven abstraction that lets you create Agents from scrp scratch if you want to or use our built-in agents in combination with your own agents uh and even create multi-agent systems where multiple agents cooperate to solve a problem users of llama index tell us that they really love workflows uh and they love the event-based approach and find them a Superior Solution to graph-based approaches that you might find in other Frameworks workflows let you create Loops branches retries and they have built-in State Management and they allow you to take existing workflows and extend them simply by sub classing them observability is a first class feature of workflows and they come with a built-in visualizer this is an example of uh the built-in visualizer in action this is a simple identic workflow in this one you can see uh that the start event kicks off the flow and the existence of a start event triggers whatever function is marked as handling start events in this case the answer query function answer query tries to answer the question contained in the start event uh it could do that by using one of the built-in agents that we discussed earlier uh and then it performs self-reflection and judges whether the answer is good enough if the answer is good enough it generates a stop event and the workflow completes if it can't answer the question it generates a failed event and that triggers the improved query function which tries to come up with a better more detailed question that generates a query event which Loops us back to answer query and the process starts again but this is a pretty simple flow a much more complex flow that we have built that's about to become a buil-in in llama index itself uh is the multi-agent conge we found this pattern very useful for people building support assistant or customer service Bots or sales assistant what it allows you to do is to create multiple agents Each of which is specialized in answering a specific kind of question or taking a specific kind of action and it allows you to combine them into a single system with an orchestrator agent which is the thing that talks to the user the orchestrator based on its conversation the user decides which agent is best to answer the question and it passes the user back and forth between agents as necessary uh We've expanded this pattern to allow human approval if the agent is going to take some important action in our example uh it's a financial services bot which will let you log in and check your balance but it requires approval before it will do something irreversible like transfer money uh you can check out how all of this works uh at the repo link on this slide now said that one of the great things about llama index is that we help you go from prototype to production uh and that is where llama deploy comes in this is a framework that can take workflows built in llama index and automatically turn them into microservices each workflow can be scaled and managed independently with built-in retries and other features you need to make a distributed system work well llama deploy works by spinning up a control plane which manages State between workflows and acts as an orchestrator IT manages a message bus which the agentic microservices communicate with passing messages back and forth between agents we are eager to see what people build with llama deploy and you can check it out at the GitHub repo in this link uh so that's the lightning tour of llama index llama index and the things that it can do for you and how it lets you build complex uh powerful identic systems so now let's pass it back to Ricardo uh who will show us a demo of some of these pieces in action thank you Lori appreciate your comprehensive explanation about yma index uh so let's resume our presentation so right now what I'm going to do is uh what we call put everything together right so I think that's a perfect time for us like take that deep breath and say wow that was a spin right like a a lots of information about Amazon badrock amazingly presented by Anthony Prasad and lots of cool information about how the framework LMO index work also amazingly presented by Lori so I think this is all interesting but nothing replace that experience of seeing everything together in a real demo so that's what I'm going to do right now so Lori Anthony and I we've working together to create this demo which is essentially a chatbot that uses all these ingredients that we have discussed so far so it uses Amazon badrock as the LM right uh specifically uh a a12m labs for for the Foundation model and then we're constructing this agent using Lima index so you get to see the code and I'm going to explain how it works so what the code does is we're upload a PDF document about this uh it's about this called chavy Colorado right from it's actually out dat it's 2022 so but it doesn't matter for the sake of what we are going to present here today and then this agent this bot is going to allow us to uh ask questions about this pedf and then we're going to see all the components like uh working together so I'm going to actually switch my screen here to the notebook and don't worry because in the end of the presentation we're going to make this notebook the code sample that holds this notebook available for you so if you can try yourself if you want it but as you can see here this architecture is essentially uh the materialization of we have discussed so far so we're GNA create this ingeson pipeline using yma Index right I'm going to explain the code in in a in a minute but as you can see here this uh red uh icons here Radice is being used throughout the entire architecture as this kind of a glue right this this there's two types of glue in this architecture so bear with me the first one it is the runtime glue so as the messages and the embeddings and the the the orchestration executes in runtime rad is is taking care of making sure that all the components right the messages are flowing through right so that's the first type of glue the second type of glue is yma index but is more in a code level so this is what you're going to use to tie everything together so the flow is we're going to upload this to liac cloud and then we're going to process this using this inges pipeline we're going to programmatically create and then it's going to create a set of embeddings that in turn is going to be started at Radice and then our agent is going to use those uh embeddings to enrich the questions that are being the answers that are being provided along with the llm and then this is what the user is going to receive right which is the typical workflow for agents but later in the Dem we're going to also show how the santic cache works so we can get to see about a question being asked and a similar question being also asked and the uh delay of how much it takes to respond the question going to be much much faster right so in this notebook right I've already executed a set of the steps the installation of the dependencies because I mean yeah it's very trivial uh the creation of the darker container that is a cutes Rus this is actually very interesting for you to know because um in this notebook you can actually spin up very quickly a new Reds container with red Stacks that contains the compo all the components that required for this architecture so right now the container is running um if you are going to try yourself this notebook mind that uh you're going to have to provide two AP II Keys the first one is for dma Index right for dma cloud you're you create an account there you can create an account for free and then you create an API key for yourself and you're going to open this envy. Json file where you're going to set your API key there so that's your first API key the second one is surprise surprise from AWS so in order for you to interact with the Amazon Bedrock as a service you have to provide an API key and that's important an API key that is associated with an account that has ACC uh Direct access rice to Amazon Bedrock right so this is the heads up number one heads up number two and this is more like a Amazon Bedrock thing uh not only by default when you create an account AWS not only foundation models are enabled by default there are 100% available for you to use but they're simply not enabled so you have to in the AWS console you have to manually request access to these models for you to use and then well all of them will be available to you right so keep that in mind and and the AWS documentation everything is fully described so uh if you are in doubt you can always go there uh this is the document that we're going to use so uh this is the part of the notebook that downloads this uh PDF here locally is already downloaded and then the last step that I've already executed is this so this is the first step for you to understand about how yma uh yma index works as a framework so what we have done here is using the built-in constructs from yma to par the PDF documents and create what we call documents right which is like a series or list of chunks or regions of the documents so each one of those documents represent a portion of the PDF document so they're all available here in this component called documents so what we're going to start doing together and that's the exciting part right it is the construction of our index at radis that is going to store and obviously as a name implies is going to index all those embeddings right so the first thing you got to do there's this two constructs that you have to be aware of and as you can see uh it all uses the yma index API and SDK the first one you have to instantiate an embedding model right so there's a bunch of uh this is as an abstract class in Python so you have a different implementations of embedding models one of them being Bedrock embedding right which in this case we're going we're going to use Amazon Titan and then the second one you going to actually Define the schema at radus right uh where you are going to dictate like a different information about the the the feuds that are going to be associated with this index and more importantly the type of vector that we're going to use like so for example in this case we are using a flat Vector with uh 1536 Dimensions right and then the the algorithm that we're going to use to calculate the distance matric which is the how close the embedding is to the question that is been asked is going to be the algorithm called cuine right so I'm going to execute this and this is going to be the prerequisits to start impling this right so this is the pipeline let me execute before I forget this is the pipeline that uses uh yma index for us to create so uh I'm going to skip the import part right and then basically as you can see here it is a it's a very elegant API that allows you to construct this injection pipeline so it's very at least for for for my taste as a developer is very clear to me so you are creating a new injesting pipeline where you are providing almost as a Constructor actually not almost as a construct this is is a constructure in Python language for what is going to be the Transformations which is an array of Transformations you can have set a lot of them uh what's going to be the do doc store or docent store because as you store your in badings at r which is going to be your facts you optionally can use radus to also store your Dogman so oh Ricardo why I'm going to do this because guess what you can allow now not uh multi- types of searches you can search embeddings and vectors and you can search the actual document the full text search for whatever is in the PDF document so obviously this is an optional step but it is always good idea for you to kind of use the same data store for all of them and as similarly this is going to be what the vector store they're going to use and then option you can also use rad is in this case for the cache right so R why cash because in runtime when Lima index does what it's supposed to do which is the orchestration parts which is the communication between the layers there has been a lot of like multi-stage in between let's call state that the framework has to materialize and periodically flush to maintain the consistence of the execution so this is what we call the G in Cache right for and and the document strategies is basically uh how the documents are going to be created like if there's one existing we can up insert then which is uh instead of create a new one with a new version just up update with the existing one and then here's the interesting part like I'm going to actually uh bring up radis Insight uh for those of you that never uh heard about radis Insight this is a UI that radis provides for free for you to actually interact with your radis database and there's a bunch of powerful features here but right now as you can see here this is a brand new container running radish so there's absolutely no keys start so what I'm going to do I'm going to click here on this execution of the pipeline so we're going to provide the list of documents that has been chunked by Li index and then we're going to execute in this pipeline that we've just defined here so what we're going to notice here is that uh a set of embeddings are going to be created right so this is are the vectors that represents the edance created and if you go back to Red Insight you going to actually see all those let's call chunks actually this is actually the proper term that actually is a chunk is a portion of your document that is it currently state in a hash at Vector right and you can also see the vector uh obviously you're not going to be able to actually see this is a a form of expression here but this is actually the vectorization representation of that chunk right so this is actually a good practice a lot of people ask Ricardo do I have to create multiple chunks for a specific one doc or maybe one huge chunk I mean you can it's more about you shouldn't like it's better if you can like break it down into smaller chunks this going your searches and uh is going to perform faster and more accurately right so it's mostly a trade-off between performance and uh storage complexity right want a huge single line on your Radice or you want multiple ones so this is uh more like in side of a best practices for you to understand now just recap what we have so far we have radis up and running rad is already containing all the chunks from your PDF document so everything is already here along with some of the uh there's a a hash here that also stored the references of all the chunks as a document not not as a vector so we have everything we need to start working with your agent right so what we're going to do now is actually start the creation of the agent so this is the building block that you're going to use first of all you have to instantiate an llm right so as just like the the model that we used before there's different implementation support implementation of llm supported by Li index one of them is badrock so in this case we are using AI 21 labs and then we are also using instantiate and chat storage and chat memory remember chat memory so we can keep the context of the conversation ongoing so the llm doesn't forget what have been discussed before so this is what we call chat memory uh and the chat starts as then PL it's going to be where we're going to store our chat and then with that done this is the creation of the document itself so we're created this function called build doc agent going to provide a document as a parameter and this is going to be the creation of the agent I'm going to run through the entire code you can do this on your own time when you see the sample right let's actually try this out so I'm going to provide the very first document of that array of chunks that we've created so we can instantiate the agent itself so is here and a doc summary so if we look to the doc summary um do summary so you you can understand like we provide the description here provide information about the specs and attributes of CHF Colorado right so it basically is explaining you what is the nature of that document so now let's start using the agent right so we have here a set of questions that are related to the very own nature of the docent so the first one is what is the seating capacity of the vehicle right so if we execute this part this is the agent already interacting with the llm and it's being augmented with facts from the PDF so in this case the extended cab has two seats and the crew crab has five seats so this is information that is coming from the PDF so what is the towing capacity right is another question so if we execute this we're going to have the llm correctly responding that it is 3500 pounds of uh for the two in capacity right this is another information if you do like a common a or control F on the PDF document you're going to see this information there right um and then you can always check the history of the shf so far so all the questions and answers that has been in this interaction between the human and the machine is always started on our shat and just to wrap up uh let's talk about the santic cach right I'm going to create one here and as you can see here santic cach is another construct or building block that uh supported natively Alma index so yet another reason for you to start using Li Enix in your implementations and then what we're doing here is basically uh uh in the in the caching and descripted system word we call this cach aside uh pattern right so what what is Cash aside is when you start first investigating if the backend system has the information you wanted and if it doesn't you basically go to the back end provides the answer and cash it right so the next time you go it you can basically use the cash for this right this is all good and and stuff but the main problem of the nature of llms is that you don't want to provide an answer just for the sake of responding faster if the answer is incorrect right so for example I'm going to execute here this question how many doors the truck have right so the answer here I mean it's it's currently answer here uh okay doesn't matter the point the point really matter is that it took 7 Seconds to execute this now if we execute the same question again like how many doors does the truck have and how many passenger doors doors the truck have you're going to see that the same answer is going to be provided is the wrong answer no mod about this probably some something's going on with the back end system here but the point is the the very own question was associated with the previous one so if you execute continuously this now it's hitting the cache instead of going to the llm so uh in the real world what we going to avoid is this like a very huge delay to go into the llm because again it is a distrupted system right so this is for you to know understand uh what's going to be like the the the idea of the santic cash right and then as you go to this uh notebook yourself don't forget to kind of a stop the container that is running Docker right because you created one in the beginning of the notebook I highly encourage you to actually try this notebook yourself and with that said let's wrap up what we have said so far so if you want to create Rag architectures and make this red entire implementation and execution to be fast Radice is the component that you can use for this right this is our rad and bu that's what we do best like how making things fast right and Amazon badrock is this amazing piece of architecture that provides you with a different Foundation models and as pad presented in the beginning right it has a complete integration the set of features that is going to be very handy when you were implying Solutions like this and as the glue for the developers to create your own orchestrations both and the development level and the production level yemma index is the component that ties everything together so as I promise you can try this demo yourself I would highly encourage you to take a screenshot of this barcode if you scan this barcode is going to bring you exactly to the uh GitHub reposer that has that notebook so we can try the demo yourself hopefully the sented cach part is going to work for you uh and not have the glitch that I've had here and this is more like a personal request if you have some time after this webinar please scan this barcode and give us your feedback how we did go today right um so we provide you a lot of information I know that's kind of a hard to digest but your feedback is very important for us to understand what we can improve what can do better and also to kind of uh uh share your experiences with us as well so with that said Thank you on behalf of all my friends Anthony prad Lori you guys are all amazing and I'm looking forward to see you back in another webinar very soon thank you cheers guys

Timestamped Segments

[00:01] all right uh thanks everybody for coming here today to this webinar I'm super excited to be here talking about how to architect AI agents with yma index and Technologies from AWS such as Amazon Titan and Amazon badrock uh this is a huge topic these days and probably you you are aware of this but we're going to do like this presentation that's going to be broken down into three parts we're going to talk about rag as an architect we're going to talk about Amazon badrock and its foundation mods this support it has for it and then we're also going to start talking about the yemma index so
[00:35] um this is the agenda for today right uh and I'm going to let the the presenters like introduce themselves in a minute but uh after this we're going to start talking about rag agents and then Red's AI for agents and how R fits on this entire architecture and obviously we're going to talk about Amazon badrock and also orchestration with yma index with me today we have and prad from AWS which is an amazing person and knows tons of AWS specifically Amazon badrock and he's going to be talking about this in a
[01:08] minute so prad if you want to look introduce yourself for a minute thank you thank you Ricardo uh pleasure to be here with everyone uh I'm Anthony prad deage senior partner sa here at AWS uh been with the company for close to nine years uh I still am very much a recovering data engineer uh which used to be uh the first part of my life at Amazon uh at the end of the day uh the important Essence is uh garbage in garbage out no matter how sophisticated your llm is no matter how great your application is if you don't have the right data set and the right the way you
[01:41] structure your data is actually going to power your gen applications so with that being said I'll hand it over to my good friend and Pier Ricardo take it away thank you pad um and also we have Lori which is the VP of developer relations of yma enx so Ari hi everybody yeah uh so I run developer relations at llama index you're going to hear more about what llama index is and why you should use it uh in the course of this presentation um as background on me uh a long time ago I was a co-founder of npm
[02:14] Inc so uh you may remember me from when I used to talk incessantly about javascripts uh now I talk incessantly about AI but the Common Thread is that I talk incessantly that's super cool I I remember that Larry so I remember that uh thank you everybody for coming here today uh and also we have Ricardo fer that happens to be this person here so I am the lead developer Advocate at rad is and um I've just joined a company I'm actually came in front aw so prad and I we were like co-workers uh a month ago actually that's quite new and uh I'm
[02:47] very excited with all the things that Rus has to offer and integration with the other technology so I'm going to start off like uh I would like to put everybody on the same page where about the understanding about rag right uh rag is a common term these days like it it's definitely one of the most like Hot Topics in the technology industry but uh it is always important for us to kind of uh make sure that the thms and the understanding of it is the same right whether if you are an expert whether you were uh hearing about rag for the first time so the building block of rag
[03:21] architectures is what we call edings right so when it comes to data right uh this is EXA the raw material that rag architect are going to use to do what it's supposed to do and Ricardo what is rag in the end of the day rag is the ability for you to augment uh request and answer scenario with facts right so the way you create and store those facts in the form of embeding right the retrieval part is actually the process of when you actually are responding to a question that has been sent to a llm or
[03:56] large language model or a foundation model in the jargon ad Bedrock Amazon badrock is it's going to make sure that the process of the answering is going to be as accurate as possible right and we call all of this process end to endend the amen and generation right so this is what uh ragged is all about um if we look at the typical architectur for rag uh architector this is usually what we're going to see out there whether you were imployment with uh radis or not whether you implment with uh Amazon
[04:29] Bedrock or some specific llms right you're always going to see what we call The Usual Suspects right so the knowledge basis one of them right the knowledge base is where you're going to set your context and those facts that were mentioned before right and also you're are going to have the the the Persona of the user quer is exactly with the the questions that your user or customers are asking to the LM component right and then the component that's going to tie the llm to the knowledge base and the embeding service is going
[05:02] to be your orchestration framework to which in this case we're going to be using yma index and there's a huge reason for you to start developing your agentic rag with yma index so Lor is going to explain a little bit more later but usually this is what you are going to see um uh when a typical rag implementation all right you're also going to use this component of the embeding service which is the service that's going to transform data into a set of V or embeddings right that's going to become your facts right and then here is what we are calling the
[05:37] evolution of a rag architectur evolution in a sense of whether you were implementing uh regardless of the technology that you are using to implement rag you are going to face some problems let's call them industry customer problems that are uh unipr presentent everywhere right one of them is for example the process of the need for event skew right for example all of this in the end of today isri is a descripted system that in creates a lot of network interaction right so let's
[06:10] say there's a your website just came live and it become a white success so a lot of customers are using the AI agents to ask questions about whatever the their business needs are and then you have to have some component that's going to throttle those interaction so you don't overload your backend systems right so this is something that we are fixing and soing with radis these days also there's the component of a santic cach right santic cach is the ability for you to surprise surprise use radis as a cache which is the most popular use case for radis but with a huge
[06:44] difference right and here comes the part we call santic cache it's a specialized type of cache that knows how to interpret a question and make sure that even though it's not the same wordss that are being asked but if some similar question has use the same answer it retrieves from the cash instead of going for it entire to the llm and the huge value for this because you're going to save a lot of a token interaction for the llm service and also the identic memory so you can create a more comprehensive context every time you are
[07:16] going to deal with questions so everything that has been discussed before is considered in the upcoming and forthcoming questions right uh and lastly we are also have this component called santic router which is uh it is a component where you can use for when you don't want to rely only one specific llm that specialized in aing you can create different llms or use different llms that has been training into specific domain models or domain expertises and then you can route accordingly depending of the context that you are dealing with
[07:49] so all of this create a more complete and more human friendly version of architector right and of course uh the usual using R is as a knowledge base to store the facts um another Trend that is probably kind of a very it's not so common as ragged itself but it's starting to kind of a uh keep up these days is the concept of agent supervising so agent supervising is this ability for you to create a much larger view of one
[08:23] specific agent or multiple agents so you create a more human friendly version of your the the human interactions that your customers are going to use with the agent AI so again radus is going to be using as a persistent layer and then there's going to be some buil-in integration with the agent supervising here that's can be also Implement with Y index um and lastly red is for AI which is kind of the the the component that we're using for this new generation of uh rag architecture is it has this
[08:56] foundation in this component that we call rag Square engine right this is something that we' build on top of radus for the radus that you know and love and then we have this kind of a radis quare engine that it it is was originally created to allow radus to perform Json document search on it but when we start adding support for vectorization it also the same engine allows this vectorization searches and centic cache and all the components that we've discussed so far to also work this is the foundation that actually gives birth to the flexible development that we have
[09:30] available including integration with yma index so there is some very interesting partnership that we did with um yma index to integrate the Frameworks and the SDK and the code level and also the components that we are calling then if you look the architecture from the architectur perspective and last that a code perspective those are the components that we support that are backed by the qu engines and all of this is what we are using to sustain the use cases that you can implment with this architecture right so this is the first
[10:05] step for you to understand the this new type of approach for us to implement red architectures but one very important concept that you have to master to understand this is exactly the how the llms are implemented and how this integrate with the radish technology and for this I am going to actually invite my friend prad that's going to give you an overview about how Amazon Bedrock fits into this thank you Ricardo I would still need your help to share the slides absolutely no worries my friend it's all it's almost we're almost getting there I
[10:37] swear uh but thank you um let's start with the previous slide so I just want to spend a moment uh you know help to help you guys understand uh why we're having this conversation today if you think about it in 2023 uh a lot of our customers were um still doing poc's right and a bunch of questions kept uh coming up hey um how do we go from po to production what am I supposed to do uh how am I supposed to prioritize my projects or um how can I lower my costs
[11:12] um you know stuff like that while Ricardo is trying to fix uh the slideshow but uh long story short customers have realized that uh we were actually able to build production ready systems and the reason we were able to do that is because uh we leveraged Amazon Redrock in combination with isv Solutions like redis right um Ricardo I think you can just share the slides I think there's a screen recording delay when you share slides like if you go to presenter viw or just present
[11:47] slideshow uh there might be yeah it's fine but hey what's without a imperfect demo that's how it is but having said that um in 2024 we've achieved massive success uh with our customers uh we've seen systems in production uh last year at Rin vent there was a presentation by NASDAQ would actually involved a fraud detection system that was leveraging database software solution uh and we were able to take that to production um and we also noticed uh for example uh uh doctors were able to diagnose the uh
[12:21] treatment that was supposed to be given to a patient based on historical data so we created production grade ready systems uh in 2024 and now we're taking it to the next step with that being said um I want to cover a few topics today on uh Amazon bedrock and what it is uh and how we've got here right uh Ricardo I think you're sharing this wrong slide maybe you want to go back a few slides um go back and for yeah uh we can see the Bedrock models slide now no we gotta go the other
[13:03] way um in Reverse I think you're moving forward Ricardo okay um yes let's start from here um what are the needs what is the service itself uh I'm going to talk about some of the key features and we're going to touch upon uh some of the customers the only slide that I want you guys to maybe take a photo of is the last slide where we'll share a QR code on everything that I've
[13:36] just talked about it's a Hands-On Workshop uh it's a 4-Hour Workshop that lets you actually get started on everything we've talked about uh let's move on to the next slide please yes okay the top three questions that we come across from all our customers today hey Anthony there's so many models out there what model do I use right where do I get started second one how quickly can I go to production now for this evolution of gen U during my days as a data engineer I remember when we had spent six months to productionize a click probability model this model all
[14:10] it was supposed to do was predict if this ad on amazon.com was going to get a click or not over the next seven days sounds like a simple ml problem sure but then getting data is uh is is is a problem right so that's where we took six months to go to production but today we don't have six months time last but not the least Anthony is my data secure is my data only accessible to me these are the top three questions that every customer had next slide please and to address all these questions is why we
[14:43] launched Amazon Bedrock what Amazon Bedrock does for you is think of it as a platform that can host multiple llms and Foundation models uh some of which Ricardo just talked about but think of it as a single API call you are able to Leverage the power of all these llms you're able to deploy AI agents you're able to deploy the entire rag flow that Ricardo just touched upon right right from retrieval augmentation and finally generating your output everything can be automated using bedrock in the console
[15:17] itself right and last but not the least on security I don't know how many of you guys are familiar with private link but AWS private link is a service that ensures your data coming from outside the internet into your a us bpc is absolutely secure and Bedrock is private link enabled by default so you don't have to worry about any data security consents let's move on Ricardo all right just to summarize with bedrock you can pick a bunch of models llms and Foundation models you can play
[15:49] with you can customize your model as well so let's say you say hey Anthony I want to tweak a few parameters to adhere to my use case yes you can L tweak that and customize that and guess what the copy is only in your control unless you specify permissions to your I am roles and policies the other people in the room don't get access to your model the next one is you can integrate seamlessly with other native AWS services like red shift for example if you have a data ering request you can store data in red shift and then you can feed it into
[16:23] Amazon Redrock you are not going to be charged for additional access to the service Bedrock will only charge you for the number of tokens you consume per minute to run your llm last but not the least we talked about the security and governance if your data lies in Europe we have all the local governing policies aduring to our standards and we are available in most regions across the world I would have a slide talking about that as well next please all right so this slide is uh very very outdated uh clearly outdated
[16:57] and it will be outdated next week as well as uh those of you who were following reinvent we launched something known as Amazon Nova which was our very own uh Foundation model as well it was based on or inspired from the work we did with uh amazon.com shopping experience if you guys have tried out Rufus uh and then Alexa of course the conversational style AI Hardware that has existed for years where you could place orders or you know even requests for things that you had purchased in the past um so Amazon Nova I don't I know it's not mention on the slide but I will
[17:30] share the link in the chat in a moment when I get through but the point of this slide is you have a ton of options we work with some of the top grade llm providers like anthropics of the world coheres meta Mr Lei and stability a all these guys have their LMS offered to us through the Bedrock console so that's the key takeaway of the slide next one please and I want to focus a little bit on Claude in particular right because uh anthropics cloud is actually uh one of our best performing models in terms of
[18:03] the sophistication that brings to the table in terms of the cost benefit it has in terms of the ability to scale as well so and the option I want to give to the customers here is we have something for everyone let's say you want to start small you don't have a very complex use case you don't require that level of intelligence and sophistication but you're focused on speed and cost alone then better off with claw 3 Haiku because it's the fastest performance and the low slowest cost but you get a compromise on the ability of the LM
[18:37] itself if you want something somewhere you know in between all these worlds the second best performing one is claw three office which gives you intelligence which gives you the speed and it also gives you the cost benefits but our most powerful one till date is Claude 3.5 son so if you really want to build a sophisticated intelligent application we recommend going there um and again you pay the cost but we have cost mechanisms to optimize your workload as well I'll talk about that in a little bit let's move on um okay again this is number one of
[19:13] the most s after features I'm glad we launched this a few months ago and the reason we did this is um you know you have a ton of models out there right so let's say you say Anthony how do I know which model performs better which model performs better than the other one well guess what Bedrock introduced something known as a model evaluation report and we have two options here so if you follow my theme so far we are going to have humans we're not going to replace humans we're going to enable and help humans build faster better geni applications so we have two options here
[19:46] automatic or human evaluation if you pick automatic you will let Bedrock decide based on the metrics and the kpi I know the slide might be a little convoluted but there's a ton of metrics and there's custom metrics you can Define which will specify what the performance means for the impact of the end output or if you pick human evaluation method you can specify your email Alias or your team's email Alias and you get a report saying hey user here's how the performance metrics looks model B performs better than Model A if you give a thumbs up it kicks back the
[20:18] workflow in your Bedrock console it's that easy next one please um I want to spend a little bit time on this one right so going back to C conversation on rag right like how do you structure data how do you store your embeddings right once you figured out all that right the next question is how do I improve the accuracy of my model response right because obviously if you're a customer that cares about the data itself the quality of data itself there's a combination of techniques we want to introduce to you to enable you to provide high quality data results for your end users prompt engineering as
[20:53] Ricardo said you it's a simple standard well-known technique you provide prompts the LM gives you your response you you narrow down the search results and then you get a better response right then the other one is the rag workflow I'm not going to spend time on rag because Ricardo talked about it but then the other two approaches I want to talk about is fine-tuning and pre-training now if you're a data scientist and if you know what good looks like from your model if you know the false positives if you know the false negatives you can Define that within your S3 bucket for example and then you let the model skip records for example that are not in line
[21:28] with what you're expecting and then you can only consume the data that is required for your model so that's one way of fine tuning pre-training is for people who don't know what their data looks like let's say you you come in and you say Anthony I got this raw data dump I don't know what to do with it I haven't checked it out then you can opt for continued pre-training because then what you do is you'll allow Bedrock to go through the data and give you what is in the bucket his3 bucket itself so it'll break it down into buckets and tell you this is exactly what your data looks like right and then it can also generate mock data sets so if you're a
[22:02] data scientist you want to productionize a model you want to go from po to production you need to test your end results you need to test the use cases you need to test every possible Edge case we can generate mock data sets with contined pre-training that's one of the feature I would highly recommend you guys to check it out and if you want to get a perfect result I would say play around with the combination of all these four techniques again depends on the cost you want to incure it depends the time you want to consume let's move on all right this is something that we introduced very recently which is known
[22:34] as bedrocks prom flow so if uh you know for those who are not familiar with this it's like a drag and drop experience right so you have a bunch of icons there and let's say you built your data source which is starting from a legacy system or it's an S3 and then you pick a combination of models that you want to use you can do this through the console itself so you can select options through a drop and drag uh Dropbox bar that's how easy it is so that's why we created uh promp flow and it also helps with collaboration within the team as well right so think of it as a playground uh
[23:07] for people to test it out you can move on uh and this is how it looks like right it's that easy drag and drop interface versioning is also enabled so if you make changes to one workflow you can revert back to the previous version and really has changed the lives of developers uh and it's made it very very uh user friendly next please all right uh here I want to spend a little bit of time right Bedrock has something within the console itself a feature an optional feature known as
[23:39] knowledge base and I want to take a step back to help you guys understand what knowledge base means so let's say um you know a pharmaceutical use case where the doctor is trying to figure out what is the right patient diagnosis for patient XYZ based on five years worth of clinical trial data now question how does your llm know what your patient went through no idea it's domain knowledge and that domain knowledge is going to be input to your knowledge base so what knowledge base is actually doing is is providing domain knowledge to llm
[24:13] in educating the llm about what exactly we're looking for in this patient based on the clinical trials so it's as good as an input to your uh Bedrock uh you know llm and this is uh something that we're working with partner like redis redis is one of the few isvs who is actually present as a vector database offering on the Bedrock console we are slowly trying to incorporate redis as a knowledgebase input where the reddis database itself can be an input to knowledge base so then we're trying to eliminate the need for moving data
[24:47] outside your database exporting to S3 and so on so we're working on that with Red's team as we go on but again in this use case the user files a query looks for data within the domain that is fed as an input to your knowledge base you like the data or not you're looking for more details you provide the prompts and then in this example we've selected a combination of these four models not just one and then eventually you get your final answer and that's how knowledge bases work let's move on all right there we are uh red is Enterprise cloud is uh actually
[25:22] available on uh your Bedrock console we have been available for over a year now um we were launch launched at last reinvent we've optimized and launched even further Innovations with red Spector database there's something known as llm rag evaluation that we launched recently at ROM rent um so if you're a reddish customer you have your data in redish uh reddish Cloud as a vectors then you can definitely check it out uh I would highly recommend you to explore this offering on the console next slide please all right we already talked about fine-tuning and pre-training I would
[25:56] highly recommend you guys to go through that with the Bedrock console the step-by-step tutorial that I'll share at the end will also provide you steps to do so let's move on Ricardo all right I do want to say that since I spent a lot of time on cloud 3 uh we have fine-tuning specifically for CLA 3 models and Amazon Bedrock is the only fully managed service that provides the ability to fine-tune uh Cloud models so that's something that you guys if you're working on I would highly
[26:29] recommend checking it out because the feature exists for you out of the box next slide please all right the conversation of the day right we talked about AI agents and you know how does Bedrock fit into this piece right now no solution can be built just by one service it is a combination of services like Ricardo said you have to use reddis as a vector store for the reasons mentioned and then what next so in this way you can Define bedrock agents to execute on your behalf so what does that mean every time
[27:03] there is an API call made to look for a specific service you don't have to write separate code facets to get that API to work as expected and pull the data for example or or send it back you can orchestrate that to the Bedrock agents it's as simple as providing basic instructions through English select your foundation model if you remember the previous slide for four llms you've selected select all of them provide those instructions in the Bedrock console the next one what am I doing break it down to steps look for this API
[27:38] if I'm checking for is this asset in stock or not I'm just giving a retail example let's say you're a retail company you're trying to find out if this product exists or is in inventory or not provide that orchestration piece in the instructions the next one select the right data sources where is my model going to look look for this data is it in an S3 bucket is it in the Legacy store select that leement data source and last but not the least as for those of you are familiar with AWS reference icons that's a Lambda function right so we given an example here is let's say
[28:12] you have all the things in place and you want your Lambda function to specifically execute on this piece of code with this API call Provide those instructions as well this entire thing can be automated so it's a onetime human intervention we're not taking the human out of the loop this second thing is we also have a Chain of Thought tracing enabled which allows you to diagnose and troubleshoot the issues at the lir for those of you you are familiar with cloud formation templates we're not going to give irrelevant error codes I was one of them who kept complaining I'm sure Ricardo can attest to that some of the error codes for gibberish didn't really
[28:45] provide you exact details but the Chain of Thought tracing actually gives you specifically where the problem is next slide please all right um for those of you are familiar with the ec2 consumption pricing similarly we have on demand for those of you who are not aware of what workload it is how many tokens you're going to consume we recommend starting with on demand if you are familiar with your workloads and if you're familiar with the number of tokens you're going to consume per minute I would recommend uh provisioning with ahead of Advance because you don't want your application
[29:17] to ever go down and then we also introduced batch inference recently because there are companies who actually want to process the data in chunks or micro batches that is one way to do it uh via Redrock next slide please all right we also introduced cross region inferencing what that means is if you're a European client you're running your llms within data reiding in Europe we probably don't have enough resources to run your models then we're going to go to another region fetch the resources temporarily and help you run your workflows and that is why we
[29:51] introduced cross region inferencing we don't have extra charges for you on that account let's move on all right as we continue to grow uh these are the regions that we exist in today we also exist in govcloud if you're a gocloud customer we highly recommend checking that page we also are in deance to all the Hippa and gdpr compliance across the globe and we're contining to expand as we go on next slide please all right we're almost there to the end uh I know I've had to rush through the glitches we've had in the
[30:24] technical presentation but here we are Adidas uh they built uh you know uh a Bedrock knowledge based repository with uh documents or wikis that would enable software developers to get started when they on board to Adidas for example there were a ton of irrelevant outdated documents created by people who had left the company years ago scrapped them all out and provided only the relevant links to their Engineers to get started that's why they use Bedrock next slide please uh booking.com for example you know they would figure out Anthony's travel patterns and they would send me
[30:56] suggestions and travel I and say hey Anthony maybe you should be exploring Hawaii this season summer well past summer anyways but um you know those are the kind of things that booking.com figured out they had a lot of customer data and they were able to create personalized recommendations for every customer of theirs next slide please um NASDAQ uh this is the example I talked about they presented a reinvent uh a reinvent ago and uh they built an anti-financial crime and surveillance application using uh vrock but for those who are con ered how do I get to know
[31:29] more about these These are public references there are customer reference architectures GitHub codes available on the Bedrock customer testimonial page all you need to do is go to the Bedrock customer testimonial page and look for these customers your customer might be one of these and maybe you can draw inspiration from what we've already done so you don't have to start from scratch last but not the least the next slide please uh Ricardo this is a slide I want you guys to take a photo of if you can this QR code actually helps you get started with bedrock everything I've talked about in the last 15 minutes if
[32:03] you want a Hands-On feel step-by-step tutorial process um you know go through it provide us feedback you can reach out to me or us uh I will share these details post the session with that being said I will hand it over to the folks on the other end thank you Ricardo thank you prad I appreciated that um so let's continue with our present ation so I'm going to hand it over to our friend Lori which going to present uh about LMA index which is the
[32:36] gro that connects everything so Lori thank you all right whoops I shared the wrong screen we practice this a million times and yet I click the wrong button there we go that's the right screen okay uh so thanks Anthony uh so far we've heard about Rag and what it is and how to use reddis to build it and
[33:09] how Services uh from AWS uh fit into the equation um now let's take a closer look at llama index which is uh as regar said the the glue layer that brings this stuff together um let's start with what llama index is uh at our at our core we are an open source framework we are available in python or typescript um and what we do is we help you build worldclass generative AI applications specifically we focus on Rag and making it work well and building agents on top of your rag pipeline to provide really great results um I'll go into how we do
[33:43] that in a bit um but in the service of the goal of making rag work well uh we have a paid service called llap parse llap par parses your documents like uh PDFs word power Word documents PowerPoint spreadsheets stuff like that and converts them into a format more easily understood by llms uh this turns out to be a critical feature of any llm based application uh making your data easy to understand is a superpower that leads to better apps if you're an Enterprise uh we also have lamac Cloud
[34:16] this is a turnkey service that lets you connect your de document Source at one end for instance an S3 bucket uh and we take care of parsing it indexing it storing it in reddis for instance uh and give you an API at the other end which you can then use with the rest of llama index to quickly and easily build your gen applications llama Cloud takes care of the hosting the scaling the security uh and helps you get to Market faster but a good question is why should I use llama index in fact why should I use a framework at all uh the answer is that we will help you go faster uh we
[34:49] have thoroughly explored the space of building generative AI applications so we've got built-ins and abstractions in place that will help you skip boilerplate under avoid early pitfalls in gen development uh you also get a lot of best practices for free we will help you go from prototype to production with your identic applications let's talk a little bit about some of those built-ins so you can understand the value that we're providing here um because they're really useful for getting agentic rag apps going um a really simple one is the router query engine this allows you to
[35:22] provide your agent with multiple data sources and have it intelligently route user requests to the most appropriate source for instance if you have multiple specialized rag databases about different topics or if you have tools that provide document summaries instead of rag it can detect when a user is requesting a summary instead of asking question specific questions about the document um the sub question query engine is another identic construct this one is focused on answering comp answering complicated questions by splitting them into several simpler
[35:54] questions it then asks the llm to answer each each of those simpler questions and once it has the answers the llm takes those answers and summarizes them into a single coherent answer to the complex question those two built-ins are agentic but not really full agents uh but we have full agents built and ready to go in the framework as well um in these ones you give them a goal and some tools and they solve the problem that you give them they exist on a spectrum of complexity so some require fewer fewer llm calls and therefore less time and
[36:26] money um and some require more so you have to make trade-offs for your application the simplest one that we offer is the react agent react in this case is not the web framework it means reasoning plus action this is a simple pattern that simply looks at the current state of the world and tries to figure out what the next action might be to get closer to the goal it then takes that action usually by using a tool for instance in that tool might be query this rag database or it might be make an API call uh and it either gathers more data or it changes something about the
[37:00] world it stays in a loop trying to get closer to its goal until it either achieves that goal or it gives up it's simple but it can be very effective The Next Step Up in terms of complexity is the structured planning agent and this agent instead of trying to get to the goal one step at a time it first constructs a step-by-step plan where each step depends on one or more of the previous steps in what's called a dag or directed A directed acyclic graph you can think of a dag as basically a flowchart bunch of boxes connected by arrows going from your starting point to
[37:34] your goal it follows the flowchart and every few steps it observes the world to see if things are going according to plan if not it can repan a new dag to try and solve the problem that ability called self-reflection uh is a key feature of true agents getting even more complex is language agent tree search this is one of several tree based approaches uh to agent reasoning instead of generating a single linear plan like like a dag laps produces a group of potential next steps and then from each
[38:07] of those next steps it predicts what the state of the world would be and then it produces another set of next steps from each of those States it repeats that process until it has this tree of possible steps and states it then looks at all of the leaf States the furthest away uh and looks for one that is closest to the goal it then picks that one collapses the tree and follows the steps that go down to that particular Leaf if it's not yet reached the goal it can then repeat the process creating a new tree until it does the built-ins that I just described
[38:42] to you are super powerful but you'll still want to instill your own flavor uh and encode your own processes into your agents and that is where workflows come in in llama index workflows are a powerful event-driven abstraction that lets you create Agents from scrp scratch if you want to or use our built-in agents in combination with your own agents uh and even create multi-agent systems where multiple agents cooperate to solve a problem users of llama index tell us that they really love workflows uh and they love the event-based approach and find them a Superior Solution to graph-based approaches that
[39:15] you might find in other Frameworks workflows let you create Loops branches retries and they have built-in State Management and they allow you to take existing workflows and extend them simply by sub classing them observability is a first class feature of workflows and they come with a built-in visualizer this is an example of uh the built-in visualizer in action this is a simple identic workflow in this one you can see uh that the start event kicks off the flow and the existence of a start event triggers whatever function is marked as handling start events in
[39:49] this case the answer query function answer query tries to answer the question contained in the start event uh it could do that by using one of the built-in agents that we discussed earlier uh and then it performs self-reflection and judges whether the answer is good enough if the answer is good enough it generates a stop event and the workflow completes if it can't answer the question it generates a failed event and that triggers the improved query function which tries to come up with a better more detailed question that generates a query event which Loops us back to answer query and the process starts again but this is a
[40:22] pretty simple flow a much more complex flow that we have built that's about to become a buil-in in llama index itself uh is the multi-agent conge we found this pattern very useful for people building support assistant or customer service Bots or sales assistant what it allows you to do is to create multiple agents Each of which is specialized in answering a specific kind of question or taking a specific kind of action and it allows you to combine them into a single system with an orchestrator agent which is the thing that talks to the user the
[40:55] orchestrator based on its conversation the user decides which agent is best to answer the question and it passes the user back and forth between agents as necessary uh We've expanded this pattern to allow human approval if the agent is going to take some important action in our example uh it's a financial services bot which will let you log in and check your balance but it requires approval before it will do something irreversible like transfer money uh you can check out how all of this works uh at the repo link on this slide now said that one of the great
[41:28] things about llama index is that we help you go from prototype to production uh and that is where llama deploy comes in this is a framework that can take workflows built in llama index and automatically turn them into microservices each workflow can be scaled and managed independently with built-in retries and other features you need to make a distributed system work well llama deploy works by spinning up a control plane which manages State between workflows and acts as an orchestrator IT manages a message bus which the agentic microservices communicate with passing messages back
[42:00] and forth between agents we are eager to see what people build with llama deploy and you can check it out at the GitHub repo in this link uh so that's the lightning tour of llama index llama index and the things that it can do for you and how it lets you build complex uh powerful identic systems so now let's pass it back to Ricardo uh who will show us a demo of some of these pieces in action thank you Lori appreciate your comprehensive explanation about yma index uh so let's resume our
[42:36] presentation so right now what I'm going to do is uh what we call put everything together right so I think that's a perfect time for us like take that deep breath and say wow that was a spin right like a a lots of information about Amazon badrock amazingly presented by Anthony Prasad and lots of cool information about how the framework LMO index work also amazingly presented by Lori so I think this is all interesting but nothing replace that experience of seeing everything together in a real demo so that's what I'm going to do right now so Lori Anthony and I we've
[43:10] working together to create this demo which is essentially a chatbot that uses all these ingredients that we have discussed so far so it uses Amazon badrock as the LM right uh specifically uh a a12m labs for for the Foundation model and then we're constructing this agent using Lima index so you get to see the code and I'm going to explain how it works so what the code does is we're upload a PDF document about this uh it's about this called chavy Colorado right from it's actually out dat it's 2022 so
[43:44] but it doesn't matter for the sake of what we are going to present here today and then this agent this bot is going to allow us to uh ask questions about this pedf and then we're going to see all the components like uh working together so I'm going to actually switch my screen here to the notebook and don't worry because in the end of the presentation we're going to make this notebook the code sample that holds this notebook available for you so if you can try yourself if you want it but as you can see here this architecture is
[44:16] essentially uh the materialization of we have discussed so far so we're GNA create this ingeson pipeline using yma Index right I'm going to explain the code in in a in a minute but as you can see here this uh red uh icons here Radice is being used throughout the entire architecture as this kind of a glue right this this there's two types of glue in this architecture so bear with me the first one it is the runtime glue so as the messages and the embeddings and the the the orchestration executes in runtime
[44:50] rad is is taking care of making sure that all the components right the messages are flowing through right so that's the first type of glue the second type of glue is yma index but is more in a code level so this is what you're going to use to tie everything together so the flow is we're going to upload this to liac cloud and then we're going to process this using this inges pipeline we're going to programmatically create and then it's going to create a set of embeddings that in turn is going to be started at Radice and then our agent is going to use those uh embeddings to enrich the questions that
[45:26] are being the answers that are being provided along with the llm and then this is what the user is going to receive right which is the typical workflow for agents but later in the Dem we're going to also show how the santic cache works so we can get to see about a question being asked and a similar question being also asked and the uh delay of how much it takes to respond the question going to be much much faster right so in this notebook right I've already executed a set of the steps the installation of the dependencies because I mean yeah it's very trivial uh
[46:01] the creation of the darker container that is a cutes Rus this is actually very interesting for you to know because um in this notebook you can actually spin up very quickly a new Reds container with red Stacks that contains the compo all the components that required for this architecture so right now the container is running um if you are going to try yourself this notebook mind that uh you're going to have to provide two AP II Keys the first one is for dma Index right for dma cloud you're you create an account there you can create an account for free and then you
[46:33] create an API key for yourself and you're going to open this envy. Json file where you're going to set your API key there so that's your first API key the second one is surprise surprise from AWS so in order for you to interact with the Amazon Bedrock as a service you have to provide an API key and that's important an API key that is associated with an account that has ACC uh Direct access rice to Amazon Bedrock right so this is the heads up number one heads up number two and this is more like a Amazon Bedrock thing uh not only by default when you create an account AWS
[47:09] not only foundation models are enabled by default there are 100% available for you to use but they're simply not enabled so you have to in the AWS console you have to manually request access to these models for you to use and then well all of them will be available to you right so keep that in mind and and the AWS documentation everything is fully described so uh if you are in doubt you can always go there uh this is the document that we're going to use so uh this is the part of the notebook that downloads this uh PDF here locally is already downloaded and then
[47:42] the last step that I've already executed is this so this is the first step for you to understand about how yma uh yma index works as a framework so what we have done here is using the built-in constructs from yma to par the PDF documents and create what we call documents right which is like a series or list of chunks or regions of the documents so each one of those documents represent a portion of the PDF document so they're all available here in this component called documents so what we're going to start doing together
[48:15] and that's the exciting part right it is the construction of our index at radis that is going to store and obviously as a name implies is going to index all those embeddings right so the first thing you got to do there's this two constructs that you have to be aware of and as you can see uh it all uses the yma index API and SDK the first one you have to instantiate an embedding model right so there's a bunch of uh this is as an abstract class in Python so you have a different implementations of
[48:47] embedding models one of them being Bedrock embedding right which in this case we're going we're going to use Amazon Titan and then the second one you going to actually Define the schema at radus right uh where you are going to dictate like a different information about the the the feuds that are going to be associated with this index and more importantly the type of vector that we're going to use like so for example in this case we are using a flat Vector with uh 1536 Dimensions right and then the the algorithm that we're going to
[49:23] use to calculate the distance matric which is the how close the embedding is to the question that is been asked is going to be the algorithm called cuine right so I'm going to execute this and this is going to be the prerequisits to start impling this right so this is the pipeline let me execute before I forget this is the pipeline that uses uh yma index for us to create so uh I'm going to skip the import part right and then basically as you can see here it is a it's a very elegant API that allows you to construct this injection pipeline so
[49:59] it's very at least for for for my taste as a developer is very clear to me so you are creating a new injesting pipeline where you are providing almost as a Constructor actually not almost as a construct this is is a constructure in Python language for what is going to be the Transformations which is an array of Transformations you can have set a lot of them uh what's going to be the do doc store or docent store because as you store your in badings at r which is going to be your facts you optionally can use radus to also store your Dogman so oh Ricardo why I'm going to do this
[50:34] because guess what you can allow now not uh multi- types of searches you can search embeddings and vectors and you can search the actual document the full text search for whatever is in the PDF document so obviously this is an optional step but it is always good idea for you to kind of use the same data store for all of them and as similarly this is going to be what the vector store they're going to use and then option you can also use rad is in this case for the cache right so R why cash because in runtime when Lima index does
[51:07] what it's supposed to do which is the orchestration parts which is the communication between the layers there has been a lot of like multi-stage in between let's call state that the framework has to materialize and periodically flush to maintain the consistence of the execution so this is what we call the G in Cache right for and and the document strategies is basically uh how the documents are going to be created like if there's one existing we can up insert then which is uh instead of create a new one with a new version just up update with the
[51:40] existing one and then here's the interesting part like I'm going to actually uh bring up radis Insight uh for those of you that never uh heard about radis Insight this is a UI that radis provides for free for you to actually interact with your radis database and there's a bunch of powerful features here but right now as you can see here this is a brand new container running radish so there's absolutely no keys start so what I'm going to do I'm going to click here on this execution of the pipeline so we're going to provide the list of documents that has been
[52:14] chunked by Li index and then we're going to execute in this pipeline that we've just defined here so what we're going to notice here is that uh a set of embeddings are going to be created right so this is are the vectors that represents the edance created and if you go back to Red Insight you going to actually see all those let's call chunks actually this is actually the proper term that actually is a chunk is a portion of your document that is it currently state in a hash at Vector right and you can also see the vector uh
[52:48] obviously you're not going to be able to actually see this is a a form of expression here but this is actually the vectorization representation of that chunk right so this is actually a good practice a lot of people ask Ricardo do I have to create multiple chunks for a specific one doc or maybe one huge chunk I mean you can it's more about you shouldn't like it's better if you can like break it down into smaller chunks this going your searches and uh is going to perform faster and more accurately right so it's mostly a trade-off between
[53:23] performance and uh storage complexity right want a huge single line on your Radice or you want multiple ones so this is uh more like in side of a best practices for you to understand now just recap what we have so far we have radis up and running rad is already containing all the chunks from your PDF document so everything is already here along with some of the uh there's a a hash here that also stored the references of all the chunks as a document not not as a vector so we have everything we need
[53:57] to start working with your agent right so what we're going to do now is actually start the creation of the agent so this is the building block that you're going to use first of all you have to instantiate an llm right so as just like the the model that we used before there's different implementation support implementation of llm supported by Li index one of them is badrock so in this case we are using AI 21 labs and then we are also using instantiate and chat storage and chat memory remember chat memory so we can keep the context
[54:29] of the conversation ongoing so the llm doesn't forget what have been discussed before so this is what we call chat memory uh and the chat starts as then PL it's going to be where we're going to store our chat and then with that done this is the creation of the document itself so we're created this function called build doc agent going to provide a document as a parameter and this is going to be the creation of the agent I'm going to run through the entire code you can do this on your own time when you see the sample right let's actually try this out so I'm going to provide the very first document of that array of
[55:04] chunks that we've created so we can instantiate the agent itself so is here and a doc summary so if we look to the doc summary um do summary so you you can understand like we provide the description here provide information about the specs and attributes of CHF Colorado right so it basically is explaining you what is the nature of that document so now let's start using the agent right so we have here a set of questions that are related to the very own nature of the docent so the first
[55:36] one is what is the seating capacity of the vehicle right so if we execute this part this is the agent already interacting with the llm and it's being augmented with facts from the PDF so in this case the extended cab has two seats and the crew crab has five seats so this is information that is coming from the PDF so what is the towing capacity right is another question so if we execute this we're going to have the llm correctly responding that it is 3500 pounds of uh for the two in capacity
[56:09] right this is another information if you do like a common a or control F on the PDF document you're going to see this information there right um and then you can always check the history of the shf so far so all the questions and answers that has been in this interaction between the human and the machine is always started on our shat and just to wrap up uh let's talk about the santic cach right I'm going to create one here and as you can see here santic cach is another construct or building block that uh supported natively Alma index so yet another reason for you to start using Li
[56:44] Enix in your implementations and then what we're doing here is basically uh uh in the in the caching and descripted system word we call this cach aside uh pattern right so what what is Cash aside is when you start first investigating if the backend system has the information you wanted and if it doesn't you basically go to the back end provides the answer and cash it right so the next time you go it you can basically use the cash for this right this is all good and and stuff but the main problem of the nature of llms is that you don't want to
[57:19] provide an answer just for the sake of responding faster if the answer is incorrect right so for example I'm going to execute here this question how many doors the truck have right so the answer here I mean it's it's currently answer here uh okay doesn't matter the point the point really matter is that it took 7 Seconds to execute this now if we execute the same question again like how many doors does the truck have and how many passenger doors doors the truck have you're going to see that the same
[57:53] answer is going to be provided is the wrong answer no mod about this probably some something's going on with the back end system here but the point is the the very own question was associated with the previous one so if you execute continuously this now it's hitting the cache instead of going to the llm so uh in the real world what we going to avoid is this like a very huge delay to go into the llm because again it is a distrupted system right so this is for you to know understand uh what's going to be like the the the idea of the santic cash right and then as you go to
[58:28] this uh notebook yourself don't forget to kind of a stop the container that is running Docker right because you created one in the beginning of the notebook I highly encourage you to actually try this notebook yourself and with that said let's wrap up what we have said so far so if you want to create Rag architectures and make this red entire implementation and execution to be fast Radice is the component that you can use for this right this is our rad and bu
[59:01] that's what we do best like how making things fast right and Amazon badrock is this amazing piece of architecture that provides you with a different Foundation models and as pad presented in the beginning right it has a complete integration the set of features that is going to be very handy when you were implying Solutions like this and as the glue for the developers to create your own orchestrations both and the development level and the production level yemma index is the component that ties everything together so as I promise you can try this demo yourself I would
[59:35] highly encourage you to take a screenshot of this barcode if you scan this barcode is going to bring you exactly to the uh GitHub reposer that has that notebook so we can try the demo yourself hopefully the sented cach part is going to work for you uh and not have the glitch that I've had here and this is more like a personal request if you have some time after this webinar please scan this barcode and give us your feedback how we did go today right um so we provide you a lot of information I know that's kind of a hard to digest but
[01:00:08] your feedback is very important for us to understand what we can improve what can do better and also to kind of uh uh share your experiences with us as well so with that said Thank you on behalf of all my friends Anthony prad Lori you guys are all amazing and I'm looking forward to see you back in another webinar very soon thank you cheers guys

Full Transcript​

Timestamped Segments​

Full Transcript

Timestamped Segments