Skip to main content
Version: v1.0 (Raw Transcripts)

Transcript: What is a semantic cache？

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

Full Transcript​

See, when you're building with LLM's large language models, people don't always phrase things the same way. One person might ask, "How hot is Brazil in the summer?" And another might say, "What's the temperature of Brazil during summertime?" Same meaning, different wording. But if your system has to treat those as two separate queries every time, you're burning through time and element calls for no good reason. That's where semantic caching helps. It saves the meaning of past requests, not just the exact words. So if someone asks a new question that's semantically similar to one you've already answered, you can skip the LLM and return the cache answered right away. That makes your app faster and a lot cheaper to run. Well, under the hood, what happens is that the user query goes through an embedding model, a type of machine learning algorithm that turns unstructured data like text into a numerical representation called a vector. That vector is then compared to other vectors already stored in the cache, which is a vector database. In this case, if one is semantically close enough, that's called a cash hit and its answer is returned right away. If not, that's called a cache miss. The request goes to the LLM. The response gets stored alongside its vectorzed query in the cache and then it's returned. The next time a similar query is sent to application, the system won't have to send anything to the LLM anymore because it can just get the results straight from the semantic cache. This decreases application response times and also reduces costs. And as we know, LLM calls aren't that cheap. And Rady's makes this that simple because Radius isn't just a fast in memory database. It's also a vector database. That means you can store both the LLM responses and the vector representations of their prompts in Rady's. Having a database that allows you to cache data for precise lookups as well as semantic ones gives you a lot of flexibility. Reddis also gives you full control over expiration and eviction. You can set a time to leave for each cache the items so that all data automatically expires. And when memory gets tight, Reddit uses smart eviction policies like least recently used or least frequently used to make space without you having to worry. And here's the best part. According to our benchmarks, Reddis 8, the newest version of Reddis is the fastest vector database out there. It can scale to 1 billion vectors and still deliver low latency search. If you want to dive deeper into semantic caching or hands-on applied AI engineering, check out our Radius AI resources GitHub repository. It's packed with demos, recipes, and real world examples to get you building fast. My name is Rafael De Leo. Thank you for watching and as always, stay curious. [Music]

Timestamped Segments​

  • [00:00] See, when you're building with LLM's large language models, people don't always phrase things the same way. One person might ask, "How hot is Brazil in the summer?" And another might say, "What's the temperature of Brazil during summertime?" Same meaning, different wording. But if your system has to treat those as two separate queries every time, you're burning through time and element calls for no good reason. That's where semantic caching helps. It saves the meaning of past requests, not just the exact words. So if someone asks a new question that's semantically similar to one you've already answered, you can skip the LLM and return the cache
  • [00:33] answered right away. That makes your app faster and a lot cheaper to run. Well, under the hood, what happens is that the user query goes through an embedding model, a type of machine learning algorithm that turns unstructured data like text into a numerical representation called a vector. That vector is then compared to other vectors already stored in the cache, which is a vector database. In this case, if one is semantically close enough, that's called a cash hit and its answer is returned right away. If not, that's called a cache miss. The request goes to the LLM.
  • [01:05] The response gets stored alongside its vectorzed query in the cache and then it's returned. The next time a similar query is sent to application, the system won't have to send anything to the LLM anymore because it can just get the results straight from the semantic cache. This decreases application response times and also reduces costs.
  • [01:23] And as we know, LLM calls aren't that cheap. And Rady's makes this that simple because Radius isn't just a fast in memory database. It's also a vector database. That means you can store both the LLM responses and the vector representations of their prompts in Rady's. Having a database that allows you to cache data for precise lookups as well as semantic ones gives you a lot of flexibility. Reddis also gives you full control over expiration and eviction.
  • [01:49] You can set a time to leave for each cache the items so that all data automatically expires. And when memory gets tight, Reddit uses smart eviction policies like least recently used or least frequently used to make space without you having to worry. And here's the best part. According to our benchmarks, Reddis 8, the newest version of Reddis is the fastest vector database out there. It can scale to 1 billion vectors and still deliver low latency search. If you want to dive deeper into semantic caching or hands-on applied AI engineering, check out our Radius AI resources GitHub repository. It's packed
  • [02:24] with demos, recipes, and real world examples to get you building fast. My name is Rafael De Leo. Thank you for watching and as always, stay curious. [Music]