Transcript: Run AI in the browser - faster, cheaper, and private

Version 1: Source Transcript

This document represents the raw lecture transcript recorded from the video lecture. For architectural study notes and system design documentation, see Version 2.

• Video URL: https://www.youtube.com/watch?v=nBaGhHipWb0
• Video ID: nBaGhHipWb0
• Word Count: 1,236 words

Full Transcript

Everybody knows that AI is a cloud thing. Some of us know that you can run it locally, but did you know you can run it in the browser? The browser? Are you crazy? Well, yes, but that's beside the point. Browsers are everywhere. There's a good chance you're using one right now, and it's only natural to want to do the cool new AI stuff right there in the browser where you're already at. Because of this, all sorts of tools have popped up to let developers just like you do just that. Transformers.js is the one I've used the most. It's from hugging face and lets you do all the things that transformers are great at. Things like creating embeddings, jerating text, and image classification. It runs in node or in a web browser. It's super easy to use as it builds the entire pipeline for you. All you need to do is provide the text or the image, and it does the rest. Web LLM, as the name suggests, lets you run a generous selection of LLMs within the browser. The thing that's particularly neat about Web LLM is that it has an API that is compatible with open AIS. So you can use Langchain in the browser. It doesn't support everything that Langchain does, for example, tool use, but it's still really cool. Onyx runtime from Microsoft has a web version called, crazily enough, Onyx runtime web. If you're not familiar with Onyx, it's an open model standard that almost any model can be converted to. Onyx runtime can use these converted models on the platform of your choice. Onyx runtime web uses these converted models and of course a web browser. This lets you run all sorts of models from modern transformers to old school scikitlearn models all from JavaScript. Onyx runtime web is a bit of a white box experience compared to transformers.js and web llm, but it's a powerful tool that gives you tons of control. Behind the scenes, these tools use new browser capabilities like web GPU, emerging ones like web neural network, and more established ones like Web Assembly. Links to all these tools are in the show notes below, of course, and I even have a sample repo written with help from Claude Code if you want to try them out for yourself. The advantages to running AI in a browser are manifold for the user. It provides a snappier experience. No need to wait for server round trips when the AI is local, and it provides better privacy as the data the model operates on never leaves the browser. It can even work offline if the application is a progressive web app or something like that. It's great for the service provider as well and can save a fair bit of cash. By running the models on the user's computer, you don't have to pay for compute, bandwidth, or APIs beyond the basic functionality of your app. And it scales great since all you're doing is hosting the content. Honestly, this sounds like a veritable tech panacea. Why wouldn't you use this for everything? What could possibly go wrong? >> It's the stay puffed marshmallow man. >> Okay, there are some drawbacks to using models in the browser. First off, the typical web user has a lot less RAM and many fewer GPUs than say OpenAI or Anthropic do. So, the models that can be run need to be a bit smaller to accommodate that. And smaller models usually aren't as smart. So, the capability can be limited. The capabilities of a user's machine can also be very, very inconsistent. Some users will be hardcore gamers and have 128 gigs of RAM and a $3,000 graphics card. Others will have a hand-me-down laptop with an i5, 8 gigs, and no GPU whatsoever. And even the small models are kind of big in terms of file size. A small model could be measured in gigabytes on the high end. Downloading hundreds of megabytes during page load often isn't practical, even if the browser caches the model. And browsers themselves have memory and storage limits that can get in the way. And if you're running all this from a mobile device, these problems get worse with data limits and even smaller devices. So, is running AI in the browser really that great of an idea? Or have we run into the classic gotcha that developers and engineers so often find themselves in? You know, the one. Yeah, but your scientists were so preoccupied with whether or not they could, they didn't stop to think they should. >> Can't say I've never done that. But in this particular case, I don't think that's fair. There are lots of things you can do with small bottles that are still useful. They just need to be small things. Things like object detection. Is there a person in this photo? Sentiment analysis. Does this text message I'm about to send come off a little too harsh? Background removal in images or even video. Nobody needs to know that I'm at the bar during my Zoom call. Basic classifications of image or text so that your code can send the data off to the right service. Named entity extraction from text might be handy to find the names of people, places, and things in text so you can provide more context to the user or the software. And good old-fashioned machine learning of the psyche learn variety. Useful for all sorts of things beyond classifying irises and calculating Titanic survivability. These are all useful and I could end the video here and you'd have gotten some value. But as Billy May says, wait, there's more. These smaller browserbased models excel when part of a larger AI powered application. Use the server in the cloud to do the heavy lifting and use the browser to do the lighterwe stuff, making the user's experience nice and snappy. This can also have a nice effect on privacy. You don't have to worry about the text of that emotionally charged email or text message being sent to the server for sentiment analysis. Instead, it happens privately in your browser. And it saves cash for the service provider, plus money is spent on APIs and compute. The user's computer is free after all and bandwidth, the processed data, is usually smaller. A great example of this is semantic search. You could upload your image or your text to a server. The server could then create an embedding and use that embedding to search a vector database like Reddus and then return the results. Or you could create the embedding in the browser, upload the embedding, and then do the search. Instead of sharing your image or your text, you just share the embedding. The data itself never leaves the browser. You can search without revealing what you searched for. Privacy check and embedding is going to be a couple to a few kilobytes max. Images, they're usually megabytes. The upload is faster and uses less bandwidth. Performance check. So, that's pretty much what I've got. What started out as a crazy idea has actually turned out to be pretty useful. There are tons of wonderful tools available for browser based AI. Pair these with the tools you're already using and go out and make some awesome software.

Timestamped Segments

• [00:00] Everybody knows that AI is a cloud thing. Some of us know that you can run it locally, but did you know you can run it in the browser? The browser? Are you crazy? Well, yes, but that's beside the point. Browsers are everywhere. There's a good chance you're using one right now, and it's only natural to want to do the cool new AI stuff right there in the browser where you're already at. Because of this, all sorts of tools have popped up to let developers just like you do just that. Transformers.js is the one I've used the most. It's from hugging face
• [00:32] and lets you do all the things that transformers are great at. Things like creating embeddings, jerating text, and image classification. It runs in node or in a web browser. It's super easy to use as it builds the entire pipeline for you. All you need to do is provide the text or the image, and it does the rest. Web LLM, as the name suggests, lets you run a generous selection of LLMs within the browser. The thing that's particularly neat about Web LLM is that it has an API that is compatible with open AIS. So you can use Langchain in the browser. It doesn't support everything that Langchain does, for
• [01:04] example, tool use, but it's still really cool. Onyx runtime from Microsoft has a web version called, crazily enough, Onyx runtime web. If you're not familiar with Onyx, it's an open model standard that almost any model can be converted to. Onyx runtime can use these converted models on the platform of your choice.
• [01:21] Onyx runtime web uses these converted models and of course a web browser. This lets you run all sorts of models from modern transformers to old school scikitlearn models all from JavaScript. Onyx runtime web is a bit of a white box experience compared to transformers.js and web llm, but it's a powerful tool that gives you tons of control. Behind the scenes, these tools use new browser capabilities like web GPU, emerging ones like web neural network, and more established ones like Web Assembly.
• [01:47] Links to all these tools are in the show notes below, of course, and I even have a sample repo written with help from Claude Code if you want to try them out for yourself. The advantages to running AI in a browser are manifold for the user. It provides a snappier experience. No need to wait for server round trips when the AI is local, and it provides better privacy as the data the model operates on never leaves the browser. It can even work offline if the application is a progressive web app or something like that. It's great for the service provider as well and can save a fair bit of cash. By running the models on the user's computer, you don't have to pay for compute, bandwidth, or APIs beyond the basic functionality of your app. And
• [02:21] it scales great since all you're doing is hosting the content. Honestly, this sounds like a veritable tech panacea. Why wouldn't you use this for everything? What could possibly go wrong? >> It's the stay puffed marshmallow man. >> Okay, there are some drawbacks to using models in the browser.
• [02:42] First off, the typical web user has a lot less RAM and many fewer GPUs than say OpenAI or Anthropic do. So, the models that can be run need to be a bit smaller to accommodate that. And smaller models usually aren't as smart. So, the capability can be limited. The capabilities of a user's machine can also be very, very inconsistent. Some users will be hardcore gamers and have 128 gigs of RAM and a $3,000 graphics card. Others will have a hand-me-down laptop with an i5, 8 gigs, and no GPU whatsoever. And even the small models are kind of big in terms of file size. A
• [03:15] small model could be measured in gigabytes on the high end. Downloading hundreds of megabytes during page load often isn't practical, even if the browser caches the model. And browsers themselves have memory and storage limits that can get in the way. And if you're running all this from a mobile device, these problems get worse with data limits and even smaller devices. So, is running AI in the browser really that great of an idea? Or have we run into the classic gotcha that developers and engineers so often find themselves in? You know, the one. Yeah, but your scientists were so preoccupied with whether or not they could, they didn't stop to think they should.
• [03:45] >> Can't say I've never done that. But in this particular case, I don't think that's fair. There are lots of things you can do with small bottles that are still useful. They just need to be small things. Things like object detection. Is there a person in this photo? Sentiment analysis. Does this text message I'm about to send come off a little too harsh? Background removal in images or even video. Nobody needs to know that I'm at the bar during my Zoom call.
• [04:09] Basic classifications of image or text so that your code can send the data off to the right service. Named entity extraction from text might be handy to find the names of people, places, and things in text so you can provide more context to the user or the software. And good old-fashioned machine learning of the psyche learn variety. Useful for all sorts of things beyond classifying irises and calculating Titanic survivability. These are all useful and I could end the video here and you'd have gotten some value. But as Billy May says, wait, there's more.
• [04:39] These smaller browserbased models excel when part of a larger AI powered application. Use the server in the cloud to do the heavy lifting and use the browser to do the lighterwe stuff, making the user's experience nice and snappy. This can also have a nice effect on privacy. You don't have to worry about the text of that emotionally charged email or text message being sent to the server for sentiment analysis.
• [04:58] Instead, it happens privately in your browser. And it saves cash for the service provider, plus money is spent on APIs and compute. The user's computer is free after all and bandwidth, the processed data, is usually smaller. A great example of this is semantic search. You could upload your image or your text to a server. The server could then create an embedding and use that embedding to search a vector database like Reddus and then return the results.
• [05:21] Or you could create the embedding in the browser, upload the embedding, and then do the search. Instead of sharing your image or your text, you just share the embedding. The data itself never leaves the browser. You can search without revealing what you searched for. Privacy check and embedding is going to be a couple to a few kilobytes max. Images, they're usually megabytes. The upload is faster and uses less bandwidth.
• [05:43] Performance check. So, that's pretty much what I've got. What started out as a crazy idea has actually turned out to be pretty useful. There are tons of wonderful tools available for browser based AI. Pair these with the tools you're already using and go out and make some awesome software.