Show HN: I'm trying to make it easier to run local LLMs directly in the browser

3 months ago 2

TypeScript libraries that provide access to in-browser AI models with seamless fallback to using server-side models using the Vercel AI SDK.

Note

Only works with the new v5 of Vercel AI SDK.

@built-in-ai/core package is the AI SDK model provider for your Chrome and Edge browser's built-in AI models.
@built-in-ai/web-llm package is the AI SDK model provider for open-source models (using WebLLM) running directly in the browser.

We are also currently working on creating a model provider for transformers.js

# For Chrome/Edge built-in AI models npm i @built-in-ai/core # For open-source models via WebLLM npm i @built-in-ai/web-llm

Basic Usage with Chrome/Edge AI

import { streamText } from "ai"; import { builtInAI } from "@built-in-ai/core"; const result = streamText({ model: builtInAI(), messages: [{ role: "user", content: "Hello, how are you?" }], }); for await (const chunk of result.textStream) { console.log(chunk); }

import { streamText } from "ai"; import { webLLM } from "@built-in-ai/web-llm"; const result = streamText({ model: webLLM("Llama-3.2-3B-Instruct-q4f16_1-MLC"), messages: [{ role: "user", content: "Hello, how are you?" }], }); for await (const chunk of result.textStream) { console.log(chunk); }

For detailed documentation, browser requirements and advanced usage:

@built-in-ai/core documentation
@built-in-ai/web-llm documentation

Contributions are more than welcome! However, please make sure to check out the contribution guidelines before contributing.

If you've ever built apps with local language models, you're likely familiar with the challenges: creating custom hooks and UI components, while also building complex integration layers to fall back to server-side models when compatibility is an issue.

This library bridges this gap by providing a unified solution that lets you:

Experiment with in-browser AI models using familiar patterns
Seamlessly fall back to server-side models when needed
Use the same Vercel AI SDK eco system you already know
Avoid building complex integration layers from scratch

While there was an existing package that attempted to provide similar functionality for Chrome, it has been inactive for the past year with no maintenance or updates. We're grateful for their initial exploration and the foundation they provided for the community.

Read Entire Article