web-llm

WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU. It is fully compatible with the OpenAI API, supporting streaming, JSON-mode, function-calling (WIP), and more. Key features include in-browser inference via WebGPU, full OpenAI API compatibility, structured JSON generation, extensive model support (Llama 3, Phi 3, Gemma, Mistral, Qwen, and others), custom model integration in MLC format, plug-and-play integration via npm or CDN, streaming and real-time interactions, Web Worker and Service Worker support, and Chrome extension support. WebLLM supports multiple cache backends including browser Cache API, IndexedDB, Origin Private File System (OPFS), and an experimental cross-origin storage backend. It also provides optional integrity verification for model artifacts using SRI hashes. The project works as a companion to MLC LLM, enabling universal deployment of LLMs across hardware environments. Developers can use it as an npm package to build web applications, with examples available for chatbots, streaming, JSON mode, function calling, service workers, and Chrome extensions.