textgen

TextGen is an open-source desktop application for running large language models locally. It is designed for privacy-conscious users who want a ChatGPT-like interface without sending data to external services: the project states it is 100% offline with zero telemetry, external resources, or remote update requests. Getting started is quick. Portable builds for Linux, Windows, and macOS are published as releases, with CUDA, Vulkan, ROCm, and CPU-only variants and all dependencies included. After unzipping and launching, a window opens. GGUF models from Hugging Face can be dropped into the user_data/models folder and are detected automatically. A full installation path is also documented for users who need additional backends, training, image generation, or extensions; it uses Miniforge/Conda or a one-click start script and requires roughly 10GB of disk space. Core capabilities include: - Chat and generation: instruct mode for instruction-following, plus chat and chat-instruct modes for custom characters, with prompts formatted through Jinja2 templates. A notebook tab supports free-form generation. - Multimodal input: images can be attached to messages for visual understanding, and text files, PDFs, and .docx documents can be uploaded for discussion. - Conversation management: edit messages, navigate between message versions, and branch conversations. - Multiple backends: llama.cpp, ik_llama.cpp, Transformers, ExLlamaV3, and TensorRT-LLM, with the ability to switch backends and models without restarting. - API compatibility: OpenAI/Anthropic-compatible Chat, Completions, and Messages endpoints with tool-calling support, positioned as a local drop-in replacement for those APIs. - Tool-calling: models can invoke custom functions such as web search, page fetching, and math; each tool is a single Python file, and MCP servers are supported. - Training: fine-tune LoRAs on multi-turn chat or raw text datasets, with resumable runs. - Image generation: a dedicated tab for diffusers models with 4-bit/8-bit quantization and a persistent gallery that stores image metadata. - Interface: dark/light themes, code syntax highlighting, LaTeX rendering, and built-in or community extensions including TTS, voice input, and translation. The project exposes an extensive command-line flag set covering model loaders, context and KV cache options, speculative decoding, GPU layer offloading, tensor splitting, quantization, sampling parameters, and server/API configuration. Docker Compose files are provided for NVIDIA, AMD, Intel, and CPU-only setups. TextGen is best understood as a local model runtime and chat front end rather than a hosted service. Its main value is combining a desktop UI, an OpenAI-compatible API, and model management in one offline package, making it suitable for users who want to experiment with or deploy local LLMs while keeping data on their own hardware.