fastchat

FastChat is an open-source platform for training, serving, and evaluating large language model (LLM) based chatbots. It is the release repository for Vicuna and the technology behind Chatbot Arena (lmarena.ai), which the README states has served over 10 million chat requests for 70+ LLMs and collected over 1.5M human votes for an online LLM Elo leaderboard. Core capabilities described in the README: - Training and evaluation code for models such as Vicuna and MT-Bench. - A distributed multi-model serving system with a web UI and OpenAI-compatible RESTful APIs. - Command-line inference supporting single GPU, multiple GPUs with model parallelism, CPU-only, Apple Metal (MPS), Intel XPU, and Ascend NPU backends. - Memory-saving options including 8-bit compression and CPU offloading, plus quantization support such as ExLlama V2, GPTQ 4-bit, and AWQ 4-bit. - A web GUI architecture with a controller, model workers, and a Gradio web server; multiple workers can be registered to one controller for higher throughput or multiple models. - A side-by-side battle UI (Chatbot Arena) that can be launched locally, with support for API-based models including OpenAI, Anthropic, Gemini, and Mistral. - OpenAI-compatible APIs usable with the openai-python library and cURL, plus Hugging Face generation APIs and LangChain integration. - Evaluation via MT-bench, a set of multi-turn open-ended questions, with strong LLMs like GPT-4 acting as judges. - Fine-tuning code based on Stanford Alpaca with multi-turn conversation support, LoRA support, and documented hyperparameters for Vicuna-13B. - Model weights for Vicuna (7B, 13B, 33B, including 16K context variants), LongChat, and FastChat-T5, downloadable from Hugging Face. - Support for a wide range of models including Llama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, and xDAN-AI. - Installation via pip or from source, with optional extras for model worker, web UI, and training. The README also notes datasets and reports released by the project, including LMSYS-Chat-1M and Chatbot Arena Conversations, and cites the paper "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena."