langfuse

Langfuse is an open-source platform for LLM application observability and evaluation. It is designed to help teams collaboratively develop, monitor, evaluate and debug AI applications, and can be self-hosted in minutes. Core capabilities described in the README: - LLM application observability: instrument an app to ingest traces, tracking LLM calls plus related logic such as retrieval, embedding and agent actions; inspect and debug logs and user sessions. - Prompt management: centrally manage, version-control and collaboratively iterate on prompts, with server- and client-side caching intended to avoid added latency. - Evaluations: support for LLM-as-a-judge, code evaluators, user feedback collection, manual labeling and custom evaluation pipelines via APIs/SDKs. - Datasets: test sets and benchmarks for evaluating applications, supporting continuous improvement, pre-deployment testing and structured experiments, with integration to frameworks such as LangChain and LlamaIndex. - LLM Playground: test and iterate on prompts and model configurations, and jump from a trace to the playground. - API: OpenAPI spec, Postman collection and typed SDKs for Python and JS/TS, used to build custom LLMOps workflows. Deployment options include a managed cloud service and self-hosting via Docker Compose locally or on a VM, Kubernetes with Helm (described as the preferred production deployment), and Terraform templates for AWS, Azure and GCP. The project is built on ClickHouse. Integrations listed include SDKs for Python and JS/TS, OpenAI, LangChain, LlamaIndex, Haystack, LiteLLM, Vercel AI SDK and Mastra, plus packages and tools such as Instructor, DSPy, Mirascope, Ollama, Amazon Bedrock, AutoGen, Flowise, Langflow, Dify, OpenWebUI, Promptfoo, LobeChat, Vapi, Inferable, Gradio, Goose, smolagents and CrewAI. A quickstart shows installing the Python packages, setting environment variables for keys and base URL, and using the @observe() decorator with the OpenAI integration to log a first LLM call and view traces in the UI. The repository is MIT licensed except for the ee folders. Support is provided through documentation, FAQs, an Ask AI feature, GitHub Discussions and Issues. Contributions are welcomed via ideas, issues and pull requests.