evals

OpenAI Evals is a framework designed for evaluating large language models (LLMs) and systems built on them. It provides an existing registry of evals to test various dimensions of OpenAI models, along with the ability to write custom evals for specific use cases. Users can also build private evals using their own data to represent common LLM patterns without exposing data publicly. Setup requires an OpenAI API key, specified via the OPENAI_API_KEY environment variable. The evals registry is stored using Git-LFS, and users can fetch all or select eval data. For creating evals, cloning the repo and installing with `pip install -e .` is recommended, with optional formatters for pre-commit hooks. Running evals can be done via `pip install evals`, with full instructions in run-evals.md and eval-templates.md. Advanced use cases like prompt chains or tool-using agents are supported through the Completion Function Protocol. Results can optionally be logged to a Snowflake database. Writing evals is guided by build-eval.md, custom-eval.md, and completion-fns.md, with examples in the examples folder. Currently, custom code evals are not accepted, but model-graded evals with custom YAML files are welcome. Contributions are reviewed by OpenAI staff for model improvements. The FAQ addresses common questions, including examples of building evals from start to finish, multiple implementations (e.g., coqa.yaml), and a known issue where evals may hang at the end (safe to interrupt). Non-coders can contribute by following existing eval templates with JSON data and YAML parameters. Contributors agree to share evaluation logic and data under the MIT license, with OpenAI reserving rights to use data for service improvements, subject to usage policies.