deepeval

DeepEval is an open-source evaluation framework for large language model (LLM) systems, functioning similarly to Pytest but specialized for AI applications. It allows developers to evaluate LLM apps end-to-end as black boxes, analyze complete agent trajectories, or test individual steps such as tool use and retrieval. Key capabilities include: - A wide array of ready-to-use metrics: - Custom/All-Purpose: G-Eval and DAG. - Agentic: Task Completion, Tool Correctness, Goal Accuracy, and Plan Adherence. - RAG: Answer Relevancy, Faithfulness, Contextual Recall, and Contextual Precision. - Multi-Turn: Knowledge Retention, Conversation Completeness, and Role Adherence. - Multimodal: Text-to-Image, Image Editing, and Image Coherence. - Others: Hallucination, Summarization, Bias, Toxicity, and JSON Correctness. - Synthetic Dataset Generation: Ability to create single and multi-turn datasets for testing. - Benchmarking: Support for popular benchmarks like MMLU, HellaSwag, and HumanEval. - Prompt Optimization: Automatic prompt optimization based on evaluation results. - Broad Integrations: Works with frameworks such as LangChain, LangGraph, Pydantic AI, CrewAI, LlamaIndex, OpenAI, Anthropic, and Google ADK. The framework supports both local execution via NLP models and LLM-as-a-judge patterns, and it integrates into CI/CD environments for automated quality assurance.