cospa

COSPA (Cost Of Solving Programming Assignments) is a benchmarking framework designed to evaluate the "cost-performance" of coding agents. It measures how much capability or quality a model provides per unit of cost by recording correct answers, token usage, elapsed time, and estimated API costs. The framework utilizes an official 336-task panel comprising six benchmark suites: - BigCodeBench-Hard Agentic: Python function implementation. - SWE-Explore Verified: Code location and relevance identification. - Multi-SWE-bench Flash: Multi-language repository repair. - Terminal-Bench Core: System administration and terminal tasks. - SWE-PolyBench Verified: Repository repair for Java, JS, Python, and TypeScript. - FeatureBench Lite: Feature implementation across various repositories. COSPA does not host models; it requires an OpenAI-compatible endpoint or a provider configured via the Pi coding agent. It includes a sophisticated host guard system to manage hardware resources (RAM, Disk, PSI) during concurrent trial executions to ensure stability. Key features include: - Support for multiple agent configurations (adapters) like pi_vanilla and jouzu. - Detailed result tracking including manifests, model traces, and grader verdicts. - A built-in viewer for analyzing scores, coverage, and confidence intervals via terminal or browser UI. - Rigorous task selection criteria to ensure reference fixes are valid and starter repositories are consistently failing.