clousight-bench

Clousight Bench is an open-core benchmark runner and plugin SDK designed to measure cloud computing services. It emphasizes reproducibility and transparency by using unmodified upstream benchmark harnesses and maintaining a verifiable provenance chain for every measurement. The tool currently supports five product categories: - Agent Runtimes: SWE-bench - Data Warehouses: TPC-H, TPC-DS - Transactional Databases: TPC-C (via BenchBase) - Key-Value Stores: YCSB - LLM Endpoints: MMLU, GSM8K, HumanEval Key architectural features include a plugin-based system for BenchmarkSuites, Evaluators, DomainPacks, and ProviderAdapters. It supports various execution modes, including a 'mock' mode for local testing without cloud accounts and 'real' mode for live cloud environments (with experimental support for Aliyun AgentRun). The project includes a React-based web viewer for analyzing records and trajectories, a SQL-based query interface via DuckDB, and integration with pytest for CI/CD gating. It tracks multiple dimensions including latency, cost, and provisioning behavior alongside the primary benchmark verdict.