calibrax

Calibrax, short for Calibrate + JAX, is a research preview framework for the JAX scientific ML ecosystem, with core Tier 0 metrics validated against scikit-learn and SciPy references at 1e-6 tolerance. It adopts a 4-tier metric architecture covering 20 functional domains including regression, classification, calibration, segmentation, distance, divergence, information, ranking, statistical, clustering, fairness, forecasting, uncertainty, generative, image, text, audio, geometric, graph and manifold use cases. It provides a MetricRegistry for axiom-based metric discovery, a full geometric distance hierarchy, graph metrics, and supports metric composition, bootstrap confidence interval wrappers, classwise wrapping, metric tracking, and metric learning losses with hard/semi-hard negative mining. Its benchmarking and profiling capabilities include warm-up aware timing that separates JIT compilation, CPU/memory/GPU memory/clock/power tracking, energy and carbon footprint measurement, XLA-level FLOP counting, roofline performance analysis, XLA compilation tracing, algorithmic complexity analysis, and automatic hardware capability detection. It also includes built-in statistical analysis tools for bootstrap confidence intervals, hypothesis testing, effect sizes and outlier detection, direction-aware regression detection against stored baselines, cross-configuration comparison and Pareto front analysis, convergence validation, JSON-per-run file storage with baseline management, W&B and MLflow integration, publication-ready LaTeX/HTML/CSV table outputs and matplotlib plots, CI regression gate with git bisect automation, production monitoring alerts with configurable thresholds, and a full command-line interface. It supports optional dependency installation for GPU monitoring, image quality plugins, text quality plugins and publication export features, with multiple runnable Python scripts and Jupyter notebook examples covering usage levels from beginner to advanced.