ferrite

ferrite is a high-performance inference engine written in Rust, specifically designed for the GLM-5.3-Flash model. The engine leverages a hybrid architecture combining GatedDeltaNet linear attention, DSA sparse attention, and dense/MoE FFN layers. Its primary design goal is to address the runtime overhead associated with traditional serving frameworks by employing compile-time model specialization and static layer planning. Key architectural features include: 1. **PDAF Disaggregation**: Prefill, Decode, Attention, and FFN phases are treated as first-class citizens with a dedicated router and static operation graph, eliminating runtime dispatch overhead. 2. **Composable Parallelism**: The engine supports Tensor Parallelism (TP), Context Parallelism (CP), and Data-Parallel Context Parallelism (DCP) through a unified axis algebra system. This allows for flexible 3D mesh configurations (Q, Kv, and Head axes) that can be composed to optimize throughput and latency. 3. **Exact MHC and GatedDeltaNet**: It implements exact Multi-Head Connection (MHC) residuals and a WYF-parallel chunkwise GatedDeltaNet recurrence, verified against CPU golden standards. 4. **CUDA Optimization**: The engine includes custom CUDA kernels for sm_100a/sm_103a architectures, with support for CUDA graph capture and verification to ensure operation sequence stability during decoding. The project emphasizes numerical correctness, using a CPU backend as a golden standard for validating all other backends. It currently supports continuous batching, speculative decoding (MTP), and has achieved significant performance improvements on B300 GPUs, reaching 64.8 tok/s in decode mode with speculative execution enabled. The codebase is modular, with separate crates for types, model configuration, kernel backends, KV management, batching, scheduling, and execution.