ltm100

LTM100 is an end-to-end multi-user load benchmark designed for Long-Term Memory (LTM) systems. It drives virtual users through realistic memory workloads and reports client-observable throughput, latency, rejection, and error metrics. The tool explicitly measures load, concurrency, and scalability behavior rather than retrieval or answer quality—it does not compute precision, recall, or MRR. Key capabilities include: - Closed and open load models for fixed-concurrency and arrival-driven tests. - Four scenarios: chat-replay (replay chatbot recall and conversation ingestion), add-load (pure memory-ingest throughput), search-load (search throughput and latency), and mixed (configurable add/search mixture). - Weighted chat workload profiles with group-specific recall, timing, and closed/open session controls. - Bounded admission and rejection accounting for overload experiments. - Multi-process load generation to scale beyond a single event-loop core. - Optional pre-ingest, ramp-up, raw request records, fixed-interval E2E time series, and server-side latency metrics. - Repeated memory-growth sweeps with fixed query workloads and isolated corpus points. - Pluggable datasets and backend adapters. Supported datasets include LongMemEval (local JSON or Hugging Face loading with structured user/assistant turns for chat-replay) and a bundled synthetic dataset (deterministic, download-free). Supported backends include MemMachine REST, MemMachine MCP with REST lifecycle management, and Mem0 OSS REST. Reports are output as JSON and CSV, with successful throughput and latency reported separately from backend errors and queue rejections to prevent overload from inflating QPS or lowering service-latency percentiles. Additional output options include per-request raw NDJSON logs, fixed-interval E2E time series, and server-side latency breakdowns when the adapter supports them. LTM100 requires Python 3.10 or later and is installed via pip. The project is licensed under Apache License 2.0, with the current release being v0.4.3.