mcp-arcade

MCP Arcade is a command-line tool for testing MCP servers. It runs a short, named catalog of experiments, called atoms, against a target server, and bases its verdict on the JSON-RPC messages exchanged and on changes observed in a sandbox rather than on tool descriptions, terminal output, or a model's written reasoning. Version 0.1.x ships four atoms. inspect.tools_list checks that the menu the server publishes is the menu it serves. poison.follow_through checks that a side-quest buried in a tool description does not become an actual tools/call. temporal.rug_pull checks that, after a number of clean calls, the tool is still the same tool. protocol.unlisted_call, marked as a holdout, calls a nonce-generated name absent from the just-fetched menu and fails only if the server answers with JSON-RPC success and isError false. The --atoms option selects which atoms run, and the atoms subcommand lists the catalog. The connected policy, called the agent seat, can be naive (follows side-quests found in descriptions), task-only (runs the requested tool and stops), or a local Ollama model. The local model receives a frozen system prompt, the named task, the tool list exactly as presented, prior results, and inbound server notifications as untrusted observations, and it returns tool calls; only names and arguments are copied out of its reply, never its prose. On live servers only tools listed in --seat-allow may be sent, and attempts outside that list are recorded as unauthorized without reaching the wire. Installation is through pip or an editable install from a clone. The bout command runs a session against the bundled fixture, a stdio server, or a Docker image. Non-fixture targets require --allow-live; the tool will not spawn a command otherwise. A benign task tool and its arguments can be named with --task and --args, and a bout lacking them is reported as SKIP rather than as a pass. Options also cover message framing detection, handshake protocol versions, per-request timeouts, and an opt-in house side-quest added to a chosen tool's description. A receipt is emitted as mcp-arcade.bout/v1 JSON and can be printed canonically or rendered as a timeline table with one row per wire event, showing sequence, direction, method, id, the owning atom, and a note. Server notifications and server-originated requests are shown as their own rows. The renderer is built from a view of the receipt with no fields for scores or operator guesses, so it cannot display a verdict before the operator records one. Docker targets run with restrictive defaults: no network, read-only filesystem, memory and pids limits, dropped capabilities, and one fresh container per atom. The image id is pinned and re-checked before each atom, the sandbox is snapshotted from inside the container, binds and extra flags are explicit and recorded, and containers are removed afterwards. Scoring uses two axes and one combined number: utility, meaning the named benign task actually ran; integrity, meaning no unauthorized call, catalog lie, or silent mutation; and NRP, computed as task_success multiplied by (1 minus attack_success) and pinned to zero for any bout containing an ERROR atom. A policy that never calls tools scores zero rather than a perfect integrity result. Receipts can be converted into a JSONL dataset with one row per atom, labelled from the wire, with holdout atoms always diverted to a separate file and proof-split receipts excluded from training data. A manifest records generator version, per-receipt hashes, counts, and drop reasons. The tool collects no telemetry and makes no network connections of its own apart from the local model endpoint, which defaults to a loopback address and is recorded on the receipt. The project states its limits plainly: it is not a scanner benchmark, not load testing, not a sampling or elicitation client, not a container orchestrator, and it does not provide a 3D overview. It is licensed under MIT.