garak

garak (Generative AI Red-teaming & Assessment Kit) is a command-line tool for probing large language models and dialog systems for failure modes. It is developed by NVIDIA and released under the Apache 2.0 license. What it does - Checks whether an LLM can be made to fail in undesirable ways, including hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks and other weaknesses. - Combines static, dynamic and adaptive probes to explore these failure modes. - The README compares its role to nmap or Metasploit Framework, but applied to LLMs. Supported targets - Hugging Face Hub generative models, including local transformers pipelines, the Inference API and private Inference Endpoints. - OpenAI chat and completion models. - Replicate text models and private Replicate endpoints. - AWS Bedrock foundation models via the Converse API, covering families such as Anthropic Claude, Meta Llama, Amazon Titan, AI21 Labs, Cohere and Mistral AI. - Cohere, Groq, litellm, NVIDIA NIM endpoints, ggml/llama.cpp models, and essentially anything reachable through a REST endpoint. - Built-in test generators (test.Blank, test.Repeat) for plugin development. Installation and usage - Installed from PyPI with pip, from the GitHub main branch, or cloned from source into a Conda environment (Python 3.11 to 3.13). - General syntax is `garak <options>`; the tool needs a target model and by default runs all known probes with each probe's recommended detectors. - `--list_probes` lists available probes; `--target_type` and `--target_name` select the model; `--spec` narrows the run to a probe family or a single plugin. - API keys for hosted providers are supplied through environment variables such as OPENAI_API_KEY, HF_INFERENCE_TOKEN, REPLICATE_API_TOKEN, COHERE_API_KEY, GROQ_API_KEY, NIM_API_KEY and BEDROCK_API_KEY. Probe examples described in the README - blank, atkgen (automated attack generation), badchars (imperceptible Unicode perturbations), av_spam_scanning, continuation, dan (DAN and DAN-like attacks), donotanswer, encoding (prompt injection via text encoding), gcg (adversarial suffix), glitch (glitch tokens), grandma, goodside, leakreplay (training data replay), lmrc (Language Model Risk Cards), malwaregen, misleading, packagehallucination, promptinject, realtoxicityprompts, snowball (snowballed hallucination), and xss. Results and logging - For each probe, garak prints a progress bar and then a row evaluating results per detector, marking responses as FAIL with a failure rate when undesirable behavior occurs. - Logs include garak.log for debugging, a per-run JSONL report with an entry per probing attempt, and a hit log detailing attempts that yielded a vulnerability. - A basic analysis script at analyse/analyse_log.py outputs the probes and prompts that produced the most hits. Architecture and extensibility - Code is organized into probes, detectors, evaluators, generators and harnesses, each with a base.py defining plugin base classes. - The default operating mode uses the probewise harness, which instantiates each probe and reads its primary_detector and extended_detectors attributes to decide which detectors to run. - Developers can write new plugins by inheriting from base classes such as garak.probes.base.TextProbe, testing them interactively or with test generators and detectors, and listing them via --list_probes, --list_detectors or --list_generators. Project resources - Documentation at docs.garak.ai and reference.garak.io/readthedocs, a Discord community, the garak.ai project home, an arXiv preprint (2406.11036) and a citation entry for academic use.