autoagent

AutoAgent, from Third Layer, is an open-source framework for autonomous harness engineering: rather than hand-editing an AI agent's code, you let a meta-agent (your coding agent) build and iterate on the agent harness, following the same keep-or-discard experiment loop popularized by autoresearch. How it works: - The harness under test lives in a single file, agent.py, containing configuration, tool definitions, an agent registry, routing/orchestration, and a Harbor adapter boundary. The adapter section is explicitly marked as fixed; the rest is the meta-agent's primary edit surface. - The human does not edit the Python. Instead you edit program.md, a Markdown file that provides the meta-agent's instructions plus the directive describing what kind of agent to build. - Evaluation tasks live in tasks/ using the Harbor task format: task.toml config, instruction.md prompt, tests (test.sh/test.py, deterministic or LLM-as-judge) that write a 0.0-1.0 score, a per-task Dockerfile based on the base image, and mounted reference files. The repository ships without tasks; you add your own, and benchmark payloads may appear in benchmark-specific branches. - The loop: the meta-agent reads the directive, inspects the current harness, runs the benchmark, diagnoses failures, modifies agent.py, checks the total score produced by the task test suites, and keeps or discards the change - hill-climbing on the score. Experiments are logged to results.tsv, and Harbor job outputs go to jobs/. Requirements and usage: Docker, Python 3.10+, uv, and whatever model-provider credentials your harness needs (e.g., OPENAI_API_KEY in .env). You build the base image with Dockerfile.base, add tasks, then run a single task or all tasks in parallel through the harbor CLI with the agent import path agent:AutoAgent. To kick off an experiment, you point your coding agent at the repo and prompt it to read program.md. Design choices stated in the README: program the meta-agent rather than the harness directly; keep the harness single-file but registry-driven so it can still evolve cleanly; run the agent in Docker isolation so it cannot damage the host; make every experiment score-driven; and use Harbor-compatible tasks so the same harness can be evaluated on different datasets. The README also suggests equipping the agent with context-engineering skills (such as Agent Skills for Context Engineering and context7) to improve performance, and provides cleanup commands for Docker images, containers, and Harbor's task cache, which accumulate across runs. Licensed under MIT. The maintainers note an upcoming product around self-configuring agents and that they are hiring.