ai-shipcheck

AI Shipcheck is a local-first production readiness scanner aimed at apps largely written by AI coding tools. Its premise: AI assistants are good at producing code that runs, less good at producing code that survives production, and they report the work as finished either way. Shipcheck looks for the recurring, boring gaps — a Supabase table without row-level security, a route handler that writes without checking the caller, a NEXT_PUBLIC_ variable holding a secret, an LLM endpoint with no rate limit or token cap. Usage is a single command: npx ai-shipcheck . No signup, no API key, no source upload. It requires Node.js 22 or newer. Options include --fail-on (for example critical), --min-score, --format (json, markdown, sarif), and an explain subcommand that documents why a rule exists and how to fix it. Findings alone never fail the command; thresholds are opt-in, so adding it to an existing pipeline is reversible. Exit codes distinguish thresholds met, not met, usage errors and internal errors. What it checks Nine categories are scored independently, backed by 63 rules, each with documentation, a vulnerable fixture, a secure fixture and tests: - Security: hardcoded credentials, secrets behind NEXT_PUBLIC_, eval, shell injection, open redirects, permissive CORS, disabled TLS verification, weak crypto. - Auth: write routes without authorization checks, unauthenticated server actions, browser-only privilege checks, unverified webhooks, unsigned JWTs, exposed service-role keys. - Database: tables without row-level security, USING (true) policies, SQL built by interpolation, unfiltered deletes, destructive migrations. - Reliability: swallowed errors, missing timeouts in request paths, unhandled rejections, retries without backoff, builds that ignore type errors. - Testing: no tests, CI missing test/build/typecheck, committed .only, server code with no referencing test. - Observability: no error monitoring, console-only server logging, no React error boundary. - Performance: unbounded queries, synchronous I/O in handlers, N+1 shapes, heavy client imports. - Accessibility: missing alt text, click handlers on non-interactive elements, unlabelled form controls, positive tabIndex. - AI cost: LLM endpoints without auth or rate limit, no token cap, request-controlled model selection, provider keys in the browser. Detected stacks include Next.js (both routers), React, Vite, Express, Fastify, Hono, NestJS, Remix, Astro, SvelteKit, Nuxt, Supabase, Firebase, Prisma, Drizzle, Mongoose, Stripe, OpenAI, Anthropic, Vercel AI SDK, LangChain, tRPC and common test runners. Framework-specific rules run only when the framework is detected, monorepos included. Output and integration The report shows a score, a verdict, per-category bars, and findings with file, line, rule, severity and confidence. A blocker forces NOT READY regardless of score. Categories that cannot be assessed are excluded rather than awarded a free 100. A GitHub Action annotates findings inline on the diff, writes a Markdown report to the job summary, produces SARIF for code scanning, and outputs score, verdict, critical-count and high-count; it is bundled into a single committed file so a tag-pinned workflow runs exactly that code. Trust and scope The tool makes no network calls and sends no telemetry; nothing in the scanned repository is executed — files are read as bytes and analysed lexically. Secrets are masked wherever they could be printed, scans are bounded and say so when truncated, and there is one runtime dependency. Stated limitations are explicit: JavaScript and TypeScript only; analysis is lexical rather than semantic, with no cross-file reasoning or type information and taint tracking that follows a value one hop, so a custom auth wrapper it does not recognise can produce a false positive. Express and Fastify routes are not covered by the auth rules, described as the largest known gap. Infrastructure is invisible to it, so a table with RLS enabled in a dashboard but absent from migrations is reported as unassessed, not safe. A clean report means the checks it knows how to make found nothing, not that the code is correct. The rules were validated against 20 public repositories pinned by commit SHA; triaging their output reportedly cut findings from 5,710 to 2,819 and surfaced a lexer bug affecting line numbers in files with multi-line comments. Documentation covers rules, CLI contract, configuration, scoring, trust model, limitations, threat model, architecture, adding a rule, releasing and governance. Licensed MIT.