Skip to content
Blockchain Posts

Trail of Bits ships patching-bugs and regression-hunt agent plugins

Two open-source plugins out of Trail of Bits' Patch the Planet work: regression-hunt cuts false-positive regression claims 25% vs a baseline Codex prompt.

by 4 min read
researchtoolingtrail-of-bitsai-agentssecurity

Trail of Bits released two open-source plugins — patching-bugs and regression-hunt — for AI coding agents working on security patches. Both ship under the trailofbits/skills repository and are the engineering output of Patch the Planet, the firm's joint initiative with OpenAI. The post, by Anish Naik and Marcelo Morales, is dated October 21, 2026: Your AI agent fixed the bug, but what else did it change?.

The relevance to blockchain tooling is direct: audit automation, patch verification and regression tracking are the same primitives smart-contract review has been reaching for since the first LLM-assisted audit tools shipped.

The problem the plugins solve

An earlier Trail of Bits post reported regressions in some of their AI-written patches. The team then audited every merged upstream patch a Trail of Bits engineer had authored: of over 130 merged PRs, they confirmed 21 regressions — 16 functional, one performance, four build/test. None broke a security protection, but all were defects the agent did not catch before submission.

Their numbers on agent output quality:

- A single agent run produced 58 candidate patches
- ~36% precision (roughly 21 of 58 were real)
- Audit scope: >130 merged upstream PRs
- Regressions found: 21 (16 functional, 1 perf, 4 build/test)

The two plugins target the two failure modes those numbers expose: unverified fixes shipping (patching-bugs) and regressions missed on the way out (regression-hunt).

regression-hunt

Design points:

  • Diff traced through callers to the real interfaces the change touches, not just the files it edits.
  • Live reproduction: candidate regressions are replayed on a running instance of the project, not against a mock. The step is inconclusive if no live PoC can be produced.
  • Fresh-context replay: each PoC is re-run in an agent with no memory of the original diff, to avoid confirmation bias.
  • Adjudication step that puts the burden of proof on the regression claim.

Preliminary result on the 21 known regressions: 25% fewer false positives than a baseline Codex prompt asked to review a PR for regressions (95% CI: 14% to 38%). The tradeoff: some real regressions were left borderline because the evidence was insufficient to clear the adjudication bar.

patching-bugs

Three features matter:

  1. The three-arm proof. A patch must fail on original code, pass on patched code, and fail again when the patch is reverted. The third arm is the usual gap in agent patches: it rules out drift in the test harness or environment that would otherwise read as a fix.
  2. Invariant-guided repair. The agent targets the violated security property, not the single reported bug, so variant bugs on the same primitive get caught in the same patch.
  3. Mandatory regression-hunt pass. Every patch runs through regression-hunt before being surfaced.

What a project needs to use it

Two files in the repo, both Markdown:

  • THREAT_MODEL.md — assets, trust boundaries, supported configurations, and what counts as a vulnerability vs a regression for this project.
  • PATCHING_GUIDELINES.md (or AGENTS.md) — build and test steps, patch size, placement conventions, regression-test requirements.

The team benchmarked the plugins on GPAC, liboqs and validator.js. The methodology was derived from reviewing merged upstream patches and thousands of engineers' Codex sessions, with every claim required to produce a live proof of concept against a built, running instance.

Why this reads for blockchain builders

The pattern transposes cleanly:

  • Smart-contract repair. The "three-arm proof" is what [DeFi] bug-bounty engagements already do manually: reproduce exploit on original code, pass fix on patched code, revert and reproduce. Codifying it in agent output removes the trust fall on reviewer attention.
  • Invariant-guided repair. This is the formal verification playbook. Projects already running Certora, Halmos or Kontrol can define invariants the agent is told to satisfy and let the plugin iterate against them.
  • Regression hunting. Protocol upgrades that touch cross-module state are the regression-rich path; the live-reproduction constraint fits onchain devnets or anvil forks naturally.

Caveats, in their words

The post is explicit that the numbers are preliminary and the plugins are not finished. The 25% false-positive reduction is a point estimate on a small (n=21) known-regression set. The team flags next steps as:

  • Evaluating whether patches fix bugs and their variants without regressions.
  • Trace analysis of agent runs to see which rules Codex skips.
  • Configurability so maintainers can pick between minimal patches and broader regression hunts.

Context

The release lands against a quarter of audit-tool consolidation: see our earlier coverage of Trail of Bits' package-manager integrity work, the OpenZeppelin Chainlink standards partnership, and the Bitcoin Core policy change on LLM-found vulnerabilities earlier today. The through-line is that the agent step is now the review step; auditors are shipping the controls that make its output usable.

Sources:

Related stories