1Password’s new security research team says frontier models are still unreliable at producing clean vulnerability patches for complex, recent flaws.
Off-by-1 Labs tested 6,080 generated patches across six recently disclosed vulnerabilities in open source software. The study found that generated patches fully resolved the vulnerability without materially changing application behavior 26.0% of the time. Another 20.1% resolved the vulnerability but changed behavior. The remaining 53.9% did not resolve the vulnerability, added a new vulnerability, or both.
The target set included six recent vulnerabilities: a Linux privilege-escalation issue, ActiveMQ remote code execution, a Chrome File System Access API use-after-free on macOS, EXIM unauthenticated remote code execution, SpringAI SpEL remote code execution, and a Gemini CLI remote code execution advisory.
A patch that looks right can still be unsafe
The useful distinction in the study is not “AI can patch” versus “AI cannot patch.” It is that patch quality is multidimensional.
A security fix has to close the vulnerability, preserve expected application behavior, avoid opening a new vulnerability, and cover all affected code paths. A generated patch can look like maintainer code and still miss a duplicate vulnerable path, change an allow-list into a deny-list, reimplement a parser incorrectly, or introduce a new crash.
That is why the result matters for AI security products. The industry has spent the summer talking about frontier models finding vulnerabilities and producing fixes. Discovery is valuable, but remediation is the higher-risk step. Once a generated patch is merged, a defect moves from a model output into production software.
Human review is not optional housekeeping
1Password’s framing is direct: expert human review remains necessary. The review cannot be a quick style pass. It has to include regression tests, exploit validation, behavior comparison, duplicate-path checks, memory-safety review where relevant, and an assessment of whether the patch changed product semantics.
Help Net Security’s writeup adds a practical example from freenginx: a rejected AI-assisted patch and the maintainer’s own fix both missed or introduced problems, underscoring that patch review is hard even when humans are involved.
That does not make AI-generated patches useless. It makes them candidates. A candidate patch can save time if it points reviewers toward the right area, suggests tests, or produces a partial fix. It becomes dangerous when the organization treats plausible code as verified remediation.
