LACUNA is a new testbed for a hard LLM unlearning question: did the model actually remove the targeted knowledge from its weights, or did the unlearning method only make the model stop revealing it in ordinary outputs?
The paper was submitted to arXiv on July 2 by Matteo Boglioni, Thibault Rousset, Siva Reddy, Marius Mosbach, and Verna Dankers. It targets a specific failure mode in post-training privacy work. LLMs can memorize sensitive training data, including personally identifiable information. Unlearning methods try to remove that knowledge after training, but most benchmarks judge the result by model behavior.
LACUNA adds a different measurement target: parameter-level localization.
The testbed injects synthetic PII into known weights
The core design is controlled. LACUNA injects PII for synthetic individuals into predefined parameters of OLMo-based 1B and 7B models through masked continual pretraining.
That gives the benchmark a ground truth. If the testbed knows where the synthetic knowledge was stored, it can ask whether a localize-first, unlearn-second method actually found the responsible parameters.
That is different from output-only testing. A model may stop answering a direct query and still retain recoverable traces. The LACUNA abstract points to resurfacing attacks as the concern: knowledge that looks removed can become visible again under different prompts or attacks.
Output success is not the same as precise removal
The paper says current state-of-the-art unlearning methods can show strong output-level performance while remaining imprecise and susceptible to resurfacing attacks.
That finding is the useful warning. If an organization cares about privacy deletion, copyright removal, or policy-driven data erasure, it should not rely only on whether the model refuses or fails to produce a target string in a test prompt.
Behavioral tests are still necessary. They show what users can extract. But LACUNA argues they are incomplete when the claim is true erasure from the model.
The narrow setup is the strength and the caveat
LACUNA uses synthetic individuals, controlled injection, and OLMo-based 1B and 7B models. That makes the benchmark measurable, but it also narrows the claim.
It does not prove that every real-world unlearning problem can be reduced to parameter-level localization. It does give researchers a way to test whether a method’s localization claim holds under known conditions.
