Forget Narrowly, Retain Broadly

Unlearning as an Asymmetric Generalization Problem

Amit Peleg · Naman Deep Singh · Naama Pearl · Bibhabasu Mohapatra · Matthias Hein

University of Tübingen

arXiv Code SUITE dataset Email

Abstract

Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety. Yet existing benchmarks measure it unreliably. They miss knowledge that resurfaces under paraphrased or indirect queries, a failure we call under-forgetting, and lack the semantic, syntactic, and lexical probes needed to verify that unrelated knowledge is preserved, a failure we call over-forgetting.

Both failures reflect an asymmetric generalization problem. Forgetting must generalize intensively across every formulation of the target facts, while retention must generalize extensively across the vast, implicitly-defined set of everything else. We address this with SUITE, an evaluation protocol and training corpus that captures forget–retain structure for real-world factual domains. Methods trained on SUITE improve substantially, showing that training data is as important as algorithmic design. Building on these insights, we introduce JensUn++, which achieves the best forget–retain–utility trade-off across three LLMs, in both sequential and joint unlearning settings.

TL;DR

Three contributions

The Task

Forget one topic, keep everything else

Unlearning must remove the forget topic while keeping everything else intact.

Two failure modes benchmarks miss

Under-forgetting

Target knowledge is suppressed on the exact training prompt but resurfaces under paraphrases, reverse queries, or indirect multi-hop questions. Forgetting must generalize intensively, across every formulation of the same fact.

Example: indirect query

“After how many seconds did the disaster of January 1986 occur?”

✗ the model still answers “73 seconds”, so the fact was never really forgotten.

Over-forgetting

Unlearning spills over into semantically, syntactically, or lexically related knowledge the model should keep. Retention must generalize extensively, across the huge, implicitly-defined set of everything disjoint from the forget target.

Example: lexical probe

“What is the depth of the Challenger Deep?”

✗ the model answered this question correctly before unlearning, but now it doesn’t.

Benchmark

SUITE: pinning down the forget–retain boundary

Selective Unlearning of Isolated Topics and Events, a fine-grained evaluation protocol and training corpus that makes the boundary explicit at the semantic, syntactic, and lexical levels.

SUITE spans four diverse forget topics, each with its own retain neighborhood.

Space Shuttle Challenger, 1986.
Historical eventChallenger disasterForget the 1986 launch catastrophe, while retaining the rest of the space shuttle program history, and everything else…
The 1692 Salem witch trials.
Historical eventSalem witch trialsForget the 1692 colonial witch trials (a fairly isolated episode), while retaining everything else…
Steve Jobs.
Personal / contemporarySteve Jobs medicalForget his private medical history, while retaining his well-documented public career, and everything else…
Britney Spears performing live.
Personal / contemporaryBritney Spears conservatorshipForget the legal conservatorship, while retaining her best-selling music career & public life, and everything else…

The walkthrough below uses the Challenger disaster as the running example. The space shuttle Challenger broke apart on 28 January 1986, just 73 seconds after liftoff.

The 1986 Space Shuttle Challenger disaster.

Forget

SUITE forget side: the same Challenger-disaster fact queried as direct, reverse and indirect questions.
One fact, asked three ways: direct, reverse, and (test-only) indirect. Each is expanded into 10 paraphrases + 5 fill-in-the-blank variants. A fact counts as forgotten only when the model fails all of them.

Retain · Semantic & general knowledge

Semantic tiers: the forget topic (Challenger disaster) at the centre, ringed by SAME ENTITY, CLOSE TOPICS and DISTANT TOPICS, with tiers 0–15 placed by closeness. A general-knowledge item (Julius Caesar) sits outside the rings, entirely unrelated.
  1. s0 the Challenger shuttle
  2. s1 the Columbia disaster
  3. s2 the Apollo 1 fire
  4. s3–s14
  5. s15 the ISS

Close in meaning, different topics: the model should not suppress questions close in meaning to the forget topic. General-knowledge facts (e.g. Julius Caesar) lie entirely outside the topic and should be kept too.

Retain · Lexical & syntactic

Forget questionHow many seconds after liftoff did the Challenger break apart?

Lexicalsame word

Retain probeWhat is the depth of the Challenger Deep?

The Challenger Deep, the deepest point of the ocean.

Reuses a word from the forget question, but asks about an unrelated topic the model should keep.

Syntacticsame template

Retain probeHow many hours after striking the iceberg did the Titanic break apart?

The sinking of the Titanic, 1912 painting by Willy Stöwer.

Reuses the forget question’s template, but asks about an unrelated entity the model should keep.

Published as two HuggingFace datasets (all topics in one repo, sliced by topic): apeleg/SUITE and apeleg/SUITE-rephrasings.

Load any topic in two lines 
from datasets import load_dataset
ds = load_dataset("apeleg/SUITE", split="forget_train") \
       .filter(lambda x: x["topic"] == "challenger_disaster")

Forget topics: challenger_disaster, salem_witch_trials, steve_jobs_medical, britney_spears_conservatorship.

Method

JensUn++: refuse, don’t babble

JensUn++ trains the model to produce a natural refusal instead of suppressing the answer.

Three improvements over JensUn

Why we want a method that refuses

Suppression-based methods stop the model from outputting the ground-truth answer, but don’t control what it says. On forget queries they emit gibberish or harmful hallucinations. We should control what comes out instead.

Example forget topic: Britney Spears conservatorship

Q: What highly publicized 2007 incident became the defining media image of the pop icon’s crisis prior to her 13-year legal guardianship? ground truth: “shaving her head”

Britney Spears performing live.
⚠ Disturbing model outputs: click to show: click to hide
MethodOutput on the forget query
Llama (pretrained)“Britney Spears shaving her head.”
⚠GradDiff“2007 north american tour, sex, and child abuse allegations…”
🛡JensUn“No Doubt's lead singer, Gwen Stefani, No Doubt's lead singer, …”
⚠NPO“Leaked sex tape.”
⚠PDU“Michael Jackson death Anna Nicole Smith 2007 trial 2007 2007 …”
🛡JensUn++“Unfortunately, I am unable to verify this information.”

⚠ suppression hallucinates or emits gibberish.   🛡 refusal aims to refuse, but only JensUn++ does so cleanly.

Findings

Validating SUITE’s design choices

Now that the benchmark exists, we check that its fine-grained design actually improves unlearning. We run two controlled experiments, each swapping one choice from LKF* to SUITE: first the evaluation protocol, then the training data.

LKF*, lesser-known facts, is an existing benchmark where forgetting is probed only with paraphrases of the training questions, and its coarse retain set is scored on the training facts themselves.

Why fine-grained evaluation data matters

Here the training is fixed (every model trained on LKF*), and we change only the evaluation. Under LKF*-eval they look like they forget and retain well. SUITE-eval exposes the under- and over-forgetting it hides, and the method ranking flips.

What over-forgetting looks like

A held-out lexical probe that reuses the word “Challenger” but is semantically unrelated to the forget topic. Probes like this are exactly what expose over-forgetting.

Q: In the word “Challenger”, what is the first letter to make a second appearance? ground truth: “L”

MethodOutput on the unrelated retain query
Llama (pretrained)“L” ✓
GradDiff“S — the 1st and the last letter are the same, the 1st and the 4th…”
JensUn“No double ‘l’ in the word Challenger.”
NPO“The letter ‘C’ is the first letter to make a second appearance…”
PDU“The letter E appears twice in the word Challenger.”
RMU“in the Answer Answer Answer Answer…”
JensUn++“L” ✓

The model suppresses the answer to any question that mentions “Challenger”, because the word only ever appeared on the forget side of training. Without a lexical probe you would never know it happens. This raises the question: would better training data that draws the forget–retain boundary fix this?

Why fine-grained training data matters as much as the algorithm

Now the evaluation is fixed (every model scored with SUITE), and we change only the training data. Trained on the coarse LKF*, methods fall short. Trained on fine-grained SUITE, every method except RMU improves substantially.

Main takeaways

What fine-grained evaluation and training data reveal about unlearning.

  1. Direct questions alone hide under-forgetting. Methods look unlearned on direct questions, but the fact resurfaces under reverse and indirect phrasings.
  2. Staying close to the base model preserves retain. Methods trained to reproduce the retain ground-truth answers overfit and break on held-out facts from the trained tiers (s1–5). Those that try to mimic the base model’s behavior generalize.
  3. Training must reach remote semantic topics. Training on close neighbors isn’t enough to maintain performance on the held-out remote tiers (s6–15). Keeping those requires more distant topics in the training set too.
  4. The semantic boundary must come from the data. Every method collapses on the closest topics (s0) unless the boundary is given in training.
  5. Lexical and syntactic boundaries matter too. Without drawing the boundary at the lexical and syntactic levels as well, methods over-forget.
Results

Sequential unlearning: best trade-off, no gibberish

The four topics are unlearned one after another and averaged: Challenger disaster → Salem witch trials → Steve Jobs medical → Britney Spears conservatorship.

JensUn++ vs. the best baseline across LLMs (sequential). Four key metrics per model. The best-performing baseline is JensUn for all three. Bold = better value.
Metric
JensUn++best baseline JensUn++best baseline JensUn++best baseline
Q*All ↓3.016.04.05.04.02.0
Gib. ↓0.077.20.073.20.091.9
Retain ↑51.650.050.147.951.052.5
RGQ ↑48.538.145.940.336.227.3

Full per-metric breakdown. All values are evaluated under SUITE and averaged over the four topics. Click a model in the table above to expand its breakdown.

Joint unlearning: all four topics at once

The four topics are unlearned simultaneously in a single run, then averaged.

JensUn++ vs. the best baseline across LLMs (joint). Four key metrics per model. The best-performing baseline is JensUn for all three. Bold = better value.
Metric
JensUn++best baseline JensUn++best baseline JensUn++best baseline
Q*All ↓10.013.00.022.012.05.0
Gib. ↓0.082.50.090.30.188.6
Retain ↑51.951.551.350.852.651.7
RGQ ↑44.545.948.549.544.437.6

Full per-metric breakdown. All values are evaluated under SUITE and averaged over the four topics. Click a model in the table above to expand its breakdown.

BibTeX

@article{peleg2026forget,
    title   = {Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem},
    author  = {Peleg, Amit and Singh, Naman Deep and Pearl, Naama and Mohapatra, Bibhabasu and Hein, Matthias},
    journal = {arXiv preprint arXiv:2607.09236},
    year    = {2026}
}