October 28, 2025
Memory wars: glitter vs grit
The Continual Learning Problem
Can AI keep learning without forgetting? Fans cheer, skeptics say it’s just LoRA with glitter
TLDR: A new study claims “memory layers” let AI learn new facts while forgetting far less than LoRA. The comments split: some say it’s a real step toward always‑learning AI, others argue it’s just LoRA in a fancy outfit and that context distillation is the cleaner fix.
The authors pitch “sparse memory layers” as a way for AI to learn nonstop without wiping its brain—think a giant notebook where only a few sticky notes get updated each time. They drop eyebrow-raising numbers: when teaching new trivia, older knowledge fell by 89% with full retraining, 71% with LoRA (a popular add‑on method), but only 11% with these memory layers. Cue the crowd noise. One camp cheers: less forgetting, more learning! Another fires back: “Isn’t this just fancy LoRA with extra sparkle?” User mynti calls it “a sparse, memory‑efficient LoRA” and basically accuses the paper of a glow‑up rather than a breakthrough. Meanwhile, alyxya barges in with a minimalist hot take: skip the hardware makeover and do context distillation—turn what the model can learn from examples into permanent memory—because long prompts get mushy when attention spreads too thin (aka “context rot”). The thread devolves into memes about “the intern who never forgets” and arguments over classic catastrophic forgetting. Fans see this as a legit path to AI that keeps getting smarter; skeptics see rebranded adapters and say the real fix is smarter training tricks. Either way, the vibe is: we’re closer to an AI that remembers… but the recipe is still up for grabs.
Key Points
- •The article proposes memory layers—high-capacity and sparsely activated—as an architecture for continual learning.
- •Experiments show learning TriviaQA facts causes NaturalQuestions performance to drop by 89% with full finetuning, 71% with LoRA, and 11% with memory layers.
- •Continual learning is framed as two subproblems: generalization and forgetting/integration.
- •Paraphrasing and broader augmentations (e.g., Active Reading) help teach models semantic content and improve factual retention.
- •Naive next-token prediction on raw data streams is inadequate; the authors focus on mitigating forgetting via integration mechanisms.