Catastrophic Forgetting

Catastrophic forgetting is when an AI model, in the process of learning new information, suddenly loses knowledge it had learned before.

Think of a whiteboard. To write something new, you usually have to erase what was written before. Humans can learn new things while mostly keeping old knowledge intact, but many learning models don't work that way. When retrained on new data, the countless internal connection values inside the model can all shift at once, causing a side effect where the model suddenly loses an ability it used to have.

Because of this problem, most language models today are frozen once training ends and are then used as-is in service. If they kept learning continuously, there would be a risk that adding new knowledge could erase existing knowledge, so the choice is instead to lock the internal values in place once training is finished. Researchers like Richard Sutton argue that instead of this approach, we need to find ways for models to keep learning while still retaining old knowledge.

The term appears in the article where Richard Sutton, discussing continual learning, says "all learning should be continual, and yet catastrophic forgetting should not occur." Here, 'forgetting' doesn't mean the model is broken—it refers to the side effect where existing abilities are wiped out entirely in the process of learning something new.