Neural Attention Memory Model

A Neural Attention Memory Model is a design concept that puts two abilities into a single structure: the ability to judge which parts of the information an AI is currently looking at deserve focus, and the ability to store information for later use and pull it back out when needed.

Think of a secretary taking meeting notes. Out of all the words spoken during a meeting, the secretary highlights only the important sentences, and organizes that content into notes so it can be found again later. This concept is about building a secretary who does the highlighting and the note-organizing not as two separate steps, but together, and very fast.

When today's chatbots use the beginning of a long piece of text to naturally continue a sentence further down, or recall an earlier conversation to carry it forward, they're relying on a combination of these two abilities. But unlike a computer storage device that can hold data indefinitely, they must constantly choose what to keep and what to discard within a fixed capacity—similar to human memory in that respect, but also different from it.

You can get a feel for this ability by chatting with a chatbot yourself.

1. Start a conversation with a chatbot that mixes several topics together.
2. Slip in one minor piece of information partway through—for example, 'By the way, I like the color blue.'
3. Talk about a few other things for a while, then ask, 'What color did I say I liked?'
4. If it answers correctly, it paid attention to that information and remembered it. If it answers incorrectly or says it doesn't know, that information either fell outside the conversation's storage limit or was never captured in the first place.