AgentHands

AgentHands is an experimental technology from Google Research where a conversational AI agent makes human-like hand gestures as it speaks, pointing to objects or locations in the physical space in front of the user. These gestures are shown through XR (extended reality) screen technology, which overlays information onto the real world in front of you.

When you're only told verbally to "turn the handle on the left," you have to mentally translate that instruction into an actual spot in space to find it. It's similar to the difference between getting directions purely by words in an unfamiliar place versus having someone point out the way with their hand. AgentHands aims to fill that gap with gestures, so users can see directly with their eyes instead of having to translate words into space.

The published demos cover eight scenarios, such as pointing to a printer's settings panel, sketching out the size of a space where a drawer should go, or warning about a hot component to be careful of. However, this is still a research-stage prototype rather than a commercial product, so it will take more time before it appears in something like an actual pair of smart glasses.

Articles describe it along the lines of "Google Research unveiled an XR prototype called AgentHands." A common point of confusion here is that this isn't already a shipped product like glasses or an app — it's still a prototype at the level of a research showcase page. When reading such articles, it's worth distinguishing that this isn't a feature built into an actual device yet, but a prototype stage testing whether this kind of approach works.