Matic cleaning robot now goes where you point

Robot vacuums that respond to voice commands to clean already existed, but the core of this update lies in converting the direction a person points into actual physical coordinates. The robot first identifies where the user's arm is, then projects the direction that arm is pointing into three-dimensional space. It then compares this calculated direction against the 3D floor map it has already built and stored, to pinpoint the exact spot to clean. Voice alone gives the robot no way to understand deictic words like "over there" or "this way," but adding the physical signal of a hand gesture solves that problem.

How to use it

Based on what's been disclosed, the process works as follows:

1. With the robot active, point at the messy spot on the floor.
2. At the same time, say "Hey Matic, clean this."
3. The robot uses its cameras to recognize the position and direction of the user's arm.
4. It matches the recognized direction against the 3D map stored internally to identify the target location.
5. The robot moves to that location and cleans.

Unlike previous methods, this process requires no separate action such as opening an app or zooming into a map to mark a location.

The technical background — why on-device

Matic's robot is equipped with five RGB-IR cameras and an onboard NVIDIA computer. RGB-IR cameras combine standard color data with infrared data, allowing the robot to detect the position of objects even in dark environments. Tasks like gesture recognition, which require processing camera footage in real time at every moment, tend to suffer from latency if the footage is sent to a cloud server and results are returned from there. Using on-device processing, where the robot's internal computer completes the calculations itself, allows for immediate response without such delays.

Processing commands by recognizing voice and body movement together is a trend showing up across several fields recently. For example, the sign-language recognition model SL2T, released by Google DeepMind on August 12, also tracks hand movements on-device and offloads only the computation of converting them into text to a server. There's a growing pattern in system design where the part that reads a person's physical signals via camera stays on the device, while the heavier computation for interpreting those signals is split between a server and an onboard chip depending on the situation.

Why robots need to understand human gestures

Telling a home robot exactly where to go has long been a difficult problem to solve. Voice alone makes it hard to specify a location, and marking a spot on an app map is cumbersome when your hands are full or you're in a hurry. This update appears aimed at letting the robot accept the natural way people already ask others to clean something — pointing while speaking — rather than forcing users into some other interface. The ability to interpret physical gestures and language together is a foundational technology that could extend beyond home robots to robots collaborating with people in warehouses or factories.

So what does this change

With this update, Matic robot users can now specify a cleaning spot using only hand gestures and speech, instead of opening an app and manipulating a map. This can be read as a sign that robot vacuums are moving past the stage of simply mapping a home on their own, into a stage where they interpret human gestures in real time and respond to them. As more cameras and onboard computing power get built into home robots, the way people communicate with them is likely to keep shifting from touchscreens toward gesture and conversation.