RDMA over Converged Ethernet (RoCE)

This is a communication method that, when several computers are connected by cables, lets data move straight from one machine's memory to another's without passing through either computer's CPU.

Normally, when computers exchange data, the process looks a lot like mail going through a post office. The sending CPU packs the data into an envelope, and the receiving CPU has to open that envelope and check it before passing the contents along to where they're needed. Each of these steps takes time, and the CPU has to stop whatever else it's doing to handle the packing and checking. This method skips that post-office step entirely, moving data directly from the sending computer's memory to the receiving computer's memory. Meanwhile, the CPU stays free to focus on other work.

In setups where multiple computers are linked by cables to work almost like a single machine, the speed of data transfer between them often determines how fast the whole job runs. This method makes that kind of direct transfer possible even over ordinary Ethernet cables and equipment, so fast data movement doesn't require pricier, specialized hardware.

The article explains that a single port on the DGX Spark shows up in Linux as two Ethernet interfaces, plus a matching pair of RoCE interfaces, adding up to four network interfaces per port. A common point of confusion is that RoCE isn't a separate cable or device — it's a communication method that runs over the same Ethernet cable. Physically there's just one cable plugged in, but on top of it, both regular Ethernet traffic and direct memory-to-memory traffic can operate at the same time.