One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

cactus-compute/needle

★ 7,907+3,838 this weekPython

A 14MB AI model small enough to run tool-calling on a phone or watch, without internet

Needle 2 is a 45-million-parameter model built by Cactus Compute for calling tools and pulling structured data out of text, packed into a single 14MB file that runs a session in about 28MB of RAM. It is compressed to 2-bit precision and paired with its own lightweight inference engine, so it works fully offline on small devices. The Python package lets developers describe functions in code and have the model decide when and how to call them, plus fine-tune it with their own data.

What it does

  1. What: an open small language model specialized in tool calling, on-device use, and structured extraction (turning text into JSON), aimed at phones, wearables, smart home devices, and robots.
  2. How: built on the team's own 'Simple Attention Network' design (described in a linked paper) and compressed with a custom quantization method called Cactus Quants down to 2-bit weights, baked into a single self-contained engine with no internet access needed.
  3. Result: on the paper's benchmarks, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, while being 5x to 70x smaller and using 2-bit precision versus their 16-bit (f16) format.
  4. Extra features: responses come with a confidence score so apps can decide when to ask a human instead, a retrieval system narrows a large list of tools down to the top 5 relevant ones per turn, and a 256-token memory window keeps RAM usage bounded no matter how long a conversation runs.
  5. Workflow: install via pip, decorate a Python function as a tool, and call agent.run(); developers can also fine-tune the model with LoRA (a lightweight way to adapt a frozen base model) on their own tool data and export a new compact file.

Why it matters

Running AI on-device instead of in the cloud means it works offline, keeps data private, and avoids server costs — important for wearables, smart home gadgets, and robots that can't always reach the internet. This shows a working recipe for squeezing a capable, structured-output model into a footprint small enough for constrained hardware.

Terms in this repo

  • LoRA · a lightweight fine-tuning method that trains a small add-on to a frozen base model instead of retraining everything
  • quantization / 2-bit (CQ2) · compressing a model's numbers to use far fewer bits per value, shrinking file size at some cost to precision
  • grammar-constrained decoding · forcing the model's output token-by-token to match a required JSON structure so it can't produce invalid data
  • KV sink / sliding window · a technique to keep only a limited, fixed amount of conversation memory in RAM instead of it growing without bound
  • GQA (grouped-query attention) · an efficient variant of the attention mechanism that reduces memory use in transformer models

Repository description (English)

14MB foundation model for tiny devices; phones, wearables, smart home, and robots.

Open on GitHub

Trending repos

All repos →

Latest from METAL LAB