AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox

METAL LAB

goccy/go-llama

★ 40GoMIT

Running large language models with nothing but pure Go code

go-llama takes llama.cpp, the widely used engine for running AI models, compiles it to WebAssembly, and then translates that into plain Go source. The result runs GGUF model files without cgo or any shared library, as a single static binary. No wasm runtime is needed at execution time since everything has already become native Go.

What it does

  1. llama.cpp is compiled to WebAssembly via the llama-wasm project, then translated into Go source code by wasm2go, so no wasm runtime runs at execution time.
  2. A single Llama instance can load multiple models, and each model can spawn multiple contexts (each keeping its own KV cache) so one model can serve several independent conversations, or multiple models can be loaded together for speculative decoding, where a small draft model speeds up a larger target model.
  3. It supports streaming, where generated text is delivered piece by piece as it decodes, and interruption, which lets another goroutine (Go's lightweight thread) stop a running generation immediately.
  4. Sandbox options let you restrict which directory the engine can see and cap its memory growth, so an oversized model fails safely inside the guest rather than crashing the host process.
  5. Because wasm32 caps linear memory at 4 GiB, and model weights plus every context's KV cache must fit inside that space, the practical target is roughly a 3B-parameter model quantized to Q4.

Why it matters

Because it needs no cgo and no separate shared library, a single Go build produces a portable binary that can be cross-compiled and deployed anywhere Go runs, from servers to CLI tools to embedded systems. This removes the usual friction of installing native inference libraries when adding LLM features to a Go project.

Terms in this repo

  • cgo · a Go feature that lets Go code call C libraries
  • GGUF · a compact file format used to store AI model weights, common in the llama.cpp ecosystem
  • WebAssembly (wasm) · a low-level bytecode format designed to run portably across environments
  • KV cache · per-conversation memory that stores previous computation results to speed up generating the next word
  • speculative decoding · a technique where a small model predicts ahead and a larger model verifies the predictions in one pass, speeding up generation

Repository description (English)

llama.cpp in pure Go

Open on GitHub

Trending repos

All repos →

Latest from METAL LAB