jundot/omlx
A menu-bar app that runs and manages local LLMs on your Mac
oMLX is an open-source server for running large language models directly on Apple Silicon Macs, controlled from a native menu-bar app. It combines continuous batching, which handles many requests at once, with a two-tier cache that keeps reusable computation results in RAM and overflow on SSD. It speaks the OpenAI and Anthropic APIs, so existing tools can connect without changes.
What it does
- Serves not just text models but also vision-language models, OCR models, embeddings, and rerankers from a single server.
- Keeps previously computed context in a fast RAM tier and a larger SSD tier, so even mid-conversation context changes stay reusable, and it survives server restarts by restoring from disk instead of recomputing.
- Includes an experimental feature to split one large model across multiple Macs with different memory sizes over a network.
- With dedicated custom kernels enabled, models like GLM-5.2 and MiniMax M3 process about 845 tokens per second on an M3 Ultra, roughly 30x faster than the ~29 tokens per second fallback without them.
- An admin dashboard lets you download models, chat, run benchmarks, and tune per-model settings with a few clicks, no manual config editing needed.
Why it matters
It gives developers and researchers running LLMs locally on a Mac an easy way to manage memory and performance without touching the terminal. This matters especially for coding assistants, where reusing cached context instead of recomputing it saves real time and cost.
Terms in this repo
- Continuous Batching · Processing many concurrent requests together instead of one at a time, for efficiency
- KV Cache · A temporary store of a model's previously computed results so it doesn't repeat the same calculation
- Apple Silicon · Apple's own chips for Mac, such as M1, M2, M3, M4
- MCP (Model Context Protocol) · A standard way for AI models to connect to external tools or data
- LRU eviction · Automatically removing the least recently used item from memory first
Repository description (English)
managed from the macOS menu bar
Open on GitHubTrending repos
- cathrynlavery/diagram-designA skill that makes AI coding tools draw magazine-quality diagrams instead of generic rounded boxes
- public-apis/public-apisA giant crowd-curated directory of free APIs for developers
- semantica-agi/semanticaAn open-source graph infrastructure that lets AI agents show their work, not just their answers
- cactus-compute/needleA 14MB AI model small enough to run tool-calling on a phone or watch, without internet
- unslothai/unslothA desktop app that lets you run and train AI models on your own computer, no coding required
- macro-inc/macroAn all-in-one workspace where email, chat, docs, tasks, and CRM are cross-linked and share one AI memory
- harry0703/MoneyPrinterTurboAn open-source tool that turns a single topic or keyword into a finished short video, complete with script, footage, subtitles, and music
- basecamp/omarchyA ready-made, opinionated Linux setup built by DHH
Latest from METAL LAB
- Grok leaks conversations, location data via commands hidden in ciphertext
- Cursor lets agents handle long-running tasks with new "/goal" command
- NVIDIA releases Switchyard, an LLM routing proxy for coding agents
- Caveman Cuts Claude Code Token Usage by 33% Using Caveman-Speak
- Apple Keeps 2027 Target for Camera AirPods, Qwen3.8-27B Makes 80 Tool Calls Locally