Liquid AI open-sources on-device benchmark Pipette

Most benchmark platforms up to now have been built around measuring the capability and speed of large models running in the cloud. Models running on phones, laptops, PCs, or embedded hardware are a different story. Even the same model can swing wildly in speed and score depending on how many bits it's been compressed to, which execution engine it's running on, and which chip it's sitting on. Pipette is a tool built to hold those conditions fixed and measure things in a reproducible way.

The unit of comparison isn't "one model"

In Pipette, a single row for comparison bundles together model + quantization + runtime + device. So, for example, "LFM2.5-1.2B-Instruct, compressed with q4_k_m, running on the llama.cpp/ios runtime, loaded onto an iPhone 17 Pro" is treated as one comparable unit.

Quantization here means lowering the precision of a model's weights to shrink its footprint and memory usage. Compress a model down to 4 bits and it fits on a phone, but accuracy takes a small hit. llama.cpp is a widely used open-source execution engine for running models on personal devices rather than servers, and it's the runtime family that fills out the initial release.

Looking at the model list on the dashboard, Liquid AI's own LFM2.5 lineup (230M, 350M, 1.2B, 2.6B, 8B-A1B) sits alongside names like Qwen3.5, gemma-4, granite-4, Llama-3.2, Ministral-3, MiniCPM5, Olmo-3, Falcon-H1R, Ornith-1.0, and Nanbeige4.2, all on the same screen. On the iPhone 17 Pro view, 27 models are lined up on a single axis.

What's in the initial release

Item	Initial release
Model classes	About 35, from multiple providers
Quantization	7 levels, based on llama.cpp
Runtime	llama.cpp family (macOS, iOS, Android)
Devices	MacBook Pro (M5 Max), iPhone 17 Pro, Galaxy S26 Ultra, AMD Ryzen AI Max+ 395 + Radeon 8060S (coming soon)
Context	256–8,192 tokens, within device memory limits
Result count	Over 10,000 verified results, more than 1,000 performance configuration combinations

Six things were released together: an interactive dashboard for exploring and comparing results, a public dataset with over 10,000 entries, benchmark clients for macOS, Windows, iOS, and Android, a companion dashboard on the Artificial Analysis side showing composite scores, benchmark management/execution/submission/scoring infrastructure released under Apache 2.0, and native iOS and Android apps that run directly on-device. The original materials are somewhat inconsistent about exactly which quantization levels are covered on laptops/desktops versus phones, so here we'll just note that coverage "varies by device across the seven levels."

Speed is measured on-device, scores on H100

One notable design choice is that the measurement setup is split in two. Speed and memory are measured on the submitter's actual device, while quality evaluation runs and gets scored on H100 servers. It's a separation meant to keep phone-to-phone differences in heat and power from muddying accuracy scores.

According to the published submission data, there are 4,284 submission IDs on the speed/memory side and 303 on the quality evaluation side. Here's the breakdown by runtime:

Runtime	Records	Share	Submission IDs
llama.cpp:macos-arm64	2,430	45.4%	1,944
llama.cpp:ios	1,475	27.5%	1,180
llama.cpp:android-arm64-v8a	1,450	27.1%	1,160

Quality evaluation has accumulated 303 records across three llama.cpp-based benchmarks. That means speed data outnumbers quality data more than tenfold — which makes sense, since timing a device is far lighter work than grading a test on a server.

The mobile composite score is a simple average of five tests

The composite score for iPhone 17 Pro shown on the Artificial Analysis dashboard is a simple average across five evaluations: a subset of BFCL, IFBench, AA-Omniscience, GPQA Diamond, and MATH-500, all capped at a 16K-token context. The E2E time on the horizontal axis refers to seconds elapsed processing a 1,024-token prompt and generating 256 tokens. This particular view shows 22 of the 38 models.

The chart includes a Pareto frontier — a line connecting the models that are either more accurate at the same speed or faster at the same accuracy. When narrowing down candidates for an actual device deployment, this is the line to look at first. It's also worth reading the conditions noted in the dashboard caption: the view described above locks in q4_k_m quantization, the llama.cpp/ios runtime, a 1,024-token input, and reasoning turned off. Change those conditions and the ranking shifts too.

How to actually use it

Where to start — there are four entry points: the Pipette dashboard, the companion dashboard on Artificial Analysis, the iOS and Android apps, and the macOS/Windows client. Links to each are laid out item by item in the second tweet of Liquid AI's announcement thread.

1. Open the dashboard and pick the hardware you plan to deploy on from the device selector at the top. Choose iPhone 17 Pro, for instance, and only results from that device remain.
2. Narrow your candidates using the model filter. With everything turned on, all 27 models show up at once, so it's easier to start by comparing models of similar parameter size.
3. Try switching the axes. You can set the vertical axis to an accuracy metric like IFBench and the horizontal axis to E2E latency or memory.
4. Use the Constraints button to lock in conditions. Numbers only mean something if the quantization level and input token length match your actual deployment setup.
5. Start with candidates that sit on the Pareto frontier. Points below the line are combinations that deliver the same value more slowly.
6. If the combination you want isn't in the dataset, run the iOS/Android app or desktop client on your own device to measure it yourself and submit the result. Approved submissions get added directly to the public dataset.
7. If there's a combination you want measured but the client doesn't support it, Liquid AI says to file an issue.

Who can use it — the infrastructure is Apache 2.0-licensed, so there are few restrictions on commercial use. The clients support macOS, Windows, iOS, and Android, and the team says the system is built to let new devices, runtimes, model families, and quantization levels be added from day one.

What you can try — say your team wants to put a 3B-class model into an internal app. You could check the E2E seconds and memory for 4-bit combinations on a Galaxy S26 Ultra first, and narrow it down to two or three candidates. Building a laptop tool? Compare MacBook Pro (M5 Max) results with the upcoming AMD Ryzen AI Max+ 395 numbers to get a sense of what the Windows-side experience will feel like. And if there's a combination nobody's measured yet, it's often faster to just run it yourself and generate the first data point.

Editor's view

Anyone who's actually had to pick an on-device model will immediately see the value here. Up to now, fitting a small model onto a phone or laptop meant piecing together estimates from other people's blog posts, GitHub issue comments, and community-built spreadsheets. The problem was that those numbers were never measured under the same conditions — one person tested at 8-bit, another at 4-bit; one on an M2, another on an M4. When the conditions differ, what you're doing isn't comparison, it's guesswork dressed up as data. By locking the combination down as a single unit, Pipette is addressing a real methodological gap.

Splitting where measurements happen also reflects practical thinking. Running the full GPQA Diamond suite on a phone would eat a whole day, and thermal throttling would slow things down as it goes. Since the correct-answer rate doesn't actually depend on the device, it makes sense to score that once on H100 and reserve on-device measurement for speed and memory. The tradeoff is that this structure can't capture performance degradation from sustained load on a device — which happens to be one of the more painful realities of real-world deployment. Whether a sustained-load metric gets added down the line will be the next real test for this platform.

There's no need to be suspicious just because Liquid AI's own models appear on its own benchmark chart. What matters is how to read the setup. Three things back up the credibility of this table: the dataset is public, third-party submissions are open, and scoring involves an independent analysis firm that doesn't build its own models. If any one of those three erodes, the value of the whole table erodes with it. The one thing readers need to do is always check the conditions listed in the caption. If q4_k_m, a 1,024-token input, and reasoning-off don't match your own deployment setup, that ranking isn't really about your use case.

For teams in Korea, having a Galaxy S26 Ultra among the initial devices is genuinely useful — it lets you check, without relying on someone else's guess, just how large a model realistically runs on an Android flagship. If your team is evaluating an on-device feature, our advice: before writing "using a 3B-class model" into a spec, pick a target device and actually run two or three combinations sitting on the Pareto frontier. As more submitted data comes in over the coming weeks and the runtime list expands beyond llama.cpp, that's when this table will really become usable for product decisions.