Liquid AI’s Pipette Benchmarks On-Device AI as a Full Deployment Stack

Liquid AI this week open-sourced Pipette, a benchmarking platform that measures AI model performance on edge devices by treating the full deployment stack (model, quantization format, runtime, and hardware) as a single unit. Built in partnership with Artificial Analysis as an independent methodology validator, Pipette launches with a dataset of five performance metrics across more than 1,000 configurations, covering 30-plus models and context lengths from 256 to 8,192 tokens. Initial verified results span a MacBook Pro with M5 Max, an iPhone 17 Pro, and a Galaxy S26 Ultra. Everything ships under Apache 2.0 with no waitlist.

What happened

Detail	Fact
Release	Liquid AI open-sourced Pipette, validated by Artificial Analysis
Dataset size	5 metrics across 1,000+ model × quantization × runtime × device × context configurations
Models covered	30+ models, llama.cpp builds for macOS, iOS, Windows, Android
Context lengths	256 to 8,192 tokens
Initial test devices	MacBook Pro (M5 Max), iPhone 17 Pro, Galaxy S26 Ultra
Coming soon	AMD Ryzen AI Max+ 395 and Radeon 8060S results
License	Apache 2.0
Waitlist	None. Dashboard and apps available now; community result submission in beta.

The core argument behind Pipette is simple: model cards report quality under server-class, full-precision conditions. Those numbers do not predict what the same model does on a phone. Liquid AI’s answer is to define the unit of measurement as a deployment configuration: model plus quantization plus runtime plus device. A benchmark then adds a metric and a token shape to produce a latency, throughput, or memory result.

Quality is tracked separately on IFBench, GPQA Diamond, and MATH-500. Those scores come from llama.cpp runs on NVIDIA H100 80GB reference systems and are then matched to on-device results that share the same model and quantization. The quality number shown next to a phone throughput result was not produced on the phone.

Why does deployment context change the answer?

Liquid AI published four comparisons that show how much a configuration choice moves real numbers.

Context length scaling

On a Galaxy S26 Ultra at Q4_K_M quantization, Granite-4.0-H-350M keeps 78.4% of its decode throughput when input tokens scale from 256 to 4,096. Granite-4.0-350M, at the same parameter count and quantization on the same device, keeps only 33.8%. Same size, same phone, very different behavior under load.

Sparse activation and memory

At 2,048 input tokens on the Galaxy S26 Ultra, LFM2.5-8B-A1B decodes 2.4x faster than Qwen3.5-4B and 2.6x faster than Ministral-3-3B-Instruct-2512. The model activates 1.5B of its 8.5B parameters per token. Despite that efficiency, it still peaks at 5.29 GiB of memory because all expert weights must sit in RAM.

Speed versus quality trade-off

Model	Device	Quantization	Elapsed time (2,048-in / 256-out)	MATH-500 score
MiniCPM5-1B	iPhone 17 Pro	Q4_K_M	3.47 s	(lower)
LFM2.5-1.2B-Instruct	iPhone 17 Pro	Q4_K_M	4.12 s	+9.0 points vs MiniCPM5

MiniCPM5-1B is 15.8% faster on that workload. LFM2.5-1.2B-Instruct scores 9.0 points higher on MATH-500. Speed and quality do not always live in the same model.

Near-identical profiles hiding task reversals

On an M5 Max at Q4_K_M with 2,048 input tokens, Granite-4.1-8B and Ministral-3-8B-Instruct-2512 differ by just 2.4% in decode throughput and 1.2% in peak RAM. Granite leads IFBench by 7.3 points; Ministral leads GPQA Diamond by 14.0 points. A single aggregate score would mask which model actually fits a given task.

How Pipette produces its measurements

Performance runs follow a published protocol: fixed token shapes, greedy decoding, one discarded warm-up pass, five measured repetitions, and a readiness gate before each timed run. That gate checks thermal and load conditions on the device; any run that fails the check is not published. Evaluations use a separate, deterministic, model-blind scoring protocol. Every submission records benchmark version, token shape, model artifact, quantization, runtime version, device hardware, and OS.

The three Apache 2.0 packages are pipette-mgmt (orchestration), pipette-clients (device-side measurement), and pipette-scores (evaluation). Solo developers and seed-stage teams can use the hosted dashboard and the iOS and Android apps without running any infrastructure. Larger teams can run the clients across an internal device fleet, and enterprises can operate the full pipeline behind a firewall.

Why it matters

Edge AI deployment is growing fast across consumer electronics, automotive, industrial robotics, healthcare devices, and anywhere latency, privacy, or connectivity makes cloud inference impractical. Until now, teams picking a model for a phone or embedded chip had to run their own ad-hoc tests or trust vendor claims. Pipette gives a shared, reproducible baseline for those decisions.

The use cases are practical: choosing a model and quantization before a sprint commits, validating SoC procurement, catching regressions when a runtime or OS update ships, planning context-length capacity, and verifying vendor performance claims independently.

For teams already thinking about AI integration in their products, having a reproducible benchmark that separates hardware behavior from model quality is genuinely useful. It changes procurement from guesswork to a testable checklist. Our recent coverage of where enterprises are actually running AI models shows this infrastructure layer is exactly where buying decisions are being made right now.

Our take

Pipette addresses a real gap. Benchmark theater, where a model looks excellent on an H100 leaderboard but performs poorly on a mid-range Android device, is a genuine problem for anyone shipping inference to hardware they do not control. The decision to make Artificial Analysis the methodology validator was smart: it gives the dataset credibility that a self-reported benchmark from a model vendor would not have.

The honest caveat is that quality scores still come from H100 reference runs. That means you are comparing server-measured quality against device-measured throughput, which is not the same as end-to-end on-device evaluation. Liquid AI flags this clearly, which we appreciate, but it is a limitation worth keeping in mind when using MATH-500 or GPQA Diamond scores to justify a model choice for a phone deployment.

The Apache 2.0 license and no-waitlist access are the right call. A benchmarking tool only becomes useful when enough teams contribute results to cover more devices. The community submission beta will be the real test of whether Pipette becomes an industry reference or stays a Liquid AI showcase.

What to do about it

1. Visit the Pipette hosted dashboard and filter by the device class closest to your deployment target.
2. Download the iOS or Android benchmark app and run your candidate models against your actual target hardware before committing to a sprint.
3. Cross-reference throughput results with the matching quality scores on IFBench or MATH-500, keeping in mind those scores come from H100 runs, not the device.
4. If you operate a device fleet, pull the pipette-clients package and run the full configuration matrix internally to catch runtime or OS regression before a production update ships.
5. Submit results back to the community dataset once the beta opens fully. A wider device coverage base benefits everyone using the tool.

If your product decisions currently rely on model card numbers alone, Pipette is worth an afternoon of testing before your next hardware or model selection call.

Source: Marktechpost