Inkling-Small: 276B MoE Model That Runs on One GPU
Thinking Machines Lab releases Inkling-Small, a 276B total / 12B active open-weights MoE model with 1M token context, multimodal input, and single-GPU NVFP4 deployment.
Thinking Machines Lab has released Inkling-Small, an open-weights Mixture-of-Experts model with 276 billion total parameters and 12 billion active per token. Released under Apache 2.0, it supports native text, image, and audio input with a 1 million token context window. A quantized NVFP4 checkpoint cuts the VRAM floor to 180 GB, meaning a single NVIDIA B300 can serve the full model. On several key benchmarks it outperforms its much larger sibling Inkling, which carries 975B total parameters.
What happened
| Detail | Inkling-Small |
|---|---|
| Total parameters | 276B |
| Active parameters per token | 12B |
| Context window | 1M tokens |
| BF16 VRAM floor | 600 GB (4x B300 or 8x H200) |
| NVFP4 VRAM floor | 180 GB (single B300 or 2x H200) |
| License | Apache 2.0 |
| Training hardware | NVIDIA GB300 NVL72 |
| Supported runtimes | SGLang, vLLM, TokenSpeed, Unsloth, Hugging Face |
Thinking Machines Lab published Inkling-Small on Hugging Face with full weights and a model card. The model is a 42-layer decoder-only transformer with a sparse MoE (Mixture-of-Experts) feed-forward layer, where each token activates 6 of 256 experts plus 2 shared experts that run on every token. Attention alternates between local and global layers.
Rather than using a separate encoder, the model handles images and audio natively. Images are split into 40×40-pixel patches processed by a four-layer hMLP. Audio arrives as dMel spectrograms from 16 kHz WAV files, ideally under two minutes. All modalities merge into the same decoder. Output is text only.
How it compares to its larger sibling
| Benchmark | Inkling-Small | Inkling (975B total) |
|---|---|---|
| Humanity’s Last Exam (text) | 31.6% | 29.7% |
| SWE-bench Verified | 80.2% | 77.6% |
| ARC-AGI-2 | 40.1% | 36.5% |
| Toolathlon Verified | 54.4% | 45.5% |
| GPQA Diamond | 89.5% | not reported |
| AIME 2026 | 95.5% | not reported |
| SimpleQA Verified (factual recall) | 20.6% | 43.9% |
| Tau 3 Banking | 15.5% | 23.7% |
All evaluations ran at effort 0.99 and temperature 1.0, with a 256K max-token trajectory limit on coding evals. External scores came from Artificial Analysis, Scale AI, and ARC Prize.
The gains came partly from training method. Inkling-Small started training after Inkling, which let the team revise the pre-training data mix. It was then post-trained using on-policy distillation (using the larger Inkling as a teacher model to guide the smaller one) and two additional weeks of scaled agentic coding reinforcement learning.
Why it matters
The single-GPU deployment path is the headline business fact here. A 276B model normally demands a rack of high-end hardware. With the NVFP4 checkpoint, a startup can rent a single B300 instance and self-host the full model. Organizations with existing H200 capacity can run it on two cards in W4A16 mode without buying new hardware.
For regulated industries, the Apache 2.0 license and private-weights option matter as much as raw performance. Financial services, healthcare, insurance, telecom, and public sector teams can run the model fully on-premises. Thinking Machines Lab lists coding agents, terminal automation, document and chart understanding, call-center analytics, voice interfaces, and meeting summarization as target workloads.
Multimodal scores stay competitive: MMMU Pro is 74.0%, CharXiv RQ is 77.4% (rising to 81.3% when the model uses Python to programmatically crop and inspect charts), MMAU is 77.0%, and VoiceBench is 90.1%.
The one real regression is factual recall. SimpleQA Verified fell from 43.9% on Inkling to 20.6% on Inkling-Small, and the AA Omniscience index dropped from 2.1 to -9.0. Any deployment that relies on precise factual retrieval, such as knowledge-base Q&A or compliance lookups, needs to account for this gap. Thinking Machines Lab recommends layering a downstream moderation tool like Llama Guard on consumer-facing deployments.
Our take
The single-GPU NVFP4 story is genuinely new. Most “open weights” frontier models remain functionally closed because only hyperscalers can afford the hardware to run them. Inkling-Small changes that math for teams with access to one B300 instance.
The factual recall drop, however, is not a footnote. Falling from 43.9% to 20.6% on SimpleQA is a 53% relative decline. If you are building anything that needs the model to retrieve and state facts accurately, such as a customer support bot, a compliance tool, or a financial research assistant, you will need retrieval-augmented generation (RAG) on top. The coding and reasoning benchmarks are impressive, but this is not yet a drop-in replacement for a model you are relying on for factual grounding.
For teams already exploring AI integration for business workflows, this is worth testing now. The Apache 2.0 license removes legal ambiguity, and the supported runtimes (SGLang, vLLM, Unsloth) are production-grade. Just be honest with yourself about whether your use case needs reasoning or recall. As we covered in our look at AI token costs blowing enterprise budgets, the real cost of AI often shows up after the model launch.
Test Inkling-Small on your actual workload before committing to infrastructure. Benchmark scores and production behavior are rarely the same thing.
Frequently asked questions
What are the hardware requirements for Inkling-Small?
The BF16 checkpoint needs at least 600 GB of aggregated VRAM, met by 4x NVIDIA B300 or 8x NVIDIA H200. The NVFP4 quantized checkpoint drops that to 180 GB and runs on a single B300 (SM100+ required) or two H200s.
What makes Inkling-Small a Mixture-of-Experts model?
Inkling-Small has 276B total parameters but only activates 12B per token. Each token routes to 6 of 256 experts, plus 2 shared experts active on every token, keeping compute costs much lower than a dense 276B model.
Does Inkling-Small outperform its larger sibling Inkling?
Yes on reasoning and coding tasks. Inkling-Small scores higher on Humanity's Last Exam (31.6% vs 29.7%), SWE-bench Verified (80.2% vs 77.6%), ARC-AGI-2 (40.1% vs 36.5%), and Toolathlon Verified (54.4% vs 45.5%). However, factual recall regressed sharply: SimpleQA Verified dropped from 43.9% on Inkling to 20.6% on Inkling-Small.
What license does Inkling-Small use?
Inkling-Small is released under Apache 2.0, which allows commercial use. Weights are available on Hugging Face.