AMD’s Instella-MoE-16B: Open Weights, Research License, Real Benchmarks

AMD released Instella-MoE-16B-A3B on August 1, 2026, a fully open Mixture-of-Experts language model trained from scratch on its own Instinct MI300X and MI325X data-center GPUs. The model carries 16 billion total parameters but activates only 2.8 billion per token, keeping inference cost low. AMD is publishing weights from every training stage, data mixtures, training configs, and inference code. The catch: weights are under a ResearchRAIL license, meaning commercial deployment is not permitted. The MIT-licensed training codebase is the more reusable asset for most teams.

What happened

Detail	Value
Total parameters	16B
Active parameters per token	2.8B
Training hardware	AMD Instinct MI300X and MI325X
Pre-training tokens	7.1 trillion
Context window	64K tokens (extended from 4K)
Weight memory (BF16)	~32 GB
Weight license	ResearchRAIL (research only)
Training code license	MIT
Base model average score	76.7
Think model average score	73.22

Instella-MoE-16B-A3B is a decoder-only model with 27 layers, a hidden size of 2048, 16 attention heads, and a 128,896-token vocabulary. Each MoE layer routes tokens through 2 shared experts plus 6 of 64 available routed experts, which is how the 2.8B active parameter count is achieved from a 16B parameter pool.

Two architectural choices worth knowing

AMD introduced two structural additions that distinguish this release from prior MoE designs.

• Gated MLA (Multi-head Latent Attention): A lightweight learned output gate is added to standard Multi-head Latent Attention. A dedicated linear projection produces an input-conditioned gate value, which is applied multiplicatively before the output projection. This refines how attention outputs are blended.
• FarSkip-Collective: Outdated and partial activations are passed into MoE and attention layers while expert-parallel communication overlaps with computation. AMD reports a 12.7% pre-training speedup and up to a 39.2% reduction in time to first token during serving.

How the training pipeline was built

Pre-training consumed 7.1 trillion tokens drawn from open corpora: Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360. A Multi-Token Prediction objective was used throughout pre-training and mid-training.

Mid-training used Dolma3 Dolmino 100B across three data variants, merged by weight averaging. A long-context stage then extended the window from 4K to 64K tokens using YaRN, an increased RoPE theta, and document masking.

Post-training ran supervised fine-tuning on Dolci-Think-SFT-7B plus Nemotron mixtures, finishing on a feedback-driven 512K-example set targeting measured weaknesses. Direct Preference Optimization (DPO) followed, with router bias updates and the auxiliary load-balancing loss disabled to prevent degradation. Reinforcement learning ran 1,400 steps of instruction-following RLVR in AMD’s Miles framework, then Multi-Teacher On-Policy Distillation folded gains back without losing math or code performance.

How does Instella-MoE benchmark against open peers?

Model	Base Average
Qwen3.5-4B-Base	79.5
Instella-MoE-16B-A3B (base)	76.7
Moonlight-16B-A3B	76.2
SmolLM3-3B-Base	70.5
OLMo-3-7B	70.1
OLMoE-1B-7B	61.9

Instella-MoE leads the fully open model group but trails Qwen3.5-4B-Base. Specific scores include 86.5 on WinoGrande and 65.7 on HumanEval+. Long-context performance is 41.5 on HELMET and 79.4 on RULER.

Post-training stage	Average score
SFT	71.58
DPO	72.67
Think (RL + distillation)	73.22

The Think checkpoint’s IFEval score (instruction-following evaluation) climbed from 77.08 to 83.70 after the full post-training pipeline, and it outscores OLMo3-7B-Think (71.97), Gemma-4-E4B Think (70.47), and Qwen3.5-4B (69.73).

Why it matters

Most MoE model releases drop weights with little context on how the training was actually run. AMD is releasing the full recipe: weights at each stage, data compositions, configs, and the SGLang inference code. That transparency is useful for anyone trying to reproduce or extend an MoE training run.

The hardware angle is also deliberate. AMD is demonstrating that its MI300X and MI325X GPUs, which compete with NVIDIA’s H100 and H200 in data-center settings, can train a competitive frontier model end-to-end. That matters to teams evaluating whether to build on ROCm rather than CUDA.

The ResearchRAIL license is a hard wall for commercial teams. You cannot drop these weights into a customer-facing product without violating the license terms. But the MIT-licensed training code is genuinely open, and teams with GPU capacity could use it as a reference to train their own model on their own data.

Our take

This is a meaningful research release, not a product. AMD is targeting AI research labs, university groups, and enterprise R&D teams that have data-center GPU capacity and want to study expert-parallel training at scale. It is not a fit for a lean team looking for a hosted commercial endpoint.

What is actually useful here: the training code, the published data mixtures, and the detailed post-training pipeline. If you are a team running your own fine-tuning experiments or studying how RL post-training interacts with MoE routing, this gives you a concrete, reproducible reference. The FarSkip-Collective speedup numbers (12.7% training, 39.2% lower TTFT) are worth benchmarking independently, because AMD is measuring on its own hardware with its own stack.

For context on how open-source model releases have been reshaping competition between labs, our earlier coverage of OpenAI vs Anthropic and the shifting AI leaderboard is relevant. If your team is evaluating whether to integrate any MoE model into a production workflow, our AI integration services cover exactly that kind of scoping and deployment work.

What to do about it

1. Check the ResearchRAIL license terms before experimenting, specifically confirm your use case qualifies as academic or research.
2. Clone the MIT-licensed training repo from AMD’s GitHub if you want to study or adapt the training pipeline.
3. Pull the weights from the Hugging Face collection for evaluation runs; 32 GB BF16 weight memory means a single high-memory accelerator is enough.
4. Run your own HELMET and RULER evaluations at 64K context to verify AMD’s reported long-context numbers on your data.
5. Watch for a commercial-licensed version. AMD has not announced one, but the research release often precedes it.

The training code is the real takeaway here: open, MIT licensed, and one of the most detailed MoE recipes published to date.

Source: Marktechpost