AMD Introduces Instella-MoE: Fully Open Mixture-of-Experts Language Model Trained on AMD Hardware
Key Takeaways
- ▸AMD releases fully open, state-of-the-art Mixture-of-Experts language model (16B total / 2.8B active parameters) with competitive performance across benchmarks
- ▸Model trained end-to-end on AMD Instinct MI300X/MI325X GPUs with ROCm software stack, demonstrating AMD hardware viability for large-scale LLM training
- ▸Complete training pipeline released open-source: all model weights from every stage, training configs, data mixtures, and code using Primus and Miles frameworks
Summary
AMD has announced Instella-MoE, a state-of-the-art fully open Mixture-of-Experts language model featuring 16 billion total parameters with 2.8 billion active parameters per token. Trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs using the AMD ROCm software stack, the model incorporates architectural innovations including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity. The model demonstrates competitive performance across benchmarks against both dense and MoE baselines of comparable or larger active parameter counts.
In line with AMD's commitment to open AI development, the company is releasing all artifacts and checkpoints spanning the complete training pipeline—including pre-training, mid-training, long-context extension, supervised fine-tuning, direct preference optimization, and reinforcement learning stages. The release includes model weights, training configurations, data mixtures, and code, all built using the open-source Primus and Miles frameworks. This comprehensive release aims to accelerate community research and reproducibility in open language models while showcasing the scalability and efficiency of AMD's hardware and software stack for large-scale MoE training and inference.
- Architecture features Gated Multi-head Latent Attention (Gated MLA) and system-level innovations optimizing both training and serving efficiency
Editorial Opinion
Instella-MoE represents a significant step in AMD's strategy to establish itself as a credible alternative to NVIDIA for AI model development, proving that competitive state-of-the-art models can be trained on AMD's proprietary hardware stack. By committing to full open-source release—including weights from every training stage—AMD demonstrates confidence in the model's quality while contributing meaningfully to the open AI ecosystem. The MoE architecture and multi-stage training approach suggest AMD is not just catching up to existing models but advancing the frontier with novel architectural choices. This release could inspire broader adoption of AMD hardware among research institutions and smaller organizations seeking alternatives to NVIDIA's ecosystem.
