Soofi Open-Source Foundation Model Releases Complete Training Code
Key Takeaways
- ▸Complete training code released for Soofi S, a 30B open-source foundation model based on NVIDIA's Nemotron 3 Nano architecture
- ▸Comprehensive documentation of full pipeline: preprocessing, pretraining on 20 trillion tokens, and specialized midtraining phases
- ▸Includes advanced techniques like long-context training and multiple annealing stages for model optimization
Summary
Soofi has released the complete training code for its open-source 30B foundation model. Built on NVIDIA's Nemotron 3 Nano architecture, this comprehensive release provides the community with full transparency into how large-scale foundation models are trained.
The repository includes scripts and documentation covering the entire pretraining pipeline: corpus construction, filtering, tokenization, and mixture assembly. Detailed code is provided for preprocessing, pretraining on 20 trillion tokens, and multiple midtraining phases, including long-context training and three separate annealing stages.
With model weights coming soon, this release represents a significant contribution to the open-source AI community, offering practical insights into the training techniques and infrastructure decisions used in modern foundation model development.
- Open model weights coming soon
- Significant knowledge-sharing contribution enabling community understanding of large-scale model training
