Covenant-72B: First Large-Scale LLM Trained Through Trustless Distributed Network With Open Participation

Summary

Researchers have unveiled Covenant-72B, a 72-billion parameter large language model trained through the largest collaborative globally distributed pre-training run to date, featuring open, permissionless participation enabled by blockchain technology. The model was pre-trained on approximately 1.1 trillion tokens using SparseLoCo, a communication-efficient optimizer that supports dynamic participation with peers freely joining and leaving the network. Unlike previous distributed training efforts limited to whitelisted participants, Covenant-72B demonstrates that truly democratized, non-whitelisted participation in foundation model development is feasible at scale. The model achieves competitive performance compared to fully centralized models trained with similar or higher compute budgets, marking a significant milestone in decentralized AI development.

• Competitive performance with centralized models demonstrates that democratized distributed training can achieve state-of-the-art results at unprecedented scale

Editorial Opinion

Covenant-72B represents a watershed moment for democratizing AI development, proving that large-scale foundation models can be trained collaboratively across untrusted networks without sacrificing performance. By combining blockchain-based governance with sophisticated optimization techniques, this work challenges the centralized AI paradigm and opens new possibilities for global participation in building AI infrastructure. However, questions remain about the practical accessibility, computational requirements for participation, and how well this model scales as additional untrusted peers join the network.