Microsoft Azure Team Responds to MuJoCo Deployment Issues, Reveals Blobfuse2 and Download Mode Solution
Key Takeaways
- ▸Azure ML uses Blobfuse2 (a FUSE layer over Azure Blob Storage) rather than traditional CIFS, which explains filesystem behavior differences and symlink failures
- ▸Download Mode, which stages data to local NVMe before execution, is the recommended solution for ML workloads, providing native filesystem speed and compatibility
- ▸Azure Blob Storage or ADLS should be used as the backing store for Linux HPC and ML workloads instead of Azure Files, which is optimized for Windows lift-and-shift scenarios
Summary
Following the publication of a technical deep-dive into filesystem problems encountered while deploying DeepMind's MuJoCo on Azure ML, Microsoft's Azure team provided direct feedback addressing the root causes and recommended solutions. The original article, which gained traction on Hacker News, described issues with broken symlinks, metadata latency, and storage constraints when running robotics AI training workloads in simulation. Microsoft clarified that Azure ML uses Blobfuse2—a FUSE layer translating filesystem operations into REST API calls—rather than traditional CIFS, and recommended leveraging Download Mode to stage data onto local NVMe before execution, which resolves most of the performance and compatibility issues previously encountered.
The Azure team's response outlined four key technical recommendations: recognizing the Blobfuse2 architecture and its limitations, utilizing Download Mode for staging data to fast local storage, switching from Azure Files to Blob Storage or ADLS as the backing store for ML workloads, and understanding the architectural differences between various Azure storage options. The practical impact of these recommendations is significant—Download Mode eliminates the performance penalties of FUSE-layer I/O operations and restores native filesystem behavior, including proper symlink support, making it a categorical rather than incremental improvement for workloads involving thousands of files and heavy enumeration.
- Direct engagement from Microsoft's technical team demonstrates responsiveness to community feedback and clarifies architectural decisions
