Fusion Embedding 1: Open-Weight Multimodal Model Beats Gemini Embedding 2 with Only 16M Trained Parameters
Key Takeaways
- ▸Achieves state-of-the-art on audio↔text and emergent audio↔image retrieval, outperforming Gemini Embedding 2, ImageBind, and LanguageBind with a frozen base model and only 16M new parameters
- ▸Demonstrates emergent cross-modal alignment: training exclusively on audio-text pairs yields strong audio-image retrieval (R@10 0.407) with zero audio-visual training data
- ▸Open-source release enables reproducible research and practical deployment on resource-constrained hardware, with flexible output dimensions and compact distribution
Summary
A new open-weight multimodal embedding model, Fusion Embedding 1, outperforms Google's Gemini Embedding 2 on audio-text retrieval benchmarks while training only a 16M-parameter connector. Built by extending Qwen3-VL-Embedding-2B, the model leaves all base weights frozen and adds a perceiver-resampler adapter that maps frozen audio features into the base model's shared embedding space for text, images, video, and audio.
The key innovation is training efficiency: a small contrastive learning stage (484K audio-text pairs) plus fine-tuning on AudioCaps creates a unified embedding space where retrieval works across any modality pair. Remarkably, audio-image retrieval emerges without any audio-visual training data—audio-text alignment places the audio modality into a space the base model already shares across text, image, and video. The model supports Matryoshka truncation from 2048 down to 64 dimensions and is released with full open weights and inference code (~60MB for the adapter).
Editorial Opinion
This work is a reminder that model scale is not destiny—clever architecture and frozen base leveraging can beat end-to-end training on specialized tasks. The insight of using a small adapter to map one frozen tower into an existing multimodal space is pragmatic and reproducible, making it immediately useful for teams building retrieval systems. The open-source release and strong cross-modal performance make this a genuinely valuable contribution.
