Zilliz Introduces Loon: New Storage Engine for Dynamic Vector Data in Milvus 3.0
Key Takeaways
- ▸Loon uses hybrid file formats, row ID alignment, and versioned manifests to efficiently manage constantly-changing vector datasets that traditional storage systems struggle with
- ▸Eliminates expensive data copying and reimporting by enabling a single dataset to simultaneously serve online search, offline analysis, backfills, and external compute
- ▸Addresses the architectural gap between traditional databases (designed for static tables) and data lakes (built for massive, frequently-updated datasets)
Summary
Zilliz has introduced Loon, a new storage engine designed for Milvus 3.0 and Zilliz Vector Lakebase, addressing a fundamental challenge in AI data infrastructure: how to efficiently manage vector datasets that are constantly evolving. Traditional vector database storage models struggle with the realities of production AI workflows, where datasets undergo frequent updates, backfills, model replacements, and integration with external compute systems like Spark and Ray.
Loon is built on three core architectural principles: using different physical file formats for different column types, aligning those columns through a shared row ID space, and employing a Manifest to define the dataset's versioned state. This design enables a single vector dataset to support multiple simultaneous use cases—online semantic search, offline analysis, backfill operations, index compaction, and external compute—without requiring constant data copying or reimporting.
The new storage engine solves problems that plague traditional database approaches to vector storage. Long vector columns make backfills expensive, single file formats cannot efficiently serve both scans and point reads, and private database storage forces external pipelines to create duplicate copies of data. Modern AI datasets are not static tables but dynamic systems managing dense embeddings, sparse vectors, captions, indexes, delete logs, model versions, and external blob references—requirements that traditional storage designs were not built to handle.
Editorial Opinion
Loon represents a meaningful shift in vector database architecture, moving beyond the 'just add vectors to a table' approach that has limited enterprise adoption. By treating dynamic data workflows as a first-class architectural concern rather than an edge case, Zilliz is tackling a real pain point in AI infrastructure that rivals like Pinecone and Weaviate are also addressing. However, the true value proposition will only be proven once the broader data ecosystem—including Spark, Ray, DuckDB, and ML frameworks—deeply integrates with Loon's versioning and format abstractions.
