No LLMs in the Loop: tale.fyi Aligns 800 Audiobooks to Text in 6 Days
Key Takeaways
- ▸tale.fyi aligned 800 audiobooks (8,000+ hours) to text in 6 days using forced acoustic alignment, not LLMs or transcription
- ▸Used Meta's Massively Multilingual Speech model with CTC for direct acoustic-to-text synchronization with 20ms word-level resolution and confidence scores
- ▸Overcame engineering challenges including memory constraints on long books, optional front matter handling, and quality evaluation gates
Summary
tale.fyi has completed the alignment of 800 audiobooks (8,000+ hours) to their corresponding text in six days—a technical achievement that employed forced alignment with Meta's Massively Multilingual Speech model rather than LLMs. Using CTC (Connectionist Temporal Classification), the system achieves word-level timing with 20ms resolution and outputs confidence scores for quality assurance. The approach differs from traditional transcription-then-align pipelines by working directly with acoustics and existing text, eliminating an error source while maintaining synchronization accuracy.
The engineering required solving several non-obvious challenges. Memory constraints crashed the aligner on certain large books, prompting development of streaming-from-disk techniques; books with optional front matter (narrators skipping prologues) needed special handling; and evaluation gates prevented poor alignments from shipping after early mistakes with books like Aesop's Fables. Each gate and technique was informed by direct testing against LibriVox audiobooks and their corresponding Standard Ebooks texts.
The result is now live on tale.fyi: users can seamlessly switch between reading and listening to public domain works, combining the browser's reading experience with LibriVox's volunteer-performed audiobooks. This achievement suggests that high-quality audiobook-text alignment at scale is achievable without LLMs in the pipeline—a notable engineering win for the open web.
- Enables seamless cross-device reading/listening sync on tale.fyi using public domain LibriVox audiobooks and Standard Ebooks
Editorial Opinion
This represents elegant engineering that transformed an open-source speech model into infrastructure for literature on the web. By using forced alignment instead of transcription, Cole's system eliminates an entire error source while gaining confidence scores that enable quality gates—a 'prefer abstaining' philosophy other AI infrastructure projects should emulate. Accomplishing this in six days using public domain content and existing ML models showcases the power of agentic development workflows.
