Elastic Looped Transformers Achieve 4x Parameter Reduction for Visual Generation

Summary

Researchers have introduced Elastic Looped Transformers (ELT), a novel parameter-efficient architecture for visual generation that dramatically reduces model size while maintaining synthesis quality. The approach replaces conventional deep stacks of unique transformer layers with iterative, weight-shared transformer blocks, achieving a 4x reduction in parameter count compared to standard models under equivalent inference-compute settings. To enable effective training of these recurrent models, the team developed Intra-Loop Self Distillation (ILSD), a technique where intermediate loop configurations are distilled from the maximum training configuration in a single training step, ensuring consistency across the model's depth. The framework produces a family of elastic models from a single training run, enabling Any-Time inference with dynamic computational trade-offs while maintaining the same parameter count. ELT achieves competitive results on standard benchmarks, reaching an FID of 2.0 on class-conditional ImageNet 256×256 and an FVD of 72.8 on class-conditional UCF-101 video generation.

• Any-Time inference capability allows users to trade off computational cost and generation quality dynamically with identical model parameters

Editorial Opinion

This research represents a significant advancement in parameter-efficient visual generation, addressing a critical challenge in deploying large generative models. The novel combination of weight sharing with self-distillation is elegant and could inspire broader adoption of similar efficiency techniques across the generative AI landscape. The ability to extract multiple elastic models from a single training run is particularly promising for practical deployment scenarios where computational constraints vary.