PRAGMA: Revolut Foundation Model Brings Transformer Architecture to Banking Event Sequences
Key Takeaways
- ▸PRAGMA is a family of Transformer-based foundation models pre-trained on large-scale banking event sequences using self-supervised masked modeling
- ▸The model achieves superior performance on multiple downstream financial tasks including credit scoring, fraud detection, and lifetime value prediction
- ▸PRAGMA enables efficient transfer learning, with simple linear models on extracted embeddings delivering strong results and lightweight fine-tuning providing additional improvements
Summary
Revolut has unveiled PRAGMA, a family of foundation models specifically designed for analyzing multi-source banking event sequences. The model uses a Transformer-based architecture pre-trained with masked modeling on large-scale, heterogeneous banking data, leveraging self-supervised learning tailored to the discrete, variable-length nature of financial records. This approach enables the extraction of rich economic signals from the vast quantities of transactional and event-level data that modern financial systems generate. PRAGMA demonstrates strong performance across critical downstream tasks including credit scoring, fraud detection, and customer lifetime value prediction. The research shows that effective results can be achieved through simple linear models trained on PRAGMA's extracted embeddings, with further improvements possible through lightweight fine-tuning, establishing it as a general-purpose representation layer for financial applications.
- The approach provides a general-purpose representation layer for financial applications by encoding rich economic signals from raw transactional and event-level data
Editorial Opinion
PRAGMA represents a significant step forward in applying foundation model architecture to the financial services domain. By designing the pre-training objective specifically for the characteristics of banking data—discrete, variable-length sequences—Revolut has created a more specialized and potentially more effective tool than generic language models adapted for finance. The demonstrated efficiency of achieving strong results with simple linear probes suggests the learned representations capture fundamental patterns in financial behavior, making this approach particularly valuable for institutions managing credit risk and fraud at scale.
