Revolut and NVIDIA Publish PRAGMA, the Largest Foundation Model for Banking
Key Takeaways
- ▸PRAGMA is the largest encoder backbone for consumer banking event data published to date, trained on 26M user histories and 207B tokens across 111 countries—demonstrating that enterprise fintech companies now have the data and compute to train foundation models
- ▸The model achieves 130% improvement on credit scoring, 164% on engagement, and 40% on recommendations, with LoRA fine-tuning matching or exceeding task-specific baselines—validating the business case for pre-trained representations in banking
- ▸The technical architecture reveals banking's shift from hand-crafted features to learned representations: PRAGMA's tokenization strategy (semantic key, typed value, temporal coordinate) preserves numerical magnitude and contextual information while managing sequence length
Summary
Revolut Research and NVIDIA have published PRAGMA, an encoder-only transformer foundation model trained on 26 million user histories spanning 24 billion events and 207 billion tokens across 111 countries. The model represents a significant shift in financial services technology: moving from hand-crafted tabular features and gradient-boosted-tree models to learned representations trained on raw transaction data. PRAGMA scales from 10 million to 1 billion parameters and uses a novel input representation that preserves numerical magnitude information and temporal context while keeping sequence length manageable.
The technical approach is tailored specifically for banking's discriminative tasks. Rather than GPT-style text generation, PRAGMA uses BERT-style masked language modeling with three masking strategies: standard token masking (15%), whole-event masking (10%), and semantic-type masking (10%). Numerical values are mapped to learned percentile buckets, categorical values to single tokens, and timestamps encoded both as compressed log-seconds and periodic sinusoids. The model achieves substantial improvements across downstream tasks: 130.2% relative improvement on credit scoring (PR-AUC), 163.7% on communication uplift (AUUC), and 40.5% on product recommendation (mAP). Fine-tuning with LoRA—updating only 2-4% of parameters—consistently matches or exceeds training task-specific models from scratch.
The publication of PRAGMA signals a paradigm shift in banking technology: foundation models trained on transaction data are moving from closely-guarded trade secrets to published research. Revolut's decision to disclose results under author affiliations suggests internal foundation models have become a competitive disclosure point in the industry, not a hidden advantage. This work also echoes Nubank's earlier nuFormer publication, indicating a broader shift in fintech's approach to representation learning.
- Publishing PRAGMA with author affiliations marks a turning point: internal foundation models in fintech are moving from trade secrets to competitive disclosures, similar to Nubank's nuFormer
Editorial Opinion
PRAGMA represents a maturation moment for enterprise AI: large fintech firms now have the scale and resources to train competitive foundation models where most teams lack the data, GPU budget, or organizational will. The publication of results under Revolut's name is perhaps more significant than the technical contributions—it signals that foundation models have become table stakes for consumer financial services, not a hidden edge. However, the absolute improvements are commercially redacted and presented as relative gains against internal baselines, so real-world impact should be calibrated carefully.
