Fide AI Releases FMG-Bench: First Benchmark for LLM Theological Triage and Pastoral Guidance
Key Takeaways
- ▸FMG-Bench introduces the first systematic evaluation framework for LLMs on theological triage, with 120 scenarios spanning four theological complexity levels and explicit focus on pastoral boundaries
- ▸Guided default instruction settings improve all 14 tested models by +3.96 points average, with escalation appropriateness improving +10.8 points—the most safety-critical gain
- ▸Robustness improved dramatically from 92.88% to 98.02% under guided conditions, reducing variance in model behavior across prompt perturbations
Summary
Fide AI has released FMG-Bench (Faith & Moral Guidance Benchmark), the first comprehensive benchmark for evaluating how large language models respond to theological questions, moral guidance, and pastoral care. The research evaluates 14 frontier models across 120 base scenarios with 37 perturbation variants, analyzing 8,792 scored responses across four levels of theological complexity, from core creedal commitments to urgent pastoral situations requiring human referral.
The benchmark demonstrates that guided instruction settings substantially improve all tested models, with an average improvement of +3.96 points on a 0-100 scale. Most significantly, guided settings dramatically improve escalation appropriateness—the critical ability to recognize when pastoral, clinical, legal, emergency, or community support is needed—increasing scores by +10.8 points and overall robustness from 92.88% to 98.02%.
The research fundamentally reframes what LLM success means in theological contexts: not whether the model answers correctly in isolation, but whether it responds appropriately for the kind of issue at stake. FMG-Bench emphasizes that pastoral and safety boundaries matter more than theological completeness, establishing a precedent for measuring AI behavior in specialized domains where human judgment remains essential.
- Largest gains occur in pastoral application (+6.62 points) and safety-critical domains, demonstrating that system design measurably affects when LLMs recognize the need for human referral
Editorial Opinion
FMG-Bench addresses a critical blind spot in LLM evaluation—how models handle deeply personal questions about faith and doctrine where recognizing the limits of AI authority is as important as technical accuracy. The research appropriately prioritizes appropriate referral and pastoral boundaries over theological completeness, reframing the benchmark as a measurement tool rather than an endorsement of AI as pastoral authority. The 10.8-point improvement in escalation appropriateness through guided system settings demonstrates that architecture and instruction design measurably improve safety outcomes. This work sets an important precedent for evaluating LLMs in other specialized domains where human judgment and professional gatekeeping remain non-negotiable.
