AI Chatbots Show Consistent Political Messaging But Unreliable Citations, Analysis Reveals
Key Takeaways
- ▸AI chatbots demonstrate high semantic consistency (0.94–0.95 similarity) in political candidate information despite using different wording and varying sources
- ▸Citation reliability is problematic: the same prompt generates different source lists across runs, with each model averaging 18.1 total citations per prompt but only 2.4 consistently cited core sources
- ▸Explicit citations significantly correlate with content accuracy, showing +12pp to +27pp higher similarity to source material when sources are cited versus not cited
Summary
A comprehensive analysis by Caucus AI reveals that leading generative AI chatbots deliver remarkably consistent substantive content about political candidates—with semantic similarity averaging 0.94–0.95 across responses—but cite wildly different sources for the same information. The study examined responses from multiple models to 15 identical prompts across various geographies and candidates, finding that while the meaning and information conveyed remains stable, the specific sources cited vary dramatically from run to run, with each prompt-model combination averaging 2.4 consistently cited core sources, 3.8 frequent sources, and 12.9 rare citations.
The research reveals that chatbots primarily draw from candidate websites, with an average cosine similarity of 0.67 to official campaign and government sites. Notably, when models explicitly cite a source, the correlation between cited content and response similarity increases substantially—by 12 percentage points for campaign sites and 27 percentage points for official government websites. However, citation patterns vary significantly by model: GPT cites candidate sites in only 35% of responses but shows the strongest content alignment when citing, while Grok cites most frequently at 68% but with smaller similarity gaps, suggesting less rigorous source attribution practices.
- Chatbots predominantly draw biographical information rather than policy positions from candidate websites, suggesting current limitations in their coverage of political issues
- Citation patterns vary by model: GPT shows selective but accurate citations (35% citation rate), while Grok cites more liberally (68%) with less rigorous source correlation
Editorial Opinion
This analysis exposes a critical tension in how AI chatbots handle political information: while they maintain consistent messaging that appears reliable to users, their citation practices are fundamentally unstable and often contradictory. The finding that explicit citations correlate with higher accuracy suggests models may be selectively attributing sources when confident, while omitting citations when drawing from less reliable or synthesized information. As AI becomes an increasingly primary source for political information, this citation instability represents a significant democratic risk—voters may receive consistent political narratives without realizing the fragility of their sourcing.
