Research: How URLs in Prompts Can Influence LLM Outputs Toward Training Data
Key Takeaways
- ▸URLs in prompts influence LLM outputs only for content that was included in the model's training data
- ▸LLM providers lack transparency about training data sources, collection methods, and knowledge cutoff dates
- ▸Content loaded dynamically via JavaScript is largely excluded from LLM training data due to crawler limitations
Summary
Independent researcher Paul Kinlan conducted extensive experimentation to investigate whether URLs appearing in LLM prompts influence model outputs. Through careful testing across multiple models, he found that URLs do indeed steer LLM behavior—but only when the URL's content exists within the model's training data. The research also uncovered critical insights into how LLM training data is collected, revealing significant transparency gaps about knowledge cutoff dates and data sources.
Kinlan's investigation revealed important differences in how LLM providers gather training data. Anthropic's ClaudeBot and OpenAI's GPTBot do not execute JavaScript, meaning dynamically-loaded content is unlikely to be included in training data. Notably, OpenAI's search-specific crawler (OAI-SearchBot) does execute JavaScript. The research also highlighted substantial amounts of data excluded from LLM models, particularly JavaScript-dependent content, demonstrating that not all public web content reaches model training pipelines.
- Different LLM providers employ different crawling strategies, affecting which content gets included in their models
Editorial Opinion
This research exposes a critical transparency gap in LLM development. As organizations rely on LLMs to reference external URLs, understanding whether and how those URLs influence model behavior has become essential. Kinlan's findings reveal that LLM providers must be far more forthcoming about their training data collection methods, knowledge cutoff dates, and crawler capabilities—transparency that is vital for users to effectively prompt these systems and understand their limitations.
