Researchers Develop Automated System for Generating and Resolving AI Forecasting Questions at Scale

Summary

A new research system uses LLM-powered web research agents to automatically generate and resolve high-quality forecasting questions for evaluating AI systems. The approach addresses a critical bottleneck in AI evaluation by replacing manual curation with an automated pipeline capable of producing diverse, real-world questions at scale.

The researchers generated 1,499 diverse forecasting questions and resolved them several months later, achieving approximately 96% verifiable question quality—exceeding the rate of Metaculus, a leading human-curated forecasting platform. The system also demonstrated 95% accuracy in question resolution. Testing with different LLM-powered forecasting agents revealed clear performance differentiation, with Gemini 3 Pro achieving a Brier score of 0.134, GPT-5 at 0.149, and Gemini 2.5 Flash at 0.179.

Beyond evaluation, the research demonstrates practical applications for improving forecasting performance. By evaluating a question decomposition strategy on the generated question set, researchers achieved significant improvements in Brier scores (0.132 vs. 0.141), showing how automated question generation can drive iterative refinement of forecasting approaches.

• Automated question generation enables direct improvement of forecasting methods through systematic evaluation and iteration

Editorial Opinion

This work represents an important advancement in AI evaluation methodology by automating a labor-intensive and previously bottlenecked process. The system's ability to exceed human-curated benchmark quality while operating at scale could accelerate the development and assessment of AI forecasting capabilities. The demonstration that forecasting performance varies significantly by model and improves with better decomposition strategies suggests this automated framework will become a valuable tool for both evaluation and iterative improvement of AI systems.