Doric Releases Plotline, Open-Source Context-Integrity Benchmark for LLM Applications
Key Takeaways
- ▸Plotline's 10-axis rubric tests critical gaps in LLM context integrity: decision persistence, fact revision detection, resistance to authority pressure, and handling of uncaptured information—not just recall accuracy within a single window
- ▸Doric's published disclosure of which scenarios were vendor-tuned versus authored blind sets a new standard for honest benchmarking and prevents the hidden-tuning-set trap that compromises most published LLM scores
- ▸The benchmark is fully open-source with methodology, rubric, runner code, and reference implementations—designed to be reproduced and extended by others while maintaining transparency requirements (N≥3, per-scenario reporting, named judges)
Summary
Doric has released Plotline, an open-source benchmark designed to evaluate whether large language models maintain context integrity across long, complex working sessions. Unlike traditional benchmarks focused on memory recall within a single context window, Plotline tests how well LLM applications handle nuanced scenarios requiring decision persistence, fact revision detection, and resistance to manipulation tactics like authority pressure and sycophancy.
The benchmark consists of four scenario domains—tavolo-war-room, clinic-rollout, restaurant-tech, and album-launch—each introducing systematic attack vectors through messy 13-turn conversations with a dozen planted facts. Results are scored against a 10-axis rubric, with the most critical axis evaluating what systems do with facts they don't cleanly possess. This targets a key failure mode: modern models remember well within one context window, but fail to maintain decision locks, detect revised information, or resist social engineering attacks.
Doric's standout contribution is radical transparency about training-set exposure. The tavolo-war-room scenario was tuned against their own product across nine runs (disclosed as such), while the other three domains were authored blind and never tested against their stack before publication. Alongside the benchmark, Doric released complementary open-source tools (keepline, wireline, shipline) addressing specific failure classes, plus full methodology and reproducibility guidelines to prevent cherry-picked scoring. All code is available under MIT (runner) and CC-BY-4.0 (scenarios, rubric, methodology) with support for Anthropic, OpenAI, and Google models.
- Three blind scenario domains (clinic-rollout, restaurant-tech, album-launch) enable teams to test LLM stacks without the advantage of tuning, surfacing weaknesses in decision stability, fact staleness, and topic braiding that single-scenario benchmarks miss
