Research: Whole-Tree Rewriting Beats Granular Mutations for LLM-Based Data Editing, Regardless of Format
Key Takeaways
- ▸Whole-tree rewriting strategy beats granular mutation tools by 5.3 points overall and 33 points on multi-turn reference tasks, regardless of format
- ▸JSON and HTML formats are equally effective for LLMs in 2026; format choice no longer confers a meaningful advantage
- ▸Smaller models fail silently on multi-turn tool-calling by skipping follow-up edits entirely, not by losing node IDs—a failure mode whole-artifact rewrites avoid
Summary
An independent research study by Kevin Peckham benchmarking tree editing approaches for language models found that whole-tree rewriting significantly outperforms granular mutation tools, particularly for multi-turn interactions where models must reference their own previous outputs. The research tested four models from three vendors (Claude Sonnet 4.5, Claude Haiku 4.5, GPT-5.4, and Gemini 3.5 Flash) across 8,000 test cases with pre-registered hypotheses, comparing HTML vs JSON serialization formats and different editing strategies.
Contrary to the hypothesis that HTML's native fluency with language models would provide a measurable advantage, the research found JSON formatting was equally accurate: both formats achieved first-pass validity above 99.3% and tied on exact-answer structural reading tasks (87.1% vs 87.9%). The format proved irrelevant; instead, the editing strategy was decisive. Whole-tree rewriting outperformed granular mutation tools by 5.3 points overall and 33 points on multi-turn tasks—a gap driven by smaller models failing to execute follow-up edits in chain-tool scenarios rather than stale-ID failures.
The rigorous experimental design included fair JSON implementations using RFC 6902, identical error handling and validation strictness across all conditions, and independently compiled JSON Schema verification. The study cost approximately $225 in API spend and exemplifies transparent benchmarking by publishing findings that partially contradicted the original hypothesis.
- Modern frontier and mid-tier models (Claude, GPT, Gemini) emit both JSON and HTML at >99% validity; format fluency stopped being a moat
Editorial Opinion
This research deserves attention for both its findings and methodology. The rigor of pre-registering hypotheses, committing all seeds and prompts before evaluation, and building a 'scrupulously fair' JSON baseline sets a valuable standard for AI benchmarking. Most important for practitioners: the finding that strategy outweighs format choice is pragmatically reassuring for teams locked into JSON or other serialization decisions. The hidden failure mode in multi-turn tool-calling—where smaller models simply never execute the second edit—should prompt teams relying on Claude Haiku or similar models for complex data manipulation to favor atomic, whole-artifact rewrites over fragile tool chains.
