ActPlane Study: AI Agent Rules Require Context and Layered Enforcement to Stick
Key Takeaways
- ▸64% of statements in AI agent instruction files are behavioral policies; the remaining 36% is descriptive context like architecture notes and project background
- ▸Development Process and Implementation Details are the dominant policy concerns (87% and 85%), reflecting how developers actually instruct their AI agents
- ▸Most policies depend on multi-layered context (repository state, task history, file changes) that exceeds what single OS-level hooks can enforce
Summary
A comprehensive empirical study by ActPlane analyzing 2,116 policy statements across 64 popular repositories reveals a critical gap between the behavioral rules developers write for their AI agents and what systems can actually enforce. The analysis of CLAUDE.md and AGENTS.md instruction files shows that while 64% of statements are actionable policies (requiring, forbidding, or conditioning specific agent actions), most rules depend on contextual state—repository structure, task progress, file modifications, and prior events—that goes beyond what a single OS-level hook can monitor.
The research found that Development Process and Implementation Details dominate policy concerns at 87% and 85% respectively, far exceeding Architecture or other categories. Critically, most policies cannot be enforced through semantic analysis alone or simple content-level checks; they require layered enforcement mechanisms that track state changes over time. For example, a rule like "run the full test suite before committing" appears simple until an AI coding agent edits source files after the last test run, creating a dependency between observing the edit, tracking test validity, and enforcing the commit policy.
The study shifts focus in AI agent safety from abstract threat modeling to a concrete question: how can developers' natural-language requirements be translated into observable, enforceable system state? ActPlane's statement-level analysis, validated through stratified human review and cross-checked by independent agents, demonstrates that enforcement capability—not developer intent—is the bottleneck in AI agent governance.
- The enforcement gap between developer intent and system capability highlights a critical vulnerability in current AI agent governance and safety mechanisms
- Statement-level analysis reveals policy density varies 0–97% across repositories, showing the importance of fine-grained classification over file- or heading-level studies
Editorial Opinion
ActPlane's fine-grained analysis fills a critical gap in AI agent safety research. By moving from file-level abstractions to statement-level classification across real, production repositories, the study proves that developers are not short of rules—the bottleneck is enforcement. The finding that context-aware, multi-layer enforcement is needed for 64% of policies has immediate implications for how Anthropic and other AI platforms should architect their agent safety systems. This work shifts accountability from "Did the developer write the rule?" to "Can the system actually check it?"—a harder but far more actionable question.
