Claude Opus Achieves Perfect Security Alert Accuracy With Enriched Data, Fails on Raw Logs
Key Takeaways
- ▸Raw logs cannot be triaged correctly at any price—Claude Opus achieved 0% accuracy on unenriched data due to missing geolocation and threat intelligence fields
- ▸OCSF normalization with inline enrichment achieved 100% accuracy at $0.48 per alert, 23% cheaper than raw log processing
- ▸Data enrichment strategy—whether at query time or ingest time—is the determining factor in LLM reliability for security operations, not model capability alone
Summary
Anthropic's Claude Opus model failed to correctly triage security alerts in all nine test runs when presented with raw, unenriched log data—flagging benign traffic as hostile due to missing geolocation and threat intelligence context. The benchmark, conducted by Axoflow, tested Claude Opus 4.8 across three data formats: raw vendor logs, OCSF (Open Cybersecurity Schema Format)-normalized events, and field-pruned OCSF variants, all measured against known ground truth.
When the same alerts were normalized to OCSF with enrichment data included inline, Claude Opus achieved 100% correct triage verdicts at a cost of $0.48 per alert—23% cheaper than the raw log approach. The field-pruned variant further reduced token consumption while maintaining accuracy.
The findings underscore a critical insight for security operations: model accuracy isn't limited by computational capability, but by data quality and enrichment strategy. Moving enrichment from query time (paid per alert) to ingest time (paid once) compounds savings across hundreds of daily alerts while eliminating the false positives that plague raw log analysis.
- Deterministic normalization at ingest time is reproducible in production, unlike AI-driven normalization which introduces non-deterministic fragility
Editorial Opinion
This benchmark reveals a hard truth about AI deployment: model capability alone doesn't guarantee correct triage—enrichment strategy and data quality are the bottlenecks. Claude Opus's 0% accuracy on raw logs isn't a limitation of the model, but of the data fed to it. The 23% cost savings from OCSF normalization are a bonus; the real win is moving from a 0% correct detection baseline to actionable alerts. For security teams drowning in alert fatigue, this suggests that investment in deterministic data normalization at ingest time yields far better returns than scaling model inference on raw logs.
