OpenAI's JSON Mode Silently Corrupts Accented Characters in Production
Key Takeaways
- ▸OpenAI's JSON mode incorrectly encodes escaped Unicode sequences as \u0000XX instead of \u00XX, injecting null bytes that corrupt data in PostgreSQL and other systems
- ▸The bug affects all non-Latin Unicode characters (accented Latin, CJK, Hindi, Cyrillic) when models attempt to use escape sequences matching prompt examples
- ▸Workaround: use raw UTF-8 characters in prompt examples instead of \uXXXX escape sequences to prevent the model from attempting the corrupted escape path
Summary
Researchers at Giskard discovered a critical bug in OpenAI's JSON mode where accented and non-Latin Unicode characters are incorrectly escaped, resulting in corrupted data that breaks downstream systems. When OpenAI's API attempts to escape characters like é as \uXXXX sequences, it produces \u0000e9 instead of \u00e9—a null byte followed by literal characters. After JSON parsing, the null byte becomes a control character that downstream systems like PostgreSQL reject, causing silent data corruption in production.
The issue affects any UTF-8 character requiring Unicode escape sequences, including accented Latin characters, CJK, Hindi, and Cyrillic text. While semantically the model could output raw UTF-8 directly without escaping, the problem arises when prompt examples use escaped sequences (Python's default), causing the model to imitate and attempt the escape, hitting this blocked endpoint constraint. The bug is undocumented and applies to both OpenAI's direct endpoints and Azure OpenAI's infrastructure.
The discovery highlights a broader tension in the industry's increasing reliance on LLM structured outputs as "trustable serializers." While JSON mode and Structured Outputs enforce correct outer structure—objects, arrays, required keys, and field types—they do not guarantee the semantic correctness of string content. Teams depending on these features for production database operations face silent failures where output appears valid but contains corrupted bytes.
- Structured Outputs guarantee syntactic JSON validity but not semantic correctness of content—teams cannot assume LLM-generated JSON is production-safe without additional validation
Editorial Opinion
This bug exposes a critical gap between OpenAI's marketing of JSON mode and Structured Outputs as reliable serialization tools and their actual production readiness. While the technical fix is straightforward (allow non-control-character completions after \u00), the lack of documentation and the silent nature of the corruption make this a serious reliability issue for teams building on LLMs. The finding underscores that AI-generated structured data still requires careful validation before hitting production databases.
