Taking Clinical Decisions Out of the LLM: How Woebot Harnesses AI for Safe Therapy
Key Takeaways
- ▸LLMs should characterize input and render output, but never make clinical decisions—decisions with enumerable consequences belong in deterministic code
- ▸Prompt-based safety fails silently: emotional messages bypass instructions and produce fluent, confident clinical errors with no exception to catch
- ▸Crisis detection must run before generation, not after, to ensure acute risk moments never involve model-generated creative output
Summary
Woebot Health has detailed a novel architectural approach to using language models in clinical therapy, where deterministic code—not the model—makes all consequential clinical decisions. The company's system implements a fixed pipeline separating the LLM's two roles: scoring client state and generating therapeutic responses. All clinical decision-making (selecting therapeutic techniques, determining depth of work, approval gates) happens through rule-based logic indexed by nine therapeutic schools and four client states, preventing the model from being 'talked out of' safety constraints by persuasive user inputs.
The system addresses a fundamental problem with prompt-based safety: emotional or persuasive messages cause LLMs to violate safety instructions silently, producing fluent, confident responses that are clinically inappropriate. Crisis detection runs before therapeutic reply generation—not after—through three stages: pure-code keyword pre-screening (≤50ms), zero-temperature classification (0.8–2.0s), and continuous risk re-evaluation. On crisis detection, generation is skipped entirely and a fixed message is sent, with clinical supervisors immediately notified via email and live console alerts. This ensures acute risk moments never involve model-generated creative output.
The approach exemplifies a broader principle for high-stakes AI deployment: decisions with enumerable consequences should not be delegated to language models. Instead, LLMs should characterize input and render output while deterministic systems make substantive choices. For therapy, this means the model never sees the clinical decision table and cannot override safety guardrails through persuasion—because those decisions have already been computed.
- A deterministic state machine prevents users from bypassing safety constraints through persuasion, because decisions depend on measured state, not presentation
Editorial Opinion
This is sound engineering for high-stakes applications. The insight that clinical safety must be deterministic—that you cannot outsource judgment calls to a model trained to be agreeable—is perhaps the clearest articulation yet of where LLMs do and don't belong in regulated settings. The three-stage crisis pipeline and deliberate use of fixed messages on risk triggers represent best practices that should become industry standard for any high-consequence AI deployment.
