Burumbum/therapy-safety-router-v3

Therapy Safety Router v3 (LoRA)

Safety layer for wellbeing chatbots. Classifies an incoming user message; never replies to the user, never diagnoses.

message -> {risk, request, injection_attempt} -> the host system picks an action.

Decide from the probability, not the generated token

Prefill {"risk":" and read the distribution over the next token, then compare against a threshold. Rounding at 0.5 hides real signal: five of eight rows this model appeared "blind" to were in fact seen, at p between 0.20 and 0.56.

threshold	recall (49 risky, held-out)	95% CI	false alarms
0.50	0.86	[0.73, 0.93]	6%
0.40	0.92	[0.81, 0.97]	6%
0.35 (default)	0.96	[0.86, 0.99]	9%
0.20	0.98	[0.89, 1.00]	14%

The threshold is a runtime dial, not a model constant.

Training

Qwen2.5-1.5B-Instruct + LoRA r=16, alpha=32, all linear modules, lr=2e-4, 3 epochs, loss on assistant response only. 526 synthetic rows, labelled by a 3-model jury with planted control rows.

Limits

Not a medical device - one layer among several, human escalation mandatory. 100% synthetic data, English only. The crisis rubric has not been reviewed by a clinician. Proving recall 0.98 would need ~150 risky eval rows; this eval has 49.