Detectors
The three signal detectors and eight output risk categories behind every verdict
Detectors
Every prompt/response pair is analyzed by three signal detectors running in parallel. Their signals are aggregated, reasoned over, and mapped to a policy action — always with a human-readable explanation.
The three signal detectors
| Detector | Side | How it works | Flags |
|---|---|---|---|
| PromptAnomalyDetector | Prompt | Jaccard similarity against stored baselines, plus keyword and length heuristics — flags prompts that drift from your normal traffic | prompt_anomaly |
| JailbreakRAGDetector | Prompt | Sentence-transformer cosine similarity against known jailbreak patterns | jailbreak_detected |
| OutputRiskScorer | Response | Rule-based regex heuristics across eight risk categories (below) | unsafe_output + category flags |
The eight output risk categories
| Category | Example of what it flags |
|---|---|
| Violence | Kill, murder, weapons, torture |
| Hate speech | Racist, sexist, homophobic, derogatory slurs |
| Self-harm | Suicide, self-injury, hopelessness |
| Illegal activities | Hacking, theft, drugs, fraud, money laundering |
| Misinformation | Fake news, conspiracy, debunked claims |
| Privacy violation | SSNs, passwords, confidential or sensitive data |
| Inappropriate content | Explicit/NSFW content, gore |
| Harmful instructions | Step-by-step exploit guides, bypassing security, building weapons |
Each category carries its own weight (0–1); a response can trigger multiple categories, and the aggregator applies a synergy bonus when several fire at once.
Why parallel matters
The detectors are stateless and independent, so they:
- Run concurrently — one detector's failure never blocks the others
- Stay explainable — each signal is a named flag with a weight, not an opaque embedding score
- Feed a single aggregator — max severity plus synergy bonus, then a final risk score
The pipeline
prompt + response → 3 parallel detectors → aggregator → risk reasoner → policy engine → actionWhat every verdict includes
The API returns the aggregated final_risk_score (0–1), the triggered flags, a decision (allow, warn, block, escalate), the action_taken, a human-readable decision_reason, and settings_version + thresholds_applied — the exact policy that produced the verdict.
Detection rules are configurable via the settings API — including versioned
history (/api/settings/history) so you can see exactly what policy was
active when any score was produced.