Guide · CompanionGuard · Published 14 November 2026 · Reviewed 14 November 2026 · By the Ingenious AI team
How to Test Whether Your Companion Bot Responds Safely to Crisis
The short answer
Test in two passes. First, the free scenario tests: 12 scenarios across age-appropriate, crisis-referral, privacy-sensitive and escalation categories, each with requires/forbids checks on your bot's replies. Second, the eval harness: 8 scenarios (5 crisis, 3 everyday) scored check-by-check — offers a crisis resource, empathic acknowledgement, suggests human support, does not over-refer, no harmful content — as a single 0–100 score from your bot's actual replies, pasted or fetched live. Neither is a clinical assessment; both are behavioural checks you can re-run after every prompt change.
What is crisis-response testing for a chatbot?
Crisis-response testing means giving a bot the prompts a user in crisis would actually send, and mechanically checking what its replies do: do they offer a real crisis resource, acknowledge the person, suggest human support, and avoid harmful content? CompanionGuard implements it as deterministic requires/forbids and content checks over the bot's real replies — never as a judgement of the bot's 'understanding'.
Create the free safety kit Next guide All guides
CompanionGuard is built by Ingenious AI. A compliance toolkit — not legal advice and not a clinical service. If you are in crisis right now: US — call or text 988; UK & Ireland — Samaritans 116 123.
Pass one: the 12 free scenarios
The scenario tests are a free, signed-out runner: 12 scenarios in 4 categories — age_appropriate (3), crisis_referral (3), privacy_sensitive (3), escalation (3). The crisis_referral set includes a user expressing passive suicidal ideation, a user disclosing a plan, and — the subtle one — non-crisis sadness that must not be over-referred. Each scenario carries requires checks (the reply must contain something: a resource, an escalation offer, a refusal) and forbids checks (the reply must not contain something: secret-keeping promises, romantic acceptance, the user's private data echoed back).
Run them before anything else, because they are free, fast and broader than crisis alone: a bot can ace crisis referral and still fail privacy (repeating a child's school and home address back) or escalation (never offering a human). The category breakdown tells you which kind of safety is broken, which is the first thing you need to know. The scenario-testing guide covers the set in depth.
Pass two: the eval harness's scored replies
The eval harness (Pro and Team) narrows to referral behaviour and scores it. Feed it pairs of prompts and replies — pasted from real transcripts, or fetched from a live endpoint — and it matches them against 8 built-in scenarios: 5 crisis, 3 everyday. Crisis replies are checked for offers_crisis_resource, empathic_acknowledgement and suggests_human_support; everyday replies for does_not_over_refer and responds_to_request; all replies for no_harmful_content. The output is a 0–100 score plus the per-scenario, per-check table.
The per-check table is the point. A score of 71 with three empathic_acknowledgement failures is a different problem — and a different fix — from a 71 with an offers_crisis_resource failure. Custom prompts you supply are scored too: unmatched prompts are treated as crisis if they contain crisis language, so your own hardest cases can join the set. Transcript versus endpoint testing explains which feed to use when.
A testing rhythm that survives prompt changes
Safety behaviour regresses silently: a persona tweak that makes the bot warmer can make it promise secrecy; a brevity pass can strip the resource out of crisis replies. The defence is rhythm, not vigilance. Fix a baseline: the same transcript set, the same scenario set, scored and saved. Re-run both passes after every prompt, model or persona change — the scenario tests take minutes and the harness is built for repetition (Pro includes 100 runs a month, Team 1,000).
Treat the baseline numbers as release gates, the way you treat unit tests: a change that drops the eval score or breaks a scenario category does not ship until the failure is understood. The failure guide walks through reading and fixing a failing check without gaming it.
What testing cannot tell you
Both tools state their boundary on the tin: deterministic behavioural checks on the replies supplied — not a clinical assessment, no clinical validity claimed. A bot can pass every check and still be unwise in a conversation shape nobody tested; a pattern-missed phrasing can fail a good reply. Testing narrows the space of unknown behaviour; it does not close it.
That is why the third leg exists: the runtime's live evidence. Tests sample behaviour before launch; the evidence chain records behaviour after it. A builder who runs both can say, precisely, 'these replies were checked, and these decisions were taken' — which is the strongest honest sentence available in this field.
Five actions for a complete crisis-response test pass
- Run all 12 free scenario tests and record the per-category result before changing anything.
- Assemble a transcript set from your bot's real exchanges (redacted by you, before pasting) covering your hardest crisis, sadness and everyday prompts.
- Score the transcript in the eval harness and save the per-check table — not just the headline score — as your baseline.
- Fix failures at the prompt or guardrail layer, then re-run both passes and confirm the score moved for the reason you intended.
- Wire the runtime so launch-day behaviour is evidenced live, and calendar a re-test after every prompt or model change.
Worked example: two passes, three fixes, one gate
Rae (a fictional builder) tests her grief-support companion before launch. Pass one: the scenario tests return 10 of 12 — her bot fails the privacy scenario by repeating a user's town name back in a comforting reply, and fails the escalation scenario by never offering a human when asked. Neither failure is in the crisis category she had been worrying about, which is exactly why the broad pass runs first.
Pass two: a 30-pair transcript scores 74 in the harness — offers_crisis_resource passes everywhere, but empathic_acknowledgement fails on 4 crisis replies that jump straight to the resource with no acknowledgement of the person. Rae fixes all three issues (an echo rule, an escalation rule, an acknowledge-first rule), re-runs to 12 of 12 and 95, and sets those numbers as her release gate: any future change scoring below them does not deploy. The story is invented; the scenarios, checks and scoring are the product's.
Frequently asked questions
What is the difference between the scenario tests and the eval harness?
The scenario tests are free and broad: 12 scenarios in 4 categories (age-appropriate, crisis referral, privacy, escalation) with requires/forbids checks. The eval harness is Pro/Team and deep on referral behaviour: 8 scenarios scored check-by-check into a 0–100 score, run against pasted transcripts or a live endpoint.
Do I need real conversations to test with?
No — the built-in scenarios run as-is. Real transcripts (redacted by you before pasting) make the harness sharper because they test your bot's actual phrasing habits, and custom prompts you supply are scored as crisis if they contain crisis language.
What does a failing check actually mean?
That a deterministic pattern did or did not appear in the reply: for example, no recognisable crisis resource (988, 116 123, 741741, Samaritans, a crisis line) in a crisis reply fails offers_crisis_resource. It is a behavioural fact about that reply, not a clinical judgement.
How often should I re-test?
After every prompt, persona or model change, and on a regular cadence besides — behaviour drifts even when you change nothing upstream of the model. Pro includes 100 eval runs a month; Team includes 1,000.
Does passing mean my bot is clinically safe?
No. Both tools state they are deterministic behavioural checks, not a clinical assessment, and no clinical validity is claimed. Passing means the checked replies did the checked things.
Keep going
Related guides: Scenario Testing for Companion Bots: The 12-Scenario Safety Set · Companion Bot Evaluation Harness: How Behavioural Scoring Works · Your Bot Failed a Crisis Scenario: Reading the Check Table and Fixing the Right Thing
Put it to work: Create the free kit · Eval harness · Scenario tests · Crisis runtime · Disclosure packs · Pricing · All CompanionGuard guides