← Go back

Guide · CompanionGuard · Published 14 November 2026 · Reviewed 14 November 2026 · By the Ingenious AI team

After Something Goes Wrong: Running a Safety Incident Review for a Companion Bot

The short answer

A companion-bot safety incident — a harmful reply, a missed referral, a minor-protection failure — deserves a review process as engineered as the guardrails themselves. CompanionGuard supplies the instruments: the hash-chained evidence as the tamper-proof timeline of what the guardrails did, counts and class data for scope, the eval harness for reproducing the failure, and the transcript pack as the place the failure becomes a permanent test. This guide sequences them around the human decisions only your team can make.

What is a safety incident review?

A safety incident review is the structured response to a guardrail failure: establish what happened from records (not recollections), determine scope, reproduce the failure under test, fix it at the right layer, convert it into a permanent test, and record the outcome where the next reviewer will find it. It treats the evidence chain as the spine — every claim in the review should trace to a sealed entry, a count, or a scored run.

Create the free safety kit Next guide All guides

CompanionGuard is built by Ingenious AI. A compliance toolkit — not legal advice and not a clinical service. If you are in crisis right now: US — call or text 988; UK & Ireland — Samaritans 116 123.

Hour one: preserve, scope, and do not improvise fixes

First moves are conservation, not repair. Preserve the records: export the relevant evidence segment now (its chain integrity is strongest when captured before any temptation to tidy), note the counts and class data that bound the incident's scope, and capture the exact bot configuration in force — persona version, prompt templates, pack and locale settings, cap values. Scope from the record: which signal classes, which actions, how many sessions share the pattern — the chain answers these in its own vocabulary without exposing any user's words.

Resist the immediate prompt tweak. A fix applied before the failure is understood usually treats the screenshot that was shared rather than the mechanism that produced it, and it destroys your ability to reproduce the original behaviour. Freeze changes to the affected surface (deploy freezes are a normal incident tool), staff the review, and let the record — not adrenaline — set the agenda.

Reproduce in the harness, diagnose by layer

Move the failure into the test rig: construct the prompt-and-reply pair (from the report you received, redacted as needed) and score it. The per-check table tells you which named behaviour broke — a missing resource, a harmful-content hit, an empathy collapse, an over-referral in the other direction. Then diagnose by layer: was this a persona behaviour (the character chose wrong), a guardrail gap (no rule covered the case), a runtime event (a class matched or failed to match, an action honoured or not), or a configuration error (wrong pack, flag unset, caps misconfigured)? Each layer has a different fix and a different owner, and 'the bot messed up' is not a layer.

Check the siblings while you are in the rig: the same request type against your other bots (fleet rule — see the multi-bot guide), the same persona in your other channels, the adjacent scenario categories. Incidents are rarely singletons; they are samples from a distribution you have not mapped yet. Map it while the rig is warm.

Fix at the layer that failed, verify against the baseline

Apply the fix at the diagnosed layer: guardrail rule for a coverage gap, persona constraint for a character behaviour, configuration correction for a setup error — and write down why that layer owns it. Then verify in both directions: the incident pair now passes, and the frozen baseline pack has not regressed (fixes that cure one shape by breaking another are how incident reviews create next quarter's incident). The gate discipline from normal releases applies with double force to incident fixes, because incident fixes are written under pressure by tired people.

The incident pair joins the permanent transcript pack before the review closes — not as documentation, but as a test that will run on every future change. Over time that pack becomes the organisation's true safety memory, outliving every participant's recollection of the original event.

Close the loop with humans: disclosure, records, rhythm

Some incidents owe people an account: the user who reported it, a partner whose safeguarding process engaged, possibly a parent through your own channels. What you tell them should be drawn from the same record you reviewed — what the guardrails did, what you changed, what now tests it forever — in the precise, non-clinical vocabulary the product models. Never speculate about a user's state, never share another user's data in the accounting, and never let the account outrun the evidence: 'our records show the referral did not fire in this session; here is the class data and the fix' is a complete sentence that respects everyone in it.

Finally, feed the rhythm: the incident review itself gets a dated entry in your safety log, the rehearsal calendar gains the scenario if it exposed a process gap (an unrouted flag, a slow clock), and the next quarterly export review reads the incident's counts in context. An incident that changes only code has been half-reviewed; the full review changes code, tests and process, once, permanently. The scenario-failure guide covers the smaller, test-time version of this discipline.

Five actions for an incident review that holds

  1. Preserve first: export the evidence segment, capture counts and the exact configuration in force, and freeze changes to the affected surface before touching a prompt.
  2. Scope from the record — classes, actions, session patterns — never from anecdotes, and never by reaching for conversation content the record deliberately lacks.
  3. Reproduce the failure as a scored harness pair, diagnose by layer (persona, guardrail, runtime, configuration), and probe sibling bots and channels for the same shape.
  4. Fix at the owning layer, then verify both ways: the incident pair passes and the frozen baseline has not regressed.
  5. Add the pair to the permanent pack, write the dated review entry, and give affected people an account drawn strictly from the record.

Worked example: a missed referral, reviewed end to end

A fictional companion, Harbor, receives a user report: during a late-night conversation expressing hopelessness, the bot responded supportively but offered no resource. Hour one: the team exports the evening's chain segment and finds the session — the message's class is recorded as no-match (the user's phrasing sat outside the runtime's patterns), action allow, and the reply layer — which should have been the second net — produced warmth without referral. Scope check across the month finds three similar no-match sessions with the same reply shape.

Reproduction in the harness fails offers_crisis_resource on the reconstructed pair; diagnosis lands on the reply layer (the runtime's miss is a known, documented limit of pattern detection — see the detection guide — so the net was the layer that owed the catch). The fix strengthens the prompt-level care rule for hopelessness-adjacent language, the four real shapes join the transcript pack, the baseline re-runs clean, and the reporting user receives a precise account: what the records show, what changed, what now tests it. The review entry is two pages, dated, and cited in the next quarter's partner diligence without edits. The story is invented; the instruments are the product's own.

Frequently asked questions

What records anchor an incident review?

The hash-chained evidence (exported promptly, before any tidying), counts and class distributions for scope, the exact configuration in force, and scored harness runs for reproduction. Every claim in the review should trace to one of these.

Should we fix the prompt immediately?

No — preserve and reproduce first. A fix applied before the failure is reproduced usually treats the shared screenshot rather than the mechanism, and destroys the ability to verify the fix against the original behaviour.

How is the failure layer diagnosed?

Persona (the character chose wrong), guardrail (no rule covered the case), runtime (classification or action handling), or configuration (pack, flags, caps). The harness's per-check table plus the chain's class/action record usually make the layer plain.

What becomes permanent after an incident?

The failing exchange shape joins the transcript pack as a test that runs on every future change; the review itself is a dated entry; and any process gap it exposed (unrouted flags, slow clocks) enters the rehearsal calendar.

What do we tell the user who reported it?

An account drawn strictly from the record: what the guardrails did, what changed, what now tests it. No speculation about any user's state, no other user's data, and no claims beyond what the evidence shows.

Keep going