Guide · CompanionGuard · Published 14 November 2026 · Reviewed 14 November 2026 · By the Ingenious AI team
Transcript Testing or Live-Endpoint Testing? Choosing Your Harness Input
The short answer
The eval harness scores replies from two sources. Pasted transcripts test replies your bot actually gave — real phrasing, real failure modes, fully repeatable, but limited to the exchanges you kept. Live-endpoint testing fetches fresh replies from the bot itself — current configuration, no selection bias, but variable across runs if the model is non-deterministic. Serious testing uses both: transcripts as the comparable baseline, the endpoint as the drift detector.
What is the difference between transcript and endpoint testing?
Transcript testing scores a fixed set of prompt-and-reply pairs you supply — the harness matches prompts to its scenarios (or classifies them) and applies the checks to the replies as written. Endpoint testing has the harness obtain the bot's replies itself, live, so the scored text reflects the bot's present prompts, model and settings rather than a saved sample.
Create the free safety kit Next guide All guides
CompanionGuard is built by Ingenious AI. A compliance toolkit — not legal advice and not a clinical service. If you are in crisis right now: US — call or text 988; UK & Ireland — Samaritans 116 123.
What transcripts prove — and how they can lie
A transcript run is a measurement of history: these replies, to these prompts, scored by these checks. Its strengths are comparability and honesty about reality — the replies are ones your bot genuinely produced, including the embarrassing ones, and the identical set can be re-scored after every fix to produce a meaningful delta. Its weakness is selection: a transcript pack assembled from exchanges someone chose to keep will flatter the bot, consciously or not. The discipline that fixes it: build the pack once, from a deliberate mix (built-in shapes plus your hardest real exchanges), then freeze it — additions allowed for new failure shapes, removals never without a written reason.
Redaction is the operator's job before pasting: strip identifiers your users never consented to share with a test tool. The harness needs the conversational text, not the account data around it — and your own handling should model the product's counts-and-hashes discretion even in your testing hygiene.
What the endpoint proves — and how it can wobble
An endpoint run is a measurement of the present: whatever the bot answers today, with today's prompts, model version and settings. That makes it the drift detector — the run that catches the silent upstream model change, the config edit nobody announced, the persona tweak that shipped without a test. Its weakness is variance: if your bot samples non-deterministically, two endpoint runs can score differently with no underlying change, and a delta between runs needs a repeated measurement before it means anything.
Use endpoint runs the way you use smoke tests: regularly, on the built-in scenario set, watching for movement — and when movement appears, reproduce it with a captured transcript, which converts a wobble into evidence you can debug. The two inputs are a system: the endpoint notices, the transcript proves.
The rhythm that uses both
A workable monthly rhythm for one bot: the frozen transcript pack is the release gate — scored on every change, delta must be zero-or-better against the recorded baseline. Endpoint runs on the built-in set happen weekly (and after any suspected upstream change), as a drift alarm. New real failures, from either source or from production, are captured into the transcript pack as new permanent pairs — the pack grows by scar tissue, which is exactly the right growth pattern.
Budget the runs accordingly (Pro's 100 a month covers this rhythm on one bot several times over; see the runs guide). What the rhythm buys is the property reviewers actually want: not a score, but a practice — evidenced, dated, repeatable.
Matching prompts: how the harness reads your input
Worth knowing when assembling inputs: the harness matches supplied prompts to its 8 built-in scenarios by prompt text; a prompt that matches inherits that scenario's crisis/everyday classification, and an unmatched prompt is classified as crisis when it contains crisis language (patterns like 'hurt myself', 'kill myself', 'suicidal', 'hopeless', 'better off without me', 'don't want to be here') and everyday otherwise. Custom pairs therefore slot into the same check discipline — crisis checks for crisis-classified pairs, everyday checks for the rest, harmful-content checks for all.
That classification rule should inform how you write custom pairs: a prompt you intend as an everyday edge case that happens to contain the word 'hopeless' will be scored as crisis. That is usually the safer error — but know it is happening, or your per-check table will confuse you. The custom-prompts guide goes deeper.
Five actions for a two-input testing practice
- Build and freeze a baseline transcript pack — built-in shapes plus your hardest real exchanges, redacted by you — and gate releases on its delta.
- Run the live endpoint on the built-in set weekly as a drift alarm, and after any suspected upstream model or config change.
- Convert every endpoint wobble into a captured transcript before debugging; never chase a score you cannot reproduce.
- Add every real production failure to the transcript pack permanently; let the pack grow by scar tissue only.
- Know the matching rules: prompts match scenarios by text, and unmatched prompts with crisis language are scored as crisis.
Worked example: the drift nobody announced
A fictional companion, North, gates releases on a frozen transcript pack (baseline 93, stable for two months) and runs a weekly endpoint check as routine. One Tuesday the endpoint run returns 81 — the transcript gate, re-run, still says 93. The gap itself is the diagnosis: the saved replies are fine because they are old; the live bot has changed. Investigation finds an upstream model update over the weekend that made replies shorter and clipped the resource out of two crisis answers.
Because the endpoint run caught it, the fix is a prompt adjustment the same day, verified by a fresh endpoint run at 92 and a captured transcript of the failing exchanges added to the pack. Without the two-input rhythm, North's next evidence of drift would have been a user's screenshot. The story is invented; the harness's two input modes are the product's own.
Frequently asked questions
What inputs does the eval harness accept?
Pasted prompt-and-reply pairs (transcripts) and live endpoint fetches. Both are scored against the 8 built-in scenarios' checks — matched by prompt text where possible, with unmatched prompts classified as crisis if they contain crisis language.
Which input should gate releases?
The frozen transcript pack: identical input across runs makes score deltas meaningful. Use endpoint runs as a regular drift alarm, and capture transcripts from any wobble before debugging it.
Why can endpoint scores vary between runs?
If the bot's model sampling is non-deterministic, fresh replies differ run to run. Treat a single endpoint delta as a signal to investigate — reproduce it with captured transcripts — not as a verdict.
Must I redact transcripts before pasting?
Yes — remove identifiers and account data yourself. The harness needs conversational text to score reply behaviour; your testing hygiene should mirror the product's own identifier-free discipline.
How do custom prompts get classified?
By text match to a built-in scenario first; otherwise as crisis when the prompt contains crisis language ('hurt myself', 'suicidal', 'hopeless', 'better off without me' and similar), else as everyday. Crisis-classified pairs get the crisis checks; all pairs get the harmful-content check.
Keep going
Related guides: Companion Bot Evaluation Harness: How Behavioural Scoring Works · Custom Prompts in the Eval Harness: Testing the Cases the Built-Ins Miss · Eval Run Budgeting: 100 a Month on Pro, 1,000 on Team — Who Needs What
Put it to work: Create the free kit · Eval harness · Scenario tests · Crisis runtime · Disclosure packs · Pricing · All CompanionGuard guides