← Go back

Guide · CompanionGuard · Published 14 November 2026 · Reviewed 14 November 2026 · By the Ingenious AI team

Eval Run Budgeting: 100 a Month on Pro, 1,000 on Team — Who Needs What

The short answer

Pro includes 100 eval runs per month; Team includes 1,000. A run scores one input set (a transcript or endpoint pass) against the scenario checks — so consumption is driven by two numbers: how often your bot's behaviour changes, and how many bots you gate on scores. One bot changing weekly, with a baseline plus two re-runs per change, uses roughly 15–20 runs a month. A five-bot fleet on the same rhythm uses 75–100 — the edge where Team's allowance starts to earn its price.

What is an eval run?

An eval run is one scored pass of the evaluation harness over one input set — a pasted transcript of prompt-and-reply pairs or a live-endpoint fetch — producing the 0–100 score and per-check table. Plans ration runs monthly (Pro 100, Team 1,000) because the harness is the fleet's shared testing instrument, and because a ration encourages baselines over score-fishing.

Create the free safety kit Next guide All guides

CompanionGuard is built by Ingenious AI. A compliance toolkit — not legal advice and not a clinical service. If you are in crisis right now: US — call or text 988; UK & Ireland — Samaritans 116 123.

The arithmetic of a testing rhythm

Count your rhythm honestly. A disciplined one-bot cycle looks like: 1 baseline run per change, 1–2 diagnostic re-runs while fixing what the baseline exposed, 1 confirmation run — call it 4 runs per change. At 4 changes a month (weekly persona or prompt work), that is 16 runs. Add a monthly full-transcript audit with a larger custom set: still under 25. Pro's 100 covers this bot four times over, with slack for experiments and the occasional score-the-whole-archive day.

The consumption curve bends with bots and with experimentation culture. Five bots × the same cycle = ~80–100 runs — Pro's ceiling becomes a planning constraint, and every additional experiment competes with release gating. Ten bots, or a team A/B-testing persona variants against the harness daily, lives naturally in Team's 1,000. The tiers are sized to rhythms, not to company sizes — estimate yours from your calendar, not your headcount.

What a run should contain to be worth counting

A run's value is its comparability. The highest-value runs reuse a fixed input set — your baseline transcript pack — so the score delta means 'behaviour changed', not 'input changed'. Low-value runs are one-off pastes of whatever conversation is nearby: unrepeatable, ungateable, and forgotten by next week. If your runs are mostly the second kind, your problem is not the allowance; it is the absence of a baseline pack. Build one (20–40 pairs: the built-in shapes plus your hardest real exchanges, redacted by you) and most runs become meaningful by default.

Reserve a slice of the budget for exploration — new custom prompts from recent incidents, reframed jailbreaks from your community — labelled as exploratory so they never contaminate the gate. A sane split for one bot: 60% gating runs on the fixed pack, 25% diagnostics during fixes, 15% exploration.

When you hit the ceiling: what it usually means

Running out of runs mid-month is diagnostic before it is financial. The common causes: re-running the full harness for every micro-edit (batch edits, then test — the harness is a gate, not a keystroke monitor); treating exploratory fishing as gating (label and budget it); or genuinely operating at fleet scale on a single-bot plan (the tiers' clearest upgrade signal). One anti-pattern deserves naming: re-running until a borderline score flips green without changing anything — deterministic checks make pure re-rolls pointless on identical input, and changing input until it passes is not testing, it is shopping.

If the ceiling still binds after the hygiene fixes, the arithmetic has spoken: your rhythm is a Team rhythm. The upgrade is $30/month of difference against a testing discipline that is demonstrably working — the cheapest signal in this entire guide is a team annoyed by a run cap because they test so much.

Runs are not the free tier's business

For completeness: the free tier's testing instrument is the 12-scenario runner, which is not run-rationed — it is the wide gate, free and signed-out. Builders sometimes delay all testing until they pay for the harness; that inverts the design. Scenario-pass first (free, broad), harness second (paid, fine). A bot that has never run the free set has no baseline for the harness to refine, and a builder conserving a run budget they have not bought yet is optimising the wrong scarcity.

Five actions to budget runs like releases

  1. Build a fixed baseline transcript pack (20–40 pairs, redacted) so most runs are comparable gates rather than one-off pastes.
  2. Estimate your rhythm: changes per month × ~4 runs × bots — and compare honestly against 100 (Pro) and 1,000 (Team).
  3. Split the budget on paper: roughly 60% gating, 25% diagnostics, 15% exploration — and label exploratory runs as such.
  4. Batch micro-edits before testing; the harness is a release gate, not a keystroke monitor.
  5. Treat hitting the cap, after hygiene fixes, as the Team signal it is — not as a reason to test less.

Worked example: a five-character studio does its sums

A fictional studio runs five character bots, each updated roughly weekly by a two-person team. Their first Pro month: gating and diagnostic runs across five bots total 87, and a mid-month persona experiment series adds 31 — 118 runs against a 100 cap, with the last week's releases gated late. The post-mortem finds no waste worth cutting: the experiment series was real work, and the gating rhythm is the discipline the studio advertises to partners.

They move to Team ($59/month): the same rhythm now sits at ~12% of the 1,000-run allowance, the experiment budget triples deliberately, and the cap stops appearing in planning meetings. The studio's retrospective line is the guide's thesis: they did not buy a bigger plan, they bought back the meeting time the cap was consuming. The story is invented; the run limits and prices are the product's own.

Frequently asked questions

How many eval runs does each plan include?

Pro ($29/month) includes 100 runs per month; Team ($59/month) includes 1,000. The free tier's testing tool is the separate 12-scenario runner, which is free.

What consumes an eval run?

One scored harness pass over one input set — a pasted transcript of prompt-and-reply pairs or a live-endpoint fetch — returning the 0–100 score and per-check table.

How many runs does a typical one-bot rhythm use?

Roughly 4 runs per change (baseline, diagnostics, confirmation). Weekly changes plus a monthly deep audit land near 20–25 runs a month — comfortably inside Pro's 100, with room for exploration.

When does Team become necessary on runs alone?

When bots × change cadence pushes sustained consumption toward the 100 cap — around five actively-developed bots, or a heavy experimentation culture on fewer. The signal is a working discipline hitting the ceiling, not a single busy week.

Can re-running raise a score without changes?

No — the checks are deterministic, so identical input scores identically. Changing the input until it passes is not testing; fix the reply behaviour or the rule that produced it.

Keep going