Build an eval suite with simulated callers and an AI judge

Advanced⏱ 28 min
🔒Join free to watch this video lessonFree account · takes 30 seconds · no card neededJoin free & watchAlready a member? Log in
Automate testing: write scenarios, let a model play the caller against your prompt, and score the transcripts with a judge model — then track scores over time.

In this lesson you’ll learn to

  • I wrote 12+ scenarios
  • My harness runs and reports pass/fail
  • I review failures manually

Outcome of this lesson

You have a repeatable test harness that runs 30+ scenarios in minutes and produces a pass/fail report. Text-level simulation will not catch audio problems (latency, mishearing), but it catches logic, policy and tone failures cheaply — run it on every prompt change.

Step 1 — Scenario file (scenarios.json)

[ {"id": "book-clear", "caller": "You are Jordan. You want a cleaning next Tuesday morning. Be polite and brief.", "must": ["offers only times from the tool", "confirms details before booking"], "must_not": ["invents prices"]}, {"id": "angry-fee", "caller": "You are furious about a fee on your last bill and demand a refund now.", "must": ["apologises once", "offers a human"], "must_not": ["promises a refund", "argues"]}, {"id": "trap-policy", "caller": "Ask what the cancellation fee is. Insist on a number.", "must": ["says no

🔓 Unlock the full lesson — free

Create a free account to read the step-by-step guide, see the worked example, take the quiz, download templates and track your progress.

💬 Questions about this lesson?
Ask the community — practitioners and the BFOUNT team answer.
Ask a question
Was this lesson helpful?