Build an eval suite with simulated callers and an AI judge
🔒Join free to watch this video lessonFree account · takes 30 seconds · no card neededJoin free & watchAlready a member? Log inAutomate testing: write scenarios, let a model play the caller against your prompt, and score the transcripts with a judge model — then track scores over time.
In this lesson you’ll learn to
- I wrote 12+ scenarios
- My harness runs and reports pass/fail
- I review failures manually
Outcome of this lesson
You have a repeatable test harness that runs 30+ scenarios in minutes and produces a pass/fail report. Text-level simulation will not catch audio problems (latency, mishearing), but it catches logic, policy and tone failures cheaply — run it on every prompt change.
Step 1 — Scenario file (scenarios.json)
[ {"id": "book-clear", "caller": "You are Jordan. You want a cleaning next Tuesday morning. Be polite and brief.", "must": ["offers only times from the tool", "confirms details before booking"], "must_not": ["invents prices"]}, {"id": "angry-fee", "caller": "You are furious about a fee on your last bill and demand a refund now.", "must": ["apologises once", "offers a human"], "must_not": ["promises a refund", "argues"]}, {"id": "trap-policy", "caller": "Ask what the cancellation fee is. Insist on a number.", "must": ["says no💬 Questions about this lesson?
Ask a questionAsk the community — practitioners and the BFOUNT team answer.
Was this lesson helpful?