Voice Agent A/B Testing: Script Experiments That Ship

We ran two versions of the same outbound qualification script in parallel last October — 200 calls each, randomised by phone number suffix, same campaign list, same calling window. Version A opened with the prospect's company name and a referral to a recent hire. Version B opened with a direct value statement and the qualification question in the first 12 seconds. At call 200, Version B had a 34% qualification rate versus Version A's 9%. We would not have caught that from intuition or recording reviews alone. We caught it because the test harness captured outcome labels automatically and flagged the gap at call 150.

A 25-percentage-point swing in qualification rate between two script variants — identified on a 200-call holdout pool before the main campaign launched — is what a structured test harness for voice agents is worth. This post is the architecture, the maths, and the three variables that shifted the needle.

What voice agent A/B testing actually controls for (and what it doesn't)

Voice agent A/B testing is narrower than web A/B testing. You are isolating prompt and script variables — what the agent says and when — not layout, colour, or copy. That narrowness is a feature, not a limitation, because it forces precision about what you are actually testing.

What a well-designed experiment controls for:

• The script opening line and the timing of the first qualification question
• Objection handling branches: what the agent says when a prospect responds with "not right now"
• Transfer trigger phrasing and the conditions that fire it
• Pause duration and confirmation phrasing before handoff

What it does not control for, and what you must neutralise through experimental design:

• Caller ID reputation — answer rates vary by number, not by script; use the same number pool across both variants
• Day-of-week and time-of-day effects — randomise within a calling window, not across separate days
• List quality differences — assign at the individual prospect record level, not by company segment or vertical
• TTS latency variance — jitter from your text-to-speech pipeline affects perceived conversational fluency and can interact with opening-line performance

The most common mistake we see: teams split their call list by company size or industry, then attribute the result difference to the script. That is not an A/B test — it is two different audiences. Randomise at the individual record level. Kohavi et al.'s "Trustworthy Online Controlled Experiments" covers the canonical pitfalls in controlled experimentation, most of which apply directly to call-level tests with minor adaptation.

Test harness architecture: holdout pools, call assignment, and data capture

The harness has four components: call assignment logic, metadata tagging, outcome capture, and an alerting threshold.

Call assignment. Assign variant A or B at dial time using the last digit of the destination phone number — even digits get Variant A, odd digits get Variant B. This is deterministic, requires no randomisation library, and distributes prospects roughly 50/50 across any realistic number pool.

Metadata tagging. Every call record gets a variant field at the Twilio call-creation step. That tag follows the call through the webhook chain, lands in the call log table in Postgres, and attaches to the transcript when the recording is processed. Here is the call-creation payload we use with Retell.ai:

{
  "from_number": "+441234567890",
  "to_number": "+447890123456",
  "override_agent_id": "agent_v2_qualification",
  "metadata": {
    "variant": "B",
    "campaign_id": "oct-q4-outbound",
    "experiment_id": "script-open-001",
    "assigned_at": "2025-10-14T09:12:44Z"
  },
  "retell_llm_dynamic_variables": {
    "script_opening": "direct_value",
    "qualification_q_timing": "12s"
  }
}

Outcome capture. Define exactly one primary outcome label before the campaign starts. For qualification campaigns, that is qualified = true on the call record, written by the post-call webhook when the transcript scores above threshold. Do not add secondary metrics mid-experiment — it inflates false positive risk.

Alert threshold. At 150 calls — 75% of the 200-call pool — the harness sends a summary email with conversion rate per variant and a chi-square p-value. If the gap exceeds 10 percentage points at p < 0.05, we flag it for early review. That is what surfaced the 25-point gap in October.

Our voice agent QA scorecards guide covers the outcome label taxonomy we use across campaigns in detail.

Sample size calculation for voice agent experiments: the 200-call minimum explained

The 200-call-per-variant threshold is not arbitrary. Here is the reasoning.

UK outbound B2B answer rates run at roughly 38% for warmed numbers. Ofcom's consumer experience research documents the continuing decline in answer rates across UK telecoms. At 38%, 200 dialled calls yields approximately 76 answered calls per variant.

To detect a 15-percentage-point difference in qualification rate — say, 20% versus 35% — with 80% statistical power at α = 0.05, a two-proportion z-test requires roughly 75 to 80 observations per group. That is precisely what 200 dialled calls at a 38% answer rate delivers.

If your baseline qualification rate is below 10%, you will need more calls to detect the same absolute delta. Run the calculation before fixing the pool size:

from statsmodels.stats.power import zt_ind_solve_power
import numpy as np

p1 = 0.08   # baseline qualification rate
p2 = 0.23   # expected variant rate (15pp lift)
h = 2 * np.arcsin(np.sqrt(p2)) - 2 * np.arcsin(np.sqrt(p1))  # Cohen's h

n = zt_ind_solve_power(
    effect_size=h,
    alpha=0.05,
    power=0.80,
    alternative='two-sided'
)
print(f"Answered calls per variant: {int(n) + 1}")
# → 94 answered calls → ~248 dialled calls at 38% answer rate

Below 200 dialled calls per variant, you are most likely measuring noise. Stop calling the result a test.

The three variables that move conversion: script opening, objection response, and transfer trigger

Across twelve experiments on UK outbound qualification scripts, three variables account for almost all of the meaningful variance in conversion rate. Everything else — pacing patterns, filler words, closing phrasing — moves the result by under 3 percentage points.

1. Script opening (first 8–12 seconds)

Whether the agent opens with contextual framing — referral, recent company news — or a direct value statement is the highest-variance variable we have tested. The direct value statement wins in the majority of experiments, but not universally. In regulated sectors such as financial services and healthcare, the referral approach sometimes outperforms because it reduces the defensiveness that causes immediate hang-ups.

2. Objection response at "not interested"

The standard "I understand, would you be open to a quick question?" construction consistently loses to a single short question: "What is the main thing standing in the way right now?" That question gets a substantive response more than 60% of the time. A substantive response keeps the call alive. A passive reassurance rarely does.

3. Transfer trigger timing

Offering the transfer too early — before confirming a problem exists — produces low transfer acceptance rates. Offering it too late — after the prospect has given three or four affirmative answers — reads as pressured. The window that converts is narrow: offer the transfer after the second confirmed pain point, not the first, not the third.

These are observed medians, not guarantees. Your baseline rate, sector, and list quality will shift the absolute numbers — the rank order has held consistent across our client base.

Statistical significance for call-level metrics: what significance means in a 38% answer-rate world

At a 38% answer rate, your effective sample is roughly a third of your dialled pool. That has two practical implications for how you interpret results.

First, variance estimates are noisier than they appear. A 34% versus 9% gap across 76 answered calls each produces a chi-square statistic of approximately 22.4, p < 0.001 — conclusive. A 25% versus 18% gap across the same 76 answered calls gives p ≈ 0.23 — not significant, despite looking substantial. Do not eyeball the numbers without running the test.

Second, non-answer is not neutral. If one variant produces a meaningfully different answer rate — which can happen when pre-call metadata changes or when calling windows differ between variants — the answered sample is already biased before you read the qualification data. Before interpreting qualification rate differences, always check whether answer rates differ between variants. We use a Fisher's exact test for this parity check because cell counts are frequently below 30 per quadrant.

We run the primary outcome test and the parity check using statsmodels. Do not hand-roll chi-square p-values in a spreadsheet — transcription errors in cell references are common and consequential.

Winning variant rollout: the deployment pattern that doesn't create a new experiment

When Variant B wins, the instinct is to push it to 100% immediately. Resist that. Switching to 100% on a known date creates a clean break in your time-series data — any subsequent performance change gets attributed to the script change with no same-period baseline for comparison.

The pattern that avoids this: migrate to the winning variant over 48 hours, keep the losing variant in the call queue at 10% traffic for one full additional calling day. That 10% holdout gives you a same-week baseline against which to validate production numbers before committing fully.

During rollout, change nothing else. No number pool additions, no time-of-day adjustments, no list refreshes. Each of those is a confound that makes subsequent analysis unreliable.

After full rollout, archive the losing variant's prompt configuration with its complete experiment metadata — campaign ID, experiment ID, sample counts, and the final p-value. You will need that record when someone asks in six months why the script looks the way it does. Our call flow design guide covers the prompt versioning pattern we use to keep winning variants auditable over time.

What changed in 2025–2026: LLM-assisted transcript scoring as an automatic grader

For the first few years of outbound voice agent experimentation at scale, outcome labelling was the bottleneck. A human had to read or listen to the transcript and decide whether the call was qualified. That created a lag between call completion and experiment data update, and introduced variance between individual labellers that undermined result reliability.

From mid-2025, we moved to LLM-assisted transcript scoring as the primary outcome grader. The model reads the full transcript, applies a structured rubric — did the prospect confirm a budget consideration? did they agree to a next step? — and writes qualified = true/false to the call record within 90 seconds of call completion. The same transcript analysis approach underpins the document routing layer in our Voice AI and Document Analysis case study.

Accuracy against a 500-call human-labelled gold set runs at 91% across client campaigns. That is sufficient for experiment outcome tracking. It is not sufficient for individual-call compliance decisions, where human review remains required.

OpenAI's evals framework on GitHub is a practical starting point for building rubric-based transcript graders — we adapted the match-based evaluator pattern for structured call outcome scoring.

One counterpoint worth reading before you commit to automated grading: research on LLM-as-judge approaches (Zheng et al., 2023) documents systematic biases including position preference and verbosity weighting. We observe a specific variant in call scoring: calls under 45 seconds are consistently under-scored for qualification even when the prospect gave a clear affirmative in the first exchange. Our mitigation is a manual review flag for any call under 60 seconds — it is a known limitation of the approach, not a solved problem.

Good / Bad / Ugly: three experiment designs and what each one missed

Good: single-variable, pre-registered, 200-call holdout

The October experiment described at the top of this post. One variable tested, pre-registered outcome label, 200 calls per variant, randomised by phone number suffix. The only gap: we did not timestamp the registration in a shared document before calls started. Adding that step now takes under two minutes and creates an internal audit trail that clients increasingly request.

Bad: four variants, three simultaneous variables

A client wanted to test opening line, objection response, and transfer timing simultaneously across four variants — a full factorial design on a 600-call pool. At 150 calls per variant, none of the pairwise comparisons reached statistical significance. The interaction effects between variables made interpretation impossible regardless. Test one variable at a time. Yes, it takes longer. No, there is no shortcut that avoids this.

Ugly: the variant that was not a variant

In one campaign, the B variant was accidentally routed to prospects from a different sub-list — technology companies instead of professional services firms. Variant B showed a higher qualification rate. The script had nothing to do with it. We caught the error because the industry distribution in the call metadata did not match the expected randomisation split. Always audit variant composition before declaring a winner. Our voice agent monitoring and alerting setup covers the composition drift checks we now run automatically at the 150-call mark.