LLM Selection for Voice Agents: GPT-4o vs Claude Comparison

We benchmarked three LLMs against the same 500-call outbound qualification flow last quarter, all running through an identical STT and TTS stack on UK-region infrastructure. GPT-4o turned in a 640ms median LLM latency. Gemini 2.0 Flash came in at 300ms. Claude 3.5 Haiku sat at 420ms — but had the lowest structured output failure rate at 2.1%, versus GPT-4o at 4.3% and Gemini at 16.7%. The fastest model had the least reliable tool calling. That trade-off matters when your voice agent is deciding mid-call whether to transfer a prospect or keep qualifying them. The cheapest model cost roughly 7x less per 1,000 turns than the next tier — once you account for retry overhead on its failure rate.

What the LLM actually controls in a voice agent turn (and what it doesn't)

In a standard voice stack — STT → LLM → TTS — the LLM receives a text transcript of the caller's latest utterance, the rolling conversation history, and a system prompt. It produces either a text response or a structured function call. Everything else — audio capture, speech recognition, voice synthesis, call routing, CRM writes — sits outside the LLM layer entirely.

What this means in practice: the LLM's job in a voice agent turn is narrow but consequential. It decides whether to keep talking, book a slot, or route the call. That decision is expressed as a structured tool call. If the model misformats the call, the entire decision pipeline breaks. The caller hears silence or a generic fallback. The CRM gets no update. The call effectively fails at that turn.

Latency matters too, but differently from how most people assume. Your total turn latency includes STT processing (typically 200–400ms — see our STT provider comparison), TTS synthesis (see our TTS provider comparison), and audio buffering. The LLM latency is one component. A 340ms gap between fastest and slowest LLM adds roughly 200ms of perceived delay by the time audio plays — meaningful, but not the only variable. Our prompt engineering guide covers the full turn budget.

GPT-4o in a real-time voice stack: latency profile, function calling reliability, and streaming behaviour

GPT-4o's 640ms median in our test was measured from API request dispatch to receiving the first complete function call JSON. That is not time-to-first-token — GPT-4o starts streaming tokens almost immediately — but for voice agents using structured outputs, you cannot act on partial JSON, so completion latency is the relevant figure.

GPT-4o's structured outputs mode enforces a strict JSON schema at generation time, which prevents the model from producing invalid field names or types. Our 4.3% failure rate came not from schema violations but from the model occasionally omitting optional-but-expected fields in early-turn calls — failures that strict mode does not prevent if your schema marks those fields as optional.

Here is the function schema we used for the qualification routing decision:

{
  "type": "function",
  "function": {
    "name": "qualify_and_route",
    "description": "Evaluate the prospect and decide next action.",
    "strict": true,
    "parameters": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "qualified": { "type": "boolean" },
        "action": {
          "type": "string",
          "enum": ["transfer_to_human", "book_callback", "disqualify", "continue_qualification"]
        },
        "disqualification_reason": {
          "type": ["string", "null"],
          "enum": ["budget_too_low", "outside_uk", "wrong_sector", "no_decision_maker", null]
        },
        "confidence_score": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        }
      },
      "required": ["qualified", "action", "confidence_score", "disqualification_reason"]
    }
  }
}

The GPT-4o Realtime API handles audio-in, audio-out natively, cutting the STT and TTS hops. Stacks that rely on TTS caching cannot use it without losing the cache layer. We kept GPT-4o on the text path for this comparison.

Claude 3.5 Haiku: the case for Anthropic's fast tier in high-volume UK outbound

Claude 3.5 Haiku is Anthropic's inference-optimised tier. At 420ms median latency in our test, it sits between GPT-4o and Gemini Flash. The case for Haiku in outbound qualification is almost entirely about structured output reliability.

A 2.1% function call failure rate means roughly 21 bad turns per 1,000. At 400 calls a day with an average of eight turns per call — 3,200 turns — that is 67 malformed tool calls per day. With GPT-4o at 4.3%, that number is 137. With Gemini at 16.7%, it is 534. Those failures either require a retry (adding 300–640ms to a live conversation) or fall through to a generic error response that the caller hears as a confused agent.

Claude Haiku's tool-call failures were concentrated at higher turn counts: below 1.5% through turn eight, rising to 3.8% at turn nine and beyond. Compress context at turn eight and the failure profile becomes manageable — the trigger is a turn counter, not a random event.

UK data residency is available via AWS Bedrock in the eu-west-2 London region. We observed a 25–35ms latency increase over the standard Anthropic API endpoint.

Gemini 2.0 Flash: Google's latency claims vs what we measured from a UK datacentre

Google positions Gemini 2.0 Flash for low-latency, high-frequency tasks and cites sub-200ms median response times in their own benchmarks. From europe-west2 on Vertex AI, we measured 300ms median — not Google's claimed figure, but still the fastest of the three models we tested. The gap reflects the additional network hop to a UK-proximate region.

The problem is the 16.7% structured output failure rate. On the same qualify_and_route schema, Gemini 2.0 Flash consistently produced malformed enum values in the action field. It understood the routing intent correctly in nearly every case — a human reviewer would have made the same call — but it would output "transfer" instead of "transfer_to_human", or "disqualify_budget" instead of "disqualify", breaking the enum constraint. Google's function-calling mode does not enforce strict schema matching at generation time the way GPT-4o strict mode does.

For simple schemas — a date, a name, a yes/no decision — Gemini Flash's speed advantage is real and the failure rate drops significantly. The problems emerge with multi-field enum routing under real-time pressure.

Structured output fidelity: how each model handles tool calls under real-time pressure

Here is how the three models compared across our 500-call benchmark:

The P95 latency numbers matter more than the median for voice UX. A 560ms P95 (Gemini) is comfortable. An 1,120ms P95 (GPT-4o) means roughly one in twenty turns has a noticeable pause before the agent responds. On a ten-turn call, there is a material chance of at least one awkward gap.

Failure type also affects system design. GPT-4o's missing-field failures are predictable and catchable with a schema validator before the call exits your LLM adapter. Claude Haiku's context-drift failures give you a clear signal — turn count — to trigger mitigation before they occur. Gemini's enum failures require a string-normalisation pass on every function call output, which adds complexity and a small latency overhead to every turn.

Cost per 1,000 turns at UK outbound volume: the real bill at 400 calls a day

At 400 outbound calls a day with an average of eight turns per call, you are generating 3,200 LLM turns per day. Over a 22-day working month, that is 70,400 turns. Our benchmark calls averaged approximately 650 input tokens and 130 output tokens per turn.

Per 1,000 turns at those token counts:

Prices from provider docs; converted at £1 = $1.27 as of July 2026.

The raw gap between Claude Haiku and Gemini Flash is around 7x per turn. Factor in Gemini's 16.7% failure rate — retries double that turn's token cost — and the effective gap narrows to roughly 5–6x. At £6.70 versus £134 a month on LLM tokens, the GPT-4o line item is difficult to justify unless you specifically need its failure-mode characteristics.

Context window management mid-call: which model degrades gracefully at turn 15+

Most qualification calls end in 8–12 turns. At turn 15, your rolling context is roughly 9,750 tokens of transcript plus a 600-token system prompt — well within any of these models' limits.

The issue is attention consistency, not window size. We flagged calls where the agent's function call contradicted the system prompt — for example, routing to transfer_to_human before the prospect confirmed a budget figure. Instruction drift appeared at:

• Claude 3.5 Haiku: 1.2% of calls at turn 15, rising to 2.8% at turn 20
• GPT-4o: 2.1% at turn 15, rising to 4.4% at turn 20
• Gemini 2.0 Flash: 6.3% at turn 15, rising to 11.9% at turn 20

The mitigation is a context compression step at turn 8: summarise the transcript so far into a structured state object, drop the raw transcript history, re-inject the compressed state. Claude Haiku handles this without perceptible quality loss. Gemini Flash shows drift even with the compressed state at turn 15. See our post on voice agent call flow design for the compression pattern.

What changed in 2025–2026: reasoning model options and sub-500ms inference tiers

The headline development in 2025 was reasoning-optional models: OpenAI's o4-mini and Google's Gemini 2.5 Flash both allow toggling extended thinking on or off per request. For voice agents, you disable reasoning on every live turn — the latency would be unusable — but it opens up a useful pattern for off-call tasks: post-call scoring, CRM field extraction, and objection analysis, where you can afford 3–5 seconds and want better accuracy than a standard inference pass delivers.

Sub-500ms LLM inference as a baseline expectation arrived in early 2025. Anthropic launched Claude 3.5 Haiku in October 2024 specifically targeting the latency-sensitive tier; our 420ms UK-region median reflects that improvement over the previous Claude 3 Haiku, which ran at 580–620ms on the same infrastructure. GPT-4o also introduced inference latency tiers via a service_tier parameter in the API — "flex" versus "default" — with "flex" offering lower cost at the expense of queue priority. We tested this in 2025 and found "flex" added 150–280ms during UK peak hours (09:00–11:00 GMT), making it unsuitable for any live call.

Our recommendation by voice agent use case: qualification, booking, and inbound triage

Model selection should follow call type, not brand preference.

Outbound qualification (multi-turn, complex routing): Use Claude 3.5 Haiku. The 2.1% failure rate is the lowest of the three, context drift at high turn counts is predictable and manageable, and the 420ms latency fits inside a 1,000ms total turn budget. The additional cost over Gemini Flash is justified by reduced operational overhead on failure handling at volume.

High-volume slot booking (single or double turn, simple JSON): Use Gemini 2.0 Flash. Booking confirmation is a narrow task: extract a date, a time, and a name. The schema has three fields, no complex enums. Gemini's failure rate on simple schemas drops to 3–4% in our supplementary tests, and its 300ms latency leaves headroom elsewhere in the call budget. At £6.70/month versus £49 for Haiku at volume, the cost case is straightforward.

Inbound triage (variable complexity, high concurrency): Split by schema complexity. Simple triage — sales or support, qualify or escalate — runs on Gemini Flash. Complex triage involving multi-department routing, eligibility checks, or anything touching regulated data runs on Claude Haiku.

The good, the bad, and the ugly

GPT-4o - Good: Most predictable failure modes; streaming behaviour is clean on mixed text-and-tool responses - Bad: £134/month at 400 calls/day is hard to justify when Haiku matches it on reliability at a third of the cost - Ugly: P95 at 1,120ms — one turn in twenty has a noticeable pause

Claude 3.5 Haiku - Good: Lowest structured output failure rate; context degradation follows a predictable signal you can act on - Bad: No native UK data residency without Bedrock overhead; not cheap if Gemini Flash would suffice for your schema - Ugly: Nothing genuinely ugly — the most consistent model for production outbound

Gemini 2.0 Flash - Good: Fastest latency, cheapest token cost, excellent for simple JSON extraction - Bad: 16.7% failure rate on complex schemas requires retry infrastructure you would not otherwise build - Ugly: Instruction drift at turn 15+ causes live-call misrouting without active context compression

Our voice AI deployment work for a UK professional services client ran Claude 3.5 Haiku for outbound qualification and Gemini Flash for the post-call summarisation pipeline — exactly this split. The qualification flow ran eleven weeks without a production incident caused by a structured output failure.

Ready to select the right LLM stack for your call volume? Book a 30-minute architecture review and we will walk through the trade-offs for your specific call flow.