AI Call Quality Monitoring: Catching Failures Before Clients Do

Answer rate on a mortgage broker's outbound campaign fell from 38% to 14% in March, and nobody noticed for three days. The error logs were clean, the agent configuration had not changed and call volume was normal. A carrier routing change had silently added 2.4 seconds of ringback before the agent's first word. Basic AI call quality monitoring would have flagged it within minutes.

Instead, three days of campaign budget went to calls where prospects hung up before a single syllable of the script landed.

The five voice agent failure modes that never appear in error logs

Error logs capture your application stack, so voice agent monitoring has to start from call outcomes. Logs miss the carrier's routing layer, the network between your stack and the caller's handset and a TTS provider's render queue under load. These are the five failure modes we have seen produce clean logs while campaigns silently degraded:

1. Carrier ringback delay. The SIP handshake succeeds, but the carrier inserts 2–4 seconds of dead ringback before bridging audio. Your logs show a completed call; the caller has already hung up.

2. Number flagging. Your number pool has accumulated enough rapid hang-ups to trigger a "Potential Spam" label from a carrier analytics provider such as TNS or First Orion. Answer rates drop 40–60% overnight. No error. The SIP layer is working fine.

3. AMD false positives at scale. Under high concurrency (50 or more simultaneous calls), the answering machine detection model classifies 12–18% of live human answers as voicemail. Completed calls in your logs; missed conversations in the CRM.

4. TTS render queue saturation. Your TTS provider hits a capacity constraint and P99 render latency climbs from 280ms to 1,800ms. The caller hears two seconds of silence, then abandons. No error is thrown anywhere in your application.

5. SIP signalling lag on geographic routing. Campaigns targeting mobile numbers in specific UK regions route through an unexpected carrier path. The call connects 3–5 seconds later than expected. Callers don't wait.

All five produce 200 OK responses at your application layer. The only way to catch them before they compound is to measure outcomes, not HTTP status codes.

Answer rate monitoring: the leading indicator most teams start tracking after the first disaster

Answer rate, meaning calls where a human actually spoke divided by total dials, captures all five failure modes above.

Segment it into three outcome categories for every call:

• Human answered: caller spoke and remained past 8 seconds of two-way audio
• Voicemail or AMD: call connected but terminated by machine detection logic
• Early drop: call connected at the SIP layer but audio exchange lasted under 5 seconds

Track per 5-minute window, per number pool, and per carrier route. A single campaign-level aggregate hides the pattern: a flagged pool alongside a clean pool blends to "low" rather than "alarming" until the flagged pool handles most of the volume.

Alert threshold: amber at >15% below the 7-day rolling average for that pool; red (consider pausing) at >30%. If you just ramped volume by 3×, recalibrate the baseline first.

Our voice agent QA scorecards guide covers scoring individual call quality alongside answer rate. You need both to distinguish a volume problem from a conversation quality problem.

Voice agent latency dashboard: instrumenting STT, LLM and TTS separately in production

Total turn latency is the time from the end of the caller's speech to the first byte of TTS audio. Target: under 800ms at P95. Above 1,200ms, hang-up rates climb. Above 1,800ms, callers assume the line has dropped.

If you log total latency only, you will know something is wrong but not which stage. Instrument these three segments separately, tagged by campaign and number pool:

• STT latency: end-of-utterance (VAD firing) to transcript delivery from Deepgram or Whisper
• LLM latency: transcript received to first token generated
• TTS latency: first text chunk received to first audio byte from ElevenLabs or Cartesia

Here is the structure we use in our Node.js broker layer, publishing to Datadog via dogstatsd:

// broker/monitoring.js
const { StatsD } = require('hot-shots');
const dd = new StatsD({ host: 'localhost', port: 8125, prefix: 'voice_agent.' });

function recordTurnLatency(callId, phase, startMs, endMs, tags = {}) {
  const durationMs = endMs - startMs;
  const metricTags = [
    `call_id:${callId}`,
    `phase:${phase}`,            // 'stt' | 'llm' | 'tts'
    `campaign:${tags.campaign}`,
    `number_pool:${tags.numberPool}`,
  ];
  dd.histogram('turn_latency_ms', durationMs, metricTags);
  if (durationMs > 1200) {
    dd.increment('turn_latency_breach', metricTags);
  }
}

With this in place, your voice agent latency dashboard shows each phase and a P95 spike is diagnosable in under two minutes: phase:tts spiking points to your TTS provider; phase:llm spiking points to your model or streaming config; phase:stt high means Deepgram is queueing or your VAD sensitivity is firing late. For more on reducing latency at the prompt level, see our guide on prompt engineering for voice agent latency.

Call drop detection and the SLA thresholds that trigger a response before the client notices

A drop happens after the audio bridge was established (call connected, audio was exchanged) but terminated before a meaningful conversation occurred. Our threshold: 8 seconds of two-way audio. Calls that end before that without a voicemail classification are drops.

Track three signals:

• Early drop rate: calls terminated before 8s of two-way audio, as a percentage of connected calls
• Agent non-response events: caller spoke but the agent produced no audio within 3 seconds
• Carrier-initiated disconnects: terminated by the network rather than by either party (SIP reason codes 503, 408, or 480)

SLA thresholds for a UK outbound campaign:

Financial services callers are less tolerant of dead air than e-commerce callers. If your vertical is mortgage broking or insurance, tighten the amber thresholds by roughly 20%.

Datadog, Grafana or n8n webhook flows for voice agent alerting at UK SME scale

No single tool fits every team. At UK SME scale the trade-off looks like this:

For a 5-person ops team: n8n for operational alerts, Grafana Cloud for latency dashboards. Twilio Event Streams pushes call events to an n8n webhook; Retell webhooks deliver per-call analytics. Two n8n flows (one on answer rate, one on early drop rate) posting to a Slack channel will catch most of the failures described above. That is enough AI call monitoring to start with.

Datadog earns its cost when you want distributed traces correlating a high-latency call through your broker, LLM, and TTS in a single view. For a single deployment, Grafana Cloud plus n8n is sufficient. For a comparison of the telephony platforms that feed these tools, see our Twilio vs Retell vs VAPI breakdown.

Telling carrier faults from agent configuration problems

The fastest diagnostic is the parallel test dial. Trigger 10 test calls from a fresh number pool to numbers you control while the affected campaign runs. If the test pool also shows degraded rates, the problem is carrier-side. If it performs normally, the issue is campaign-specific.

Carrier-side signals: degradation hits all campaigns on the same routing path simultaneously; SIP reason codes 480, 503, or 408 appear; Twilio Voice Insights call quality alerts shows carrier-reported events; degradation is geographically concentrated in specific UK mobile prefixes.

Number-health or config signals: degradation is campaign-specific; CNAM lookup returns "Scam Likely"; onset was within 48 hours of a configuration deployment; outbound dials degrade but inbound test calls on the same numbers connect normally.

If number flagging is the root cause, remediation differs by flag source, so our guide on phone number warmup and spam flag recovery covers the TNS, CNAM, and carrier-scoring paths separately.

Writing the runbook for each alert type before it fires

Written after an alert fires, a runbook is a post-mortem. Written before, it is the difference between a 20-minute resolution and a 3-hour investigation at 9pm.

For each alert type, define three things before go-live: the immediate response (what one person does within 5 minutes), the escalation trigger (when it is not resolving, who gets the message and what they need to act), and the campaign action (pause, reduce concurrency, or continue with elevated monitoring).

Here is an example for the answer rate amber alert:

alert: answer_rate_amber
threshold: ">15% below 7-day baseline for any number pool"
immediate_response:
  - Check Twilio carrier health dashboard for the affected route
  - Trigger 10 test dials from reserve pool to internal test numbers
  - Run CNAM lookup on affected number pool (First Orion portal or TNS lookup)
  - Check deployment log for configuration changes in the prior 48h
resolution_criteria: "Answer rate returns within 10% of baseline for 15 consecutive minutes"
escalation_trigger: "No improvement after 30 minutes, or rate continues to drop"
campaign_action: "Reduce concurrency by 50% while investigating"
client_notification: "Notify account owner if campaign paused for more than 2 hours"

Write one entry per alert type. Store runbooks in the same repository as your n8n flows, not in a separate wiki. The person covering an absent team member should reach resolution without prior context.

What changed in 2025–2026 for voice AI observability in Retell and VAPI

Twelve months ago, monitoring a Retell or VAPI deployment meant building your own metrics pipeline from scratch. In 2025 and into 2026, both platforms shipped native analytics.

Retell's call analytics dashboard, released in Q3 2025, exposes per-call latency breakdowns for STT, LLM, and TTS, filterable by agent, date range, and call outcome. VAPI added webhook-based monitoring events in early 2026 that fire on configurable latency thresholds, so you can route events directly into an n8n flow.

Neither platform's dashboard sends alerts to Slack or PagerDuty without custom plumbing. Provider dashboards are excellent investigation tooling; the detection layer is still yours to build.

One counterpoint worth heeding: the Google SRE book on monitoring distributed systems argues against alert proliferation: every alert without a clear response procedure becomes noise that trains operators to ignore the channel. Start with answer rate and P95 turn latency; add others only once you have runbooks for them.

Three monitoring setups and the failure each one missed

Good: Grafana plus n8n plus per-phase latency, built before go-live. A fintech client running eight voice agents had Grafana dashboards for STT, LLM, and TTS latency separately, n8n alert flows for answer rate and early drop rate, and a runbook for each alert type. When a TTS provider hit a capacity issue in January 2026, the latency alert fired at P95 = 1,340ms within six minutes. The on-call person confirmed the issue was TTS-specific, switched to the ElevenLabs fallback in the broker config, and restored normal latency within 18 minutes. No campaign was paused; no budget was wasted.

Bad: Twilio call logs only, no per-phase instrumentation. A property management company had call volume reporting but no answer rate baseline, no latency instrumentation, and no drop rate tracking. An AMD configuration change in Retell caused a 22% increase in false-positive voicemail classifications over 4 days. Call volume looked normal. One in five live human answers was being classified as voicemail and disconnected before the agent spoke. The issue surfaced when a prospect called back to ask why they had been hung up on, and retrospective analysis estimated 340 missed conversations.

Ugly: No monitoring, discovered by the client. The mortgage broker from the opening of this post. No baselines, no latency tracking, no carrier health checks. The 2.4-second ringback addition was visible in Twilio Voice Insights after the fact, but nobody was looking. The campaign spent approximately £4,200 over three days before the account manager noticed the CRM was not showing expected lead volume. By then, two numbers required a formal dispute process with the carrier analytics provider, and recovery needed a full number pool rotation plus two weeks of warmup before the campaign could resume.

Answer rate baselines, per-phase latency histograms, carrier health checks and pre-written runbooks would have caught all three of these failures before they compounded. We build this kind of instrumentation into our custom AI agents. For the wider architecture, see our voice AI architecture overview and the voice AI and document analysis case study, where monitoring was built into the deployment from day one.