Why real-time enterprise voice agents use specialized models, speculative decoding and caching instead of one frontier call for every stage.
The reported result
Decagon builds conversational agents for voice, chat and email. Its enterprise customers include Avis Budget Group, Chime, Oura Health and Hunter Douglas. The company says its agents handled tens of millions of interactions in the previous year with average deflection above 80 percent.
For voice, Decagon built a production stack using fine-tuned open models, specialized checker models, custom speculative draft models, larger verifier models, prompt caching and optimized NVIDIA B200 serving.
According to a Together AI customer case, the architecture delivered:
Reported Production Results
Nearly 6x lower
Cost per turn versus closed models such as GPT-5 mini
Reported in the cited source.
Under 400 ms
p95 model latency per turn
Reported in the cited source.
| Metric | Reported result |
|---|---|
| Cost per turn versus closed models such as GPT-5 mini | Nearly 6x lower |
| p95 model latency per turn | Under 400 ms |
| Supported input size | Up to tens of thousands of tokens |
| Model release frequency | Weekly, sometimes daily |
Voice makes every model delay visible
In text, a user may tolerate several seconds before an answer appears. In speech, silence feels like failure. Users talk over the system, repeat themselves or hang up. The latency budget must cover speech recognition, endpoint detection, reasoning, tool calls, guardrails and speech synthesis.
Decagon's case states the constraint in three words: “Voice latency is audible.”
A single slow model can consume most of the conversational budget. Using the largest closed model for intent, reasoning, checks and final output also multiplies cost across every turn.
The stack separates proposing from verifying
Speculative decoding uses a smaller draft model to propose tokens ahead of a larger model. The larger model verifies those proposals. When the draft is often correct, generation begins faster without accepting its output blindly.
Decagon trains custom speculators for its applications. It also sequences fine-tuned checker models for enterprise and regulatory rules. Prompt caching reduces repeated work across long conversations. The result is a model system with several narrow components rather than a single general model.
A simplified voice path can be:
speech recognition -> intent router -> workflow model -> tools -> checker -> response model -> speech synthesis
Each stage has a different quality and latency requirement. The intent router can be small. A company-policy checker can be fine-tuned. A complex exception can escalate to a larger reasoning model. The speech layers should be specialized for the customer's vocabulary, accents and channels.
This is why enterprise voice agents need a model portfolio.
Cost per turn is better than token price
Voice costs include more than the LLM:
cost per resolved voice interaction = STT + models + tools + checks + TTS + telephony + retries + review / resolved interactions
A model with cheap output tokens can still be expensive if it produces long answers, misses the user's intent or triggers repeated tool calls. A faster model can improve both infrastructure use and call containment.
Decagon reports nearly 6 times lower cost per turn, which is the right direction of metric. The source does not disclose the exact baseline configuration, rates, traffic mix or fully loaded cost.
Evidence limits
Together AI hosts and optimizes Decagon's production inference, so the source is a commercially interested customer story. The metrics are not independently audited. The claimed comparison is with closed models “like GPT-5 mini,” not a detailed controlled table for every model in the stack.
The case still demonstrates a credible architecture: Decagon explicitly avoids large closed models for every pipeline step and publishes model-latency and cost-per-turn outcomes. It should be read as evidence for multi-model voice design, not proof that every open model will produce a sixfold saving.
What an enterprise should evaluate
Measure a full call replay set with real noise, accents, interruptions, product names and policy cases. Track:
- word and critical-term error rate;
- time to first audio response;
- p50, p95 and p99 turn latency;
- interruption recovery;
- tool-call accuracy;
- severe policy violations;
- average turns to resolution;
- transfer and abandonment rates;
- cost per resolved call.
The Frontier Model Downshift Index, Sully.ai clinical-agent case, StudyFetch speech case, telecom voice AI guide and regulated banking voice architecture provide adjacent evidence.
Building a Production-Ready System
Conscious Engines builds enterprise voice stacks across speech-to-text, text-to-speech, agent reasoning, RAG, tool use, routing and validation. We tune each component for the domain and measure the whole conversational outcome.
The enterprise pays for a resolved, compliant interaction, not for one fashionable model sitting in the middle of the call.