A healthcare migration case where lower cost came with faster note generation, faster decision support and more control over model behavior.
The migration
Sully.ai builds healthcare agents for clinical notes, reception, medical coding, intake, triage and decision support. The company originally relied on proprietary closed-source models. As traffic grew, it reported rising usage cost, inconsistent latency and limited control over model changes.
Sully moved production workloads to open models, including gpt-oss-120b, served through Baseten on NVIDIA infrastructure. In a Baseten customer case that is also discussed by NVIDIA, the company reported:
Reported Production Results
More than 90% reduction
Inference cost
Lower reported in the cited case.
About 70 seconds
Median note-generation latency
About 64% lower reported in the cited case.
About 20 seconds
Median decision-support latency
75% lower reported in the cited case.
65% reduction
Overall median latency
Lower reported in the cited case.
| Metric | Closed-model path | Open-model path | Reported change |
|---|---|---|---|
| Inference cost | Baseline | Lower | More than 90% reduction |
| Median note-generation latency | About 70 seconds | About 25 seconds | About 64% lower |
| Median decision-support latency | About 20 seconds | 5 seconds | 75% lower |
| Overall median latency | Baseline | Lower | 65% reduction |
Sully CEO Ahmed Omar summarized the operating constraint: “At our scale, inference efficiency matters as much as model quality.”
Why latency is a clinical workflow metric
Seventy seconds is not simply an infrastructure number. A clinician waiting for a note or decision-support response can change tasks, repeat an action or stop trusting the system. Unpredictable spikes make the tool harder to integrate into a live visit.
The economic denominator should therefore include clinician time:
cost per accepted clinical action = inference + integration + review + retry + delay cost / accepted actions
Moving median decision support to five seconds changes the usability of a real-time workflow. It can also reduce repeated requests and abandoned steps, although the public source does not isolate those effects.
Open models created an operating control surface
Sully's stated reasons went beyond price. Closed providers can change model behavior, versions and pricing. In healthcare, a silent quality regression can affect documentation consistency and compliance.
An open-model stack allows the company to pin a version, evaluate updates before release, tune the serving configuration and choose when to adopt a new model. That control is valuable only if the organization maintains strong validation, monitoring and incident response. Open weights shift responsibility toward the operator.
The case supports the argument for private and specialized healthcare AI, but it does not mean every provider should self-host. A small practice with low volume may be better served by a managed product. A scaled platform with repeated workloads has a different economic threshold.
Workforce results need careful attribution
Sully reports roughly 29 million minutes added to customer workforces by December 2025 and a 21 times Return on Agent Spend measure. It also reports 2.4 or more hours saved per physician, 18.5 percent more patients, 5 percent higher retention and a 98 percent reduction in abandonment.
These are broader product and customer outcomes. They cannot be attributed solely to the model migration. Workflow automation, integrations, customer adoption and operating changes contribute. The defensible migration evidence is the reported inference cost and latency change.
Evidence limits
Baseten and NVIDIA provide the infrastructure used in the new stack, so both have a commercial interest in the result. The sources do not publish raw clinical evaluation scores, traffic normalization, contract rates or fully loaded engineering cost. The metrics are reported customer outcomes, not an independent audit.
The case is still notable because it documents a direct move from closed production services to open models in a latency-sensitive, regulated workflow. That places it among the strongest direct migrations in the Frontier Model Downshift Index.
What a healthcare enterprise should test
Before migration, build a representative evaluation set across specialties, accents, note types, medications, negation, protected health information and high-risk clinical statements. Compare:
- critical medical fact error rate;
- hallucinated diagnosis or medication rate;
- note completeness;
- coding accuracy;
- p50 and p95 latency;
- cost per accepted note;
- clinician correction minutes;
- escalation and fallback rate;
- version-to-version regression.
Shadow the candidate against real traffic and retain a safe fallback. The related medical speech-recognition guide, private clinical knowledge-layer article and StudyFetch specialized speech case show how speech, RAG and model serving fit into the same workflow.
Building a Production-Ready System
Conscious Engines builds bespoke healthcare speech and language systems around the provider's specialties, vocabulary, note structures, privacy controls and latency requirements. We can combine medical STT, task-specific models, clinical RAG, validation and voice agents inside a governed deployment.
The aim is not open models for their own sake. It is a stable, measurable clinical workflow whose economics improve as usage grows.