A rare production case with both the savings and the regressions from moving a high-volume software QA agent to open weights.
The result in one minute
Ito runs software quality-assurance agents against pull requests. Its highest-volume test-runner agent executes planned tests inside development containers using a real browser. From March through June 2026, that agent ran on GPT-5.3-codex. In late June, Ito migrated it to MiniMax M3 and began processing roughly 100 billion tokens per week.
According to Ito's engineering report, the company analyzed more than half a trillion tokens of production traffic and reported:
Reported Production Results
55% lower
Median model cost per pipeline run
Reported in the cited source.
60% lower
Median model cost per step
Reported in the cited source.
36% lower
p90 model cost per step
Reported in the cited source.
17% lower
p99 model cost per step
Reported in the cited source.
About 28% lower
Weekly model spend
Reported in the cited source.
| Production metric | Reported change |
|---|---|
| Median model cost per pipeline run | 55% lower |
| Median model cost per step | 60% lower |
| p90 model cost per step | 36% lower |
| p99 model cost per step | 17% lower |
| Weekly model spend | About 28% lower |
| Workload volume | About 8% higher |
| Median step duration | 23 to 29 minutes |
| Terminal step failure rate | 2.4% to 4.5% |
| Migration time | 11 days |
This is one of the strongest cases in the Frontier Model Downshift Index because Ito disclosed the unfavorable results too.
The cheaper model did more work
On 156 matched, successful test steps drawn from the same repository set, the open model was less token-efficient:
| Median per test batch | GPT-5.3-codex | MiniMax M3 | Change |
|---|---|---|---|
| Agent turns | 94 | 116 | 1.23x |
| Output and reasoning tokens | 21,500 | 46,800 | 2.18x |
| Input tokens including cache reads | 5.4M | 13.5M | 2.51x |
| Indexed cost | 1.00 | 0.46 | 54% lower |
The replacement won because its effective token rates were 3 to 12 times lower, not because it generated fewer tokens. Ito summarized the decision cleanly: “Our invoices chose MiniMax.”
This matters for enterprise procurement. A benchmark that ranks models by tokens to completion would have favored the frontier model. A financial evaluation based on cost per successful test batch favored the open model.
Provider choice was part of model quality
Ito tested open models through Vercel AI Gateway, OpenRouter and Together AI. It found different cache behavior and evaluation results even when the named model and underlying provider appeared to be the same. The company ultimately integrated with Together AI directly.
About 97 percent of Ito's input tokens were cache reads. For a long-running agent, a small change in cache-hit rate can dominate the bill. Sampling settings, quantization, context handling and gateway behavior also change the product that the enterprise receives.
The lesson is operational: evaluate the complete route, not the model name. The test unit should include model, provider, serving configuration, cache policy, gateway and billing behavior.
Why the regression was acceptable
Median step duration rose by six minutes, and terminal failures almost doubled. That sounds disqualifying until the business context is considered. The QA tasks ran in the background, so the extra time did not block a developer's immediate work. Retries absorbed many failures before they became user-visible.
The same result would be much less attractive for a live voice agent, interactive coding assistant or clinical decision-support tool. Model selection is therefore a constrained optimization problem:
choose the lowest-cost path where quality >= floor, latency <= budget and severe-error rate <= threshold
Ito also found that savings ranged from 13 to 81 percent by repository. An enterprise-wide average would have hidden which workloads should migrate and which should remain on the stronger model.
A migration pattern enterprises can copy
Ito's rollout contains six useful controls:
- It shortlisted providers through compliance requirements first.
- It metered every agent step before changing the model.
- It compared actual invoice rates, including hidden reasoning-token effects.
- It staged the cutover repository by repository over 11 days.
- It kept a same-day revert path.
- It evaluated production outcomes by workload rather than relying on a global average.
This is the practical version of enterprise AI model routing. A router can keep the open model on background workloads with tolerant latency, then send interactive or unusually hard work to a frontier model.
What the case does not prove
Ito did not show that MiniMax M3 is universally better than GPT-5.3-codex. The open model was slower, used more tokens and failed more often. The rollout was staged, not randomized. Repository mix and traffic changed over time.
It proves something more useful: a less capable model can be the better production choice when the enterprise measures the full economic function and the workload can tolerate its tradeoffs.
Teams considering the same move should connect this evidence with the cost-per-outcome FinOps framework, production evaluation design, the AI cost-reduction playbook and Replit's routing case.
Building a Production-Ready System
Conscious Engines helps enterprises run this migration as a controlled engineering program. We instrument the incumbent path, build a workload-specific evaluation set, test smaller and open alternatives, optimize serving and cache behavior, and deploy routing with fallbacks.
The objective is not to remove a frontier provider at any cost. It is to move each production task to the least expensive approved system that meets its real quality and latency requirement.