The Enterprise Case for Small Language Models: Evidence from deployments, benchmarks and analyst reports

Most enterprise AI budgets are spent running frontier models on work that does not require them

Conscious EnginesSeptember 5, 2026

Enterprise AI has an allocation problem that usually presents itself as a cost problem. Teams stand up a pilot on a frontier model because it is the fastest way to prove that something works. The pilot works. It goes into production. And then the same model that was chosen for its ability to handle anything is left running a workload that consists almost entirely of classification, extraction, routing, summarisation and structured tool calls, none of which needed that capability in the first place.

NVIDIA Research put the problem plainly in a June 2025 position paper: the rise of agentic AI is producing "a mass of applications in which language models perform a small number of specialized tasks repetitively and with little variation."

This is not an argument that frontier models are overrated. It is an argument about fit. What has changed over the last two years is that the alternative is no longer speculative. There are now published benchmarks, named production deployments and analyst forecasts pointing in the same direction, and they are specific enough to build a procurement case on.

1. The economics have inverted

For the first several years of the generative AI cycle, the interesting cost question was training. That is no longer where the money goes.

AI-optimised IaaS spending, worldwide	2025	2026	2027
Total	$21.5B	$42.3B	$66.1B
Year-on-year growth		+96.4%	+56.5%
Of which inference	Below training	$23.3B (55%)	59% share
Of which training	Above inference	$19.0B (45%)	41% share

Source: Gartner, 10 August 2026. 2026 is the first year inference spending exceeds training spending.

Gartner's own explanation of the crossover names the shift directly:

"As organizations shift from model development to production-scale deployment, fine-tuned and domain-specific models (DSMs) are increasingly integrated into customer-facing and operational systems, requiring continuous, real-time execution rather than periodic training."

Hardeep Singh, Senior Principal Research Analyst, Gartner, August 2026

Once inference is the dominant line item, per-task efficiency stops being an engineering nicety and becomes a margin question.

What the efficiency gap actually is

Dimension	Advantage of a 7B model over a 70B to 175B model
Inference latency	10 to 30x
Energy consumption	10 to 30x
FLOPs per call	10 to 30x
Time to add or fix a behaviour via PEFT/LoRA	Overnight, versus weeks

Source: Belcak et al., NVIDIA Research, 2025.

A production example with published numbers

Databricks documented its own migration from a SaaS LLM API to a fine-tuned 7B model for automatic table documentation in Unity Catalog.

Measure	Result
Inference cost versus SaaS LLM API	10x reduction
Build cost	Under $1,000 in fine-tuning compute
Build effort	2 engineers, roughly 1 month
Fine-tuning run time	~15 minutes on ~3,600 examples
Serving hardware	A10 GPUs, more available than larger-model hardware
Quality versus SaaS baseline	Better than the cheaper tier, roughly equal to the expensive tier
Share of table metadata updates now automated	80%

Source: Databricks Engineering Blog, November 2023. Databricks sells fine-tuning infrastructure, so note the commercial interest; the engineering detail is unusually complete for a vendor account. Databricks later moved the same workload to a fine-tuned Mistral-7B.

The line worth quoting from that write-up is the negative finding, not the positive one:

"We did not yet observe a measurable benefit of using larger models for this task that would justify the increased serving costs."

Databricks Engineering

2. On narrow tasks, small models win outright

The economic argument would be uninteresting if it required accepting worse output. It does not, provided the task is genuinely scoped.

Small models matching or beating frontier models on scoped tasks

Model	Size	Task	Comparator	Result
LoRA Land fine-tunes (Mistral-7B, 25 models)	7B	25 distinct narrow tasks	GPT-4	Beat GPT-4 by 4 to 15%; beat base model by ~70%; under $8 per fine-tune; all served from one A100
LoRA Land technical report (310 models, 10 base models, 31 tasks)	Various	31 tasks	GPT-4	4-bit LoRA fine-tunes beat base models by 34 points and GPT-4 by 10 points on average
xLAM-7B (fc)	7B	Function calling, BFCL v1	GPT-4, Claude-3-Opus	Outperforms both
xLAM-1b-fc-r	1B	Function calling, BFCL	Claude-3-Opus (FC), GPT-3.5-Turbo	75.43% accuracy, surpassing both
xLAM-2-70b-fc-r	70B	τ-bench, multi-turn	GPT-4o (52.9%), DeepSeek v3 (40.6%)	56.2% success rate
DeepSeek-R1-Distill-Qwen-32B	32B	AIME 2024 / MATH-500 / LiveCodeBench	OpenAI o1-mini	72.6% / 94.3% / 57.2%, comparable to o1-mini; SOTA for dense models
DeepSeek-R1-Distill-Qwen-7B	7B	AIME 2024	QwQ-32B-Preview	55.5%, surpassing a 32B model
DistilBERT bi-encoder, DoorDash	~66M	Search-ad relevance	Incumbent GBDT production model	97% validation accuracy; +5.2% relative Precision@2 in online A/B
DistilBERT classifier, DoorDash	~66M	Conversation event classification	Fine-tuned Llama 3, vendor models	Highest F1 of all models tested; 0.159s inference

Note: xLAM and LoRA Land results are published by the organisations that built the models. DoorDash and DeepSeek results are self-published but reproducible against open checkpoints.

The DeepSeek finding is the one with the widest implications, because it reframes small models as compression targets rather than weaker peers. Their technical report found that distilling a strong model into a smaller one produced results that the small model could not reach through reinforcement learning at its own scale.

The pattern this produces is the one used by both DoorDash and DeepSeek: a large teacher model generates labels offline, where there is no latency budget to worry about, and a small student model trained on those labels is the only thing that ever touches the serving path. The expensive model shapes the cheap one and then gets out of the way.

DoorDash's engineering write-up makes the reliability argument explicitly:

"Because we wanted precise and consistent classification, a specialized model like our fine-tuned DistilBertForSequenceClassification was much more accurate and reliable."

DoorDash Engineering

For anything that sits inside a pipeline rather than in front of a human, a model that is slightly less capable but predictably parseable is the better production component.

3. Agents make the case structural

The most rigorous statement of the position comes from NVIDIA Research, which is not an organisation with an obvious commercial interest in customers buying less compute. Its June 2025 position paper argues that small models are sufficiently powerful for most agentic work, inherently more suitable for it, and necessarily more economical.

"Small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI."

Belcak, Heinrich, Diao, Fu, Dong, Muralidharan, Lin and Molchanov, NVIDIA Research and Georgia Tech, arXiv:2506.02153

Crucially, they do not argue for replacing frontier models. They argue for heterogeneous systems in which small models are the default and large models are invoked selectively.

The architecture is simple to describe. Every incoming request hits a router that classifies it by task and complexity, and the router has two destinations.

	Default path	Escalation path
Model	Small fine-tuned, 1B to 8B	Frontier
Hosting	Self-hosted	API
Handles	Classification, extraction, tool calls	Novel reasoning, long-context synthesis
Share of traffic	The majority	The exception

Invocations are logged on both paths, which is what turns everyday production traffic into the training data for the next round of specialisation.

The economics of that architecture have been measured.

Benchmark	Cost reduction versus all-GPT-4	Quality retained
MT Bench	Over 85%	95% of GPT-4
MMLU	45%	Matched
GSM8K	35%	Matched
Versus commercial routers (Martian, Unify AI)	Over 40% cheaper	Equivalent

Source: RouteLLM, LMSYS with UC Berkeley, Anyscale and Canva, July 2024; arXiv:2406.18665, ICLR 2025; code and trained routers on GitHub. On MT Bench the router kept 95% of GPT-4 quality while sending only 26% of queries to GPT-4.

One property of those results matters more than the headline number. A router trained on one model pair kept working when the underlying models changed, which is what makes routing a durable architectural choice rather than a configuration that rots every time a new model ships.

4. Constraints that money cannot solve

Cost is the argument that gets attention, but it is the weakest of the four, because API prices fall. The durable arguments are the ones where no price cut helps.

Enterprise posture on hosting and model strategy

Finding	Figure	Source
Enterprises comfortable hosting models directly, March 2024	~40%	a16z, 2025
Enterprises comfortable hosting models directly, 2025	~80%	a16z, 2025
Enterprises using five or more models	37%	a16z, 2025
Enterprise genAI spend, 2025	$37B, up 3.2x from$ 11.5B	Menlo Ventures, December 2025
Use cases bought rather than built, 2025	76%, up from 53%	Menlo Ventures, December 2025

a16z found that open-source adoption is highest at large enterprises precisely because of on-premise, compliance and fine-tuning requirements. For institutions handling personal, medical or financial data under the EU AI Act, GDPR, HIPAA or India's DPDP Act, sending data to a third-party API is often not a cost decision at all. The NVIDIA paper notes that for many financial institutions, using third-party SaaS models is impossible where PII and banking secrecy are involved.

Latency is the second constraint. DoorDash's ad relevance system separates a fine-tuned LLM teacher, used offline to generate labels at scale, from a fast student model used in serving, with query and item embeddings precomputed and cached. No frontier API delivers that inside an ad-serving path at any price.

The third is the edge. The NVIDIA authors define an SLM functionally as a model that runs on common consumer devices with response times fast enough for a single user, which puts the current boundary at roughly 10 billion parameters. That definition, rather than any specific hardware number, is the one to design against.

5. The three objections, and what each one actually implies

Every serious buyer raises the same three objections. All three are correct. None of them argues for staying on a single frontier API, and each one, followed to its conclusion, describes a specific way to build.

Objection 1: API prices are falling fast enough to make this moot

The evidence for the objection is real and it is the strongest of the three.

Cost per million tokens to reach MMLU 42	Model	Price
November 2021	GPT-3	$60.00
November 2024	Llama 3.2 3B	$0.06

"For an LLM of equivalent performance, the cost is decreasing by 10x every year."

Guido Appenzeller, a16z, November 2024

A thousandfold decline in three years, at a fixed level of capability.

What it implies. Read the claim precisely: prices fall for a fixed level of capability. That is exactly the observation that favours specialisation, because a task with a fixed capability requirement is a task whose cost should be collapsing toward zero, and it only does so if something in your architecture is willing to move down the tiers as they get cheaper. Most systems do not move. They were built on the best model available at launch and they stay there while the same quality becomes available for a fraction of the price one tier down.

So the objection is not an argument against this work. It is an argument against building anything static. A team without instrumentation cannot tell which of its calls have become overkill, and therefore captures none of the decline it is counting on. The organisations that benefit from falling prices are the ones with a routing layer and an evaluation harness already in place, because they can move a workload down a tier the week a cheaper model clears their quality bar. Everyone else pays 2021 architecture prices on 2026 infrastructure.

It also cuts the other way on cost of ownership. a16z notes that the decline has several independent drivers running at once, including quantisation, smaller models and open-source competition. Those are the same forces that make self-hosting a small model cheaper every year, not just the API.

Objection 2: fine-tuning ROI has been disappointing

Also true, and worth stating in its own words. The a16z 2025 CIO survey found enterprises seeing less return on fine-tuning than the year before, opting for open models mainly in cost-sensitive cases, and noted that companies with hyper-specific use cases continue to fine-tune.

What it implies. The disappointment is a scoping result, not a technology result. Fine-tuning underperforms when it is the first move rather than the third, applied to a task nobody measured, on data nobody curated, chosen because the team wanted a custom model rather than because a specific constraint demanded one. Base models kept improving and overtook those projects, exactly as they should have.

Fine-tuning projects that failed	Fine-tuning projects that worked
Started by picking a custom model	Started by running a baseline and building evaluations
Trained on uncurated data	Defined the task first with labelled data
No specific constraint demanded it	A latency, cost or format constraint demanded it
Overtaken when base models improved	Treated as replaceable from the outset

The projects that did work share a shape. Databricks fine-tuned after running a SaaS baseline and building a double-blind evaluation, on a task with enormous volume and a narrow output format, and their write-up says explicitly that they treat the result as replaceable. DoorDash distilled only after an LLM teacher had generated 700,000 labels defining exactly what the task was.

The correct response is therefore sequencing, not enthusiasm. Instrument, then route, then specialise only the tasks that survive that filter, and retire any fine-tune a newer base model matches. That ordering is what the migration path below encodes, and it is why nothing gets trained until stage three.

Objection 3: the Bitter Lesson says specialisation loses

Rich Sutton's argument that general methods leveraging computation eventually beat handcrafted specialisation is the deepest of the three objections.

What it implies. It is a claim about asymptotes, and it is a claim about handcrafted knowledge. Distillation and fine-tuning are not hand-coded domain rules; they are the transfer of a large model's learned behaviour into a cheaper artefact using data. DeepSeek's result makes the distinction concrete: the distilled small models beat what the same small models could reach on their own through reinforcement learning. Specialisation there is not fighting scale, it is inheriting it.

The practical version of Sutton's warning is genuinely useful, though, and it is this: never let a specialised component become load-bearing in a way that blocks you from adopting a better general model. That is an architecture requirement. It says keep the model layer swappable, keep evaluations continuous, and keep the routing policy separate from the models it routes to. Build that way and the Bitter Lesson costs you a config change. Build a monolith on one model and it costs you a rewrite.

The pattern across all three

Objection	What it rules out	What it argues for
Prices fall 10x a year	Static architecture on one model	Instrumentation and a routing layer that can move workloads down tiers
Fine-tuning ROI disappoints	Training as a first move	Sequencing: measure, route, then specialise only what survives
The Bitter Lesson	Specialised components you cannot replace	A swappable model layer with continuous evaluation

Each objection is an argument against the way most enterprises are currently deployed, and an argument for the architecture described here.

The failure case worth studying

Klarna's OpenAI-powered assistant handled two-thirds of customer service requests and did the work of roughly 700 full-time agents. Klarna credited AI with raising revenue per employee by 152% and cutting cost per transaction by 40% since Q1 2023. Then it reversed.

"We focused too much on efficiency and cost. The result was lower quality, and that's not sustainable."

Sebastian Siemiatkowski, CEO, Klarna, May 2025 (reported by eMarketer)

Siemiatkowski later conceded that cost had been "a too prominent evaluation factor". Note what actually failed. Klarna measured throughput and cost rigorously and quality loosely, then discovered the gap after scale-up was complete. The model was not the problem and its size was not the problem. The missing piece was an evaluation baseline agreed before launch, which is the same thing missing from most of the fine-tuning projects in objection two.

6. Which model tier for which workload

Workload	Recommended tier	Why
Classification, tagging, routing	Small, fine-tuned (1B to 8B)	High volume, narrow, format-critical
Structured extraction from documents	Small, fine-tuned or distilled	Format adherence and per-document economics dominate
Function and tool calling in agent loops	Small, format-trained	Repetitive and scoped; 1B models already beat GPT-3.5 and Claude-3-Opus here
Retrieval-augmented Q&A over internal corpora	Small to mid, routed	Retrieval carries the knowledge; the model does synthesis
Ranking and relevance in a serving path	Distilled encoder (sub-1B)	Latency budget makes API calls impossible
Regulated data, on-prem or air-gapped	Open-weight small, self-hosted	Not a cost decision
On-device and offline	Small, quantised	Memory and compute limits
Open-ended reasoning over novel problems	Frontier	Genuine capability requirement
Long-context synthesis across many documents	Frontier	Context handling and coherence
Low-volume, high-stakes, human-facing judgement	Frontier	Volume too low for the economics to matter

7. What to do about it

The migration path that the evidence supports is incremental and reversible, and it does not begin with training anything. It follows the conversion algorithm NVIDIA sets out in Section 6 of their paper.

1. Instrument. Log every model invocation, excluding confidential data, and classify by task type and complexity. Identify the high-volume, narrow, repetitive calls. Any task exceeding a few thousand frontier-model calls a day is a candidate. Nothing is trained and nothing changes in production, so there is no risk to weigh.

2. Route. Deploy a router with a small default model and frontier escalation. This captures much of the saving with no training at all, and it is reversible in a config change. If routed quality falls below roughly 95 percent of frontier on internal evaluations, tighten the thresholds rather than abandoning the approach. Documented saving: 35 to 85 percent depending on task type.

3. Specialise. Where an off-the-shelf small model underperforms on your highest-volume work, use logged task data to fine-tune or distil a 1B to 8B model. Justify it on a tenfold cost reduction, a strict format requirement, or a latency or privacy constraint the API cannot meet. Documented saving: 10x on those tasks, plus reliability gains.

4. Own it. For regulated data, edge deployment or offline operation, standardise on open-weight small models running in your own environment. Here the decision is made for you, and it is not a cost decision.

Reassess quarterly at every stage. If a newer base model plus prompting matches your fine-tune at comparable cost, retire the fine-tune.

What is worth avoiding is equally clear from the record: do not pre-train from scratch, do not replace people wholesale on quality-sensitive work, and do not commit your architecture to a single model provider. The organisations getting durable value are not the ones that picked the best model. They are the ones that built systems where the choice of model is a routing decision rather than a foundational commitment.

Databricks made exactly that point about their own fine-tuned model:

"We believe the LLM landscape is changing rapidly and the best model today won't be the best model tomorrow. That's why we consider this to be an iterative and continuous process."

Databricks Engineering

8. The forecast

"The variety of tasks in business workflows and the need for greater accuracy are driving the shift towards specialized models fine-tuned on specific functions or domain data. These smaller, task-specific models provide quicker responses and use less computational power, reducing operational and maintenance costs."

Sumit Agarwal, VP Analyst, Gartner, 9 April 2025

Gartner predicts that by 2027, organisations will implement small, task-specific AI models at a usage volume at least three times that of general-purpose LLMs. Their recommendations are worth reading alongside the prediction: pilot contextualised models where business context matters or where LLMs have missed quality or speed expectations, and adopt composite approaches using multiple models and workflow steps where single-model orchestration falls short.

That is a forecast rather than a finding, and forecasts are frequently wrong. But it is consistent with the deployment record, the benchmark literature, the routing research and the regulatory direction of travel. Four independent lines of evidence converging on the same architecture is about as strong a signal as this field produces.