How labeled enterprise data, fine-tuning and a constrained workflow produced a faster and lower-cost production classifier.
The decision
Checkr processes about 1.5 million background checks each month for more than 100,000 businesses. Most records could already be handled through traditional automation. The difficult problem was the roughly 2 percent of unstructured, messy records that contained subtle civil or criminal charge descriptions.
Checkr evaluated GPT-4, GPT-4 with retrieval-augmented generation and a fine-tuned Llama 3 8B model. In a detailed Computerworld interview, the company reported:
Reported Production Results
88%
GPT-4
About $12,000 reported in the cited case.
96%
GPT-4 with RAG
About $7,000 reported in the cited case.
97%
Fine-tuned Llama 3 8B
About $800 reported in the cited case.
| Candidate | Overall accuracy | Difficult 2% accuracy | Reported latency | Estimated monthly cost |
|---|---|---|---|---|
| GPT-4 | 88% | 82% | 15 seconds | About $12,000 |
| GPT-4 with RAG | 96% | 79% | 7 seconds | About $7,000 |
| Fine-tuned Llama 3 8B | 97% | 85% | About 0.5 seconds in article summary | About $800 |
The interview later describes SLM latency as under 0.15 seconds. Because the source contains both figures, the conservative statement is that latency moved from seconds to sub-second response.
Why retrieval did not solve the hardest cases
RAG is valuable when a model needs current or private facts. Checkr's difficult cases were not mainly missing facts. They represented minority classes and subtle decision boundaries. Retrieving ten similar known examples did not reliably teach the model how to treat an unusual record.
Fine-tuning changed the model's behavior using thousands of labeled examples collected over years. The prompt then needed only the record text rather than a large bundle of retrieved context. That reduced input tokens, latency and cost.
This distinction is central to choosing between RAG, fine-tuning and a bespoke model:
- use RAG when knowledge changes or citations matter;
- use fine-tuning when behavior, classification boundaries or output format must become consistent;
- use both when the task needs current knowledge and specialized behavior.
The system was more than an 8B model
Checkr's production design reportedly used one A100 GPU and an adapter architecture. Multiple low-rank adapters can share one base model, so each task does not require a separate full model copy and GPU.
The output space was also constrained to about 230 possible classifications. When the model produced an invalid label, programmatic logic could reject it and retry or invoke another fallback. Checkr's engineer put the architecture plainly: “It’s never just the model.”
That surrounding system is why narrow models work in enterprise settings. Schema validation, allowed-label constraints, confidence thresholds, fallbacks and audit logs reduce the blast radius of a probabilistic error.
The most important number is not 8 billion
The case is often summarized as small beats large. A more precise explanation is that private labels changed the ranking.
GPT-4 carried broad capabilities that the classifier did not need. Checkr's 8B model absorbed years of task-specific examples. It only had to map a bounded text input to a bounded classification set. The quality target was measurable, the volume was high and the output could be validated automatically.
This is the profile in which a bespoke enterprise model business case becomes compelling:
- stable and repeated task;
- high request volume;
- proprietary labels or corrections;
- bounded output;
- measurable error cost;
- meaningful latency or privacy constraint.
Evidence caveats
The public record does not establish that Checkr ran GPT-4 as a long-standing production service. It appears that GPT-4 and RAG were evaluated as candidate architectures before the smaller model was selected. The cost numbers are estimates shared in an interview, not audited invoices.
The accuracy figures also need their denominator. An 85 percent result on the difficult 2 percent does not mean the whole workflow accepts 15 percent bad outcomes. Checkr described additional models, logic and fallbacks around the classifier. Enterprise buyers should measure end-to-end accepted outcomes, not one model's raw score.
The case belongs in the Frontier Model Downshift Index because it shows a production selection against a frontier baseline. It should not be described as proof that every GPT-4 workload can move to an 8B model.
Related evidence includes Meta's deterministic-first classifier, Boosted.ai's finance-tuned model and the broader report on why the right model is rarely the biggest.
Building a Production-Ready System
Conscious Engines builds task-specific models for enterprises that have this same shape of problem: large operational volume, valuable proprietary examples and a narrow decision that generic AI handles inconsistently or expensively.
We turn historical cases, corrections and policy into an evaluation set, fine-tune or distill the appropriate model, add deterministic validation and fallbacks, and deploy it in the environment the enterprise controls. The model is bespoke because the decision boundary belongs to the business.