Portfolio · Case study · Model selection & fine-tuning In progress
Does every task need the biggest model? We put it to the test.
We fine-tuned a small language model with LoRA on one narrow, high-volume task, making sense of cryptic bank transaction descriptions, and compared it head to head with a large hosted model on accuracy, latency and cost per 1,000 requests. The result: a clear, evidence-based rule for when each one is the right choice.
- Client
- Financial services group
- Service
- LoRA fine-tuning & model benchmarking
- Training data
- 116,000+ bank transactions
- Status
- In progress
The challenge
A large model on every request was slow, costly and sent data outside.
Bank statements describe transactions in short, cryptic codes such as “FDRL/NATIONAL ELECTRONIC F”, “CASHDEP/GURGAON/” and “Sweep Trf To”. Turning these into clear categories powers spending summaries and personalised insights, but it happens on every single transaction.
Calling a large hosted model for each one meant per-request costs that grew with volume, noticeable latency, and sensitive financial data leaving the client’s environment. The question: could a small, specialised model do this job just as well?
Our approach
A fair, measured head-to-head.
- One narrow task: classify each transaction’s type and category into structured output.
- LoRA fine-tuning: small adapter weights trained on labelled examples, so the base model stays unchanged and training is fast and cheap.
- Same test set, same rules: both models scored on an identical held-out set the small model never saw in training.
- Three measures that matter: accuracy, latency and true cost per 1,000 requests, including hosting.
How we did it
From raw statements to a benchmarked model
- 01 · Prepare
Clean & label
Transactions are cleaned, de-duplicated and labelled, with account numbers masked.
- 02 · Split
Train / test
A held-out test set is locked away before any training begins.
- 03 · Fine-tune
LoRA adapters
The small model learns the task through lightweight adapter layers.
- 04 · Benchmark
Head to head
Both models run the same test set under real load conditions.
- 05 · Decide
Route smartly
Each request goes to the model that wins for that kind of work.
FDRL/NATIONAL ELECTRONIC F
Withdrawal{
"type": "NEFT transfer",
"direction": "outgoing",
"category": "Bank transfer",
"channel": "Online banking"
}Head to head
Small and specialised vs large and general
Results on the held-out test set. Costs include hosting for the small model and per-token pricing for the large one.
| Measure | Fine-tuned SLMHugging Face open model + QLoRA | Large hosted modelOpenAI |
|---|---|---|
| Accuracy | 91% | 92% |
| Latency (median) | 102 ms | 7,241 ms |
| Latency (p95) | 215 ms | 8,549 ms |
| Cost per 1,000 requests (approx.) | $0 | $20 |
| Valid structured output | 95% | 94% |
| Where data is processed | Inside the client’s own cloud | Third-party API |
The verdict
There’s no single winner, there’s a right tool for each job
Choose the fine-tuned SLM when…
- The task is narrow, repetitive and well defined
- Volume is high, so per-request API costs add up fast
- Responses must be fast and consistent
- Data is sensitive and should stay inside your environment
- You have enough good labelled examples to train on
Choose the large model when…
- Questions are open-ended or need multi-step reasoning
- The task changes often, so retraining would be constant
- Volume is low, so hosting your own model doesn’t pay off
- Labelled training data is scarce or expensive to create
- Rare, unusual inputs need broad general knowledge
What we shipped: a hybrid router. The fine-tuned SLM handles the high-volume classification work, and any low-confidence or out-of-scope request is passed to the large model, getting the cost and privacy benefits without giving up quality on the hard cases.
Tech stack
Tools we used
Small model
Hugging Face open-weight model
Fine-tuning
QLoRA adapters, Hugging Face PEFT
Large model
OpenAI hosted model
Benchmarking
Held-out test set, load testing, cost tracking per request
Serving
Self-hosted inference in the client’s own cloud
Privacy
Account masking, in-environment inference, no data retention
Outcomes
What the client gained
saved per 1,000 requests
faster median response time
accuracy difference vs the large model
of requests handled privately by the SLM
“We assumed we needed the biggest model for everything. NexaQore’s benchmark gave us hard numbers instead of guesses, and an honest view of where the small model wins and where the large one still does. Our fine-tuned model now runs entirely in-house, keeps customer financial data inside our environment and handles the bulk of our transaction volume faster and at a fraction of the cost. We only call the large model when it’s genuinely needed.”
Paying large-model prices for a small-model job?
We benchmark your real workloads and help you choose, or fine-tune, the model that gives the best balance of accuracy, speed, cost and privacy.