Portfolio · Case study · Model selection & fine-tuning In progress

Does every task need the biggest model? We put it to the test.

We fine-tuned a small language model with LoRA on one narrow, high-volume task, making sense of cryptic bank transaction descriptions, and compared it head to head with a large hosted model on accuracy, latency and cost per 1,000 requests. The result: a clear, evidence-based rule for when each one is the right choice.

Client
Financial services group
Service
LoRA fine-tuning & model benchmarking
Training data
116,000+ bank transactions
Status
In progress

The challenge

A large model on every request was slow, costly and sent data outside.

Bank statements describe transactions in short, cryptic codes such as “FDRL/NATIONAL ELECTRONIC F”, “CASHDEP/GURGAON/” and “Sweep Trf To”. Turning these into clear categories powers spending summaries and personalised insights, but it happens on every single transaction.

Calling a large hosted model for each one meant per-request costs that grew with volume, noticeable latency, and sensitive financial data leaving the client’s environment. The question: could a small, specialised model do this job just as well?

Our approach

A fair, measured head-to-head.

  • One narrow task: classify each transaction’s type and category into structured output.
  • LoRA fine-tuning: small adapter weights trained on labelled examples, so the base model stays unchanged and training is fast and cheap.
  • Same test set, same rules: both models scored on an identical held-out set the small model never saw in training.
  • Three measures that matter: accuracy, latency and true cost per 1,000 requests, including hosting.

How we did it

From raw statements to a benchmarked model

  1. 01 · Prepare

    Clean & label

    Transactions are cleaned, de-duplicated and labelled, with account numbers masked.

  2. 02 · Split

    Train / test

    A held-out test set is locked away before any training begins.

  3. 03 · Fine-tune

    LoRA adapters

    The small model learns the task through lightweight adapter layers.

  4. 04 · Benchmark

    Head to head

    Both models run the same test set under real load conditions.

  5. 05 · Decide

    Route smartly

    Each request goes to the model that wins for that kind of work.

Input: raw statement line
FDRL/NATIONAL ELECTRONIC F
Withdrawal
Output: model response
{
  "type": "NEFT transfer",
  "direction": "outgoing",
  "category": "Bank transfer",
  "channel": "Online banking"
}

Head to head

Small and specialised vs large and general

Results on the held-out test set. Costs include hosting for the small model and per-token pricing for the large one.

MeasureFine-tuned SLMHugging Face open model + QLoRALarge hosted modelOpenAI
Accuracy91%92%
Latency (median)102 ms7,241 ms
Latency (p95)215 ms8,549 ms
Cost per 1,000 requests (approx.)$0$20
Valid structured output95%94%
Where data is processedInside the client’s own cloudThird-party API

The verdict

There’s no single winner, there’s a right tool for each job

Choose the fine-tuned SLM when…

  • The task is narrow, repetitive and well defined
  • Volume is high, so per-request API costs add up fast
  • Responses must be fast and consistent
  • Data is sensitive and should stay inside your environment
  • You have enough good labelled examples to train on

Choose the large model when…

  • Questions are open-ended or need multi-step reasoning
  • The task changes often, so retraining would be constant
  • Volume is low, so hosting your own model doesn’t pay off
  • Labelled training data is scarce or expensive to create
  • Rare, unusual inputs need broad general knowledge

What we shipped: a hybrid router. The fine-tuned SLM handles the high-volume classification work, and any low-confidence or out-of-scope request is passed to the large model, getting the cost and privacy benefits without giving up quality on the hard cases.

Tech stack

Tools we used

Small model

Hugging Face open-weight model

Fine-tuning

QLoRA adapters, Hugging Face PEFT

Large model

OpenAI hosted model

Benchmarking

Held-out test set, load testing, cost tracking per request

Serving

Self-hosted inference in the client’s own cloud

Privacy

Account masking, in-environment inference, no data retention

Outcomes

What the client gained

≈ $20

saved per 1,000 requests

71×

faster median response time

1 pt

accuracy difference vs the large model

97%

of requests handled privately by the SLM

“We assumed we needed the biggest model for everything. NexaQore’s benchmark gave us hard numbers instead of guesses, and an honest view of where the small model wins and where the large one still does. Our fine-tuned model now runs entirely in-house, keeps customer financial data inside our environment and handles the bulk of our transaction volume faster and at a fraction of the cost. We only call the large model when it’s genuinely needed.”

CTO, financial services group

Paying large-model prices for a small-model job?

We benchmark your real workloads and help you choose, or fine-tune, the model that gives the best balance of accuracy, speed, cost and privacy.

Book a free consultation