Build Log: Fine-Tuning a CFO Brain That Won't Make Up Your Numbers
We fine-tuned an open-weight model into a CFO advisory brain — and failed our own audit on the first try. Here’s the full build log: what broke, what fixed it, and what it cost.
We turned Qwen3-32B into a CFO advisory brain for Indian mid-market companies using only synthetic data — 1,926 training pairs that survived a five-gate quality gauntlet (only 17% of generated data made it through). The result beats its base model in 75% of blind head-to-head matchups and is nearly twice as disciplined about not stating stale market rates as fact. Total cost, end to end: under $100. The weights are open on Hugging Face.
Our AI employees needed a finance brain. Not a bookkeeper — our agents already read the live numbers from the accounting system — but the judgment layer: what do these numbers mean, what’s deteriorating, what should the owner do about it? General models are mediocre at this, and worse, they’re confident when they shouldn’t be. So we built a domain model, and logged everything for anyone who wants to repeat it.
The design decision that shaped everything
The model is built for a consult architecture: the agent brings the live data (from Zoho Books, in our case), the model brings the interpretation. That one decision drove all three trained behaviours: ground every derived figure in the numbers provided (formula shown inline), refuse to state a current market rate as today’s fact, and — when data is missing — name the exact report to pull instead of assuming values. It’s the same grounding philosophy we apply to customer facts, applied to money.
Synthetic data, zero client records
There’s no public corpus of good CFO advice for Indian mid-market companies, and our clients’ books were never an option. So we generated the data: a deterministic scenario matrix (company profile × financial event × owner ask) produced 11,000 briefs, each with an internally consistent financial snapshot — revenue, margins, receivable days, debt — so the arithmetic in every answer could be checked against the question. Teacher models wrote the advisory briefings; a hand-reviewed exemplar set anchored the tone: numbers first, plain speech, healthy pushback when the owner’s framing is wrong.
Our first audit scored 44/100. Good.
Before training, we run a blinded audit: independent judges score a random sample of the training data as if reviewing a junior analyst’s work, with a ship gate of 90. The first round scored 44. Two root causes, both invisible until we looked at output tails: reasoning-mode models were silently spending their token budget thinking — so 86% of outputs were truncated mid-sentence — and the judge only read the head of each output, so it never noticed. And the teachers made money-math mistakes: decimal slips, conflating a credit limit with the amount drawn, currency conversions in the wrong direction.
The gauntlet that fixed it
Every candidate pair now has to survive five independent gates:
| Gate | What it catches |
|---|---|
| 1 · Truncation check (programmatic) | Outputs that ran out of tokens — checked before any judge sees them |
| 2 · Style & quality judge | Jargon walls, dodged questions, lecture-shaped answers |
| 3 · Dedicated math verifier | Recomputes every derived figure against the scenario’s numbers |
| 4 · Second independent verifier | Comparison logic, completeness vs the ask, calibration and hedging — must agree with gate 3 |
| 5 · Blinded 100-sample audit | The junior-analyst review, ship gate ≥ 90/100 |
The audit arc across three rounds: 44 → 85 → 91. Of 11,000 generated briefings, 1,900 survived — a 17% keep rate. That’s the uncomfortable lesson of synthetic data: generating it is cheap; most of it deserves to die. The moat isn’t the generation, it’s the filtering.
Did it actually work?
Release was gated on a blind evaluation over 38 hand-written prompts the model had never seen — including trap questions (asking for a metric that cannot be computed from the data given) and calibration probes (asking for today’s repo rate, MCLR, GST rates). Four independent judges scored anonymised pairs.
| Test | Base model | CFO Brain v1 |
|---|---|---|
| Advisory quality (blind, 0–10) | 3.79 | 5.50 |
| Head-to-head preference | — | 75% of decided matchups |
| Fabrication traps passed | 2/4 | 3/4 |
| “Current rate” questions handled safely | 5/10 | 8/10 |
That calibration line is the one we care most about: a finance model that states a two-year-old repo rate as today’s fact is worse than no model. Trained hedging — “confirm current, here’s where to check” — nearly doubled.
What it cost
Data generation and verification: roughly $50 in teacher-model and judge calls. Training: $22 (two epochs over ~3.8M tokens, 22 minutes on a rented trainer). Evaluation: about $10 of GPU time. Under $100 all-in — the expensive ingredient was the discipline, not the compute. Worth logging honestly: our training platform’s trainer was broken the day we shipped, and five jobs failed with an opaque internal error before a control experiment (same config, previously-proven data) isolated the fault to their side. Infrastructure work is like that; budget calendar time for it.
What we’d do differently
One regression we’re fixing in v1.1: our strictest verifier disproportionately killed the calibration training pairs (they discuss rates, so they trip rate-related checks), and their share of the dataset fell from 14% to 3%. The eval says the behaviour still landed — but we want that margin back. The model, the eval set, and the judging rubric are all public: weights and eval prompts on Hugging Face.
Frequently asked questions
Can you fine-tune a useful LLM on synthetic data alone?
Yes — if you treat generation as the cheap step and verification as the real work. Our CFO model used zero client records: a deterministic scenario matrix generated 11,000 candidate briefings, and a five-gate gauntlet (truncation, style, recomputed math, a second verifier, and a blinded audit) kept only the 17% worth training on.
How much does it cost to fine-tune a 32B model?
For a LoRA adapter: about $22 of training compute for two epochs over ~3.8M tokens in our case, plus roughly $60 in data generation, verification and evaluation. Under $100 end to end — the constraint is data discipline, not GPU budget.
How do you stop a finance LLM from making up numbers?
Three trained behaviours plus testing: ground every derived figure in numbers supplied in the prompt with the formula shown, refuse to state current market rates as fact, and name the missing report instead of assuming values. Then test for it explicitly — trap questions and calibration probes in a blind eval — rather than hoping.
Is the model's output financial advice?
No. It’s an analytical reading of the numbers you provide, built for decision support inside an agent workflow with a human reviewing. Anything consequential should be verified and taken to a qualified professional.
Own the brain your agents think with.
Our models ship open-weight with their evals attached — and they’re already at work inside our AI employees. See the family, or read how we’d build one for your domain.
See the models