Build Log: How Our Marketing LLM Stopped Making Up Statistics
Version one of our marketing model passed its evals and shipped. Then we counted how often it invented statistics: 21 outputs out of 50. The problem wasn’t the model. It was the diet.
Our first fine-tuned marketing model learned to fabricate statistics from its own training data — stat-stuffed synthetic examples taught it that numbers signal quality. The fix was a data rebuild, not a bigger model: an autopsy that cut the old dataset’s filler (nearly 9 in 10 pairs), an anti-fabrication rule baked into 60% of training briefs, per-pair quality judging, and a blind eval gate. v2 scores 7.16 vs its base’s 4.79, wins 86% of head-to-head matchups, fabricates a third as often as its base — and passed its acceptance test by writing the copy on our own website. Weights are open on Hugging Face.
Our marketing AI employees run on an owned model — a fine-tuned adapter that turns positioning briefs into B2B copy. Version one worked well enough to ship. This is the log of why we rebuilt it anyway, and what the rebuild taught us about where model behaviour actually comes from.
The failure we almost didn’t measure
When we benchmarked v1 against candidate base models, we added one count nobody had asked for: how often does each model state a statistic that appears nowhere in the brief? v1’s number was 21 out of 50 outputs. “73% of B2B buyers say…” — plausible, authoritative, invented. For marketing copy that ships under a client’s name, that’s not a quirk; it’s a liability.
The uncomfortable part: v1 fabricated more than untuned base models. Fine-tuning had made it worse. The autopsy explained why — our first synthetic dataset was full of stat-stuffed examples, teacher-written copy that reached for a number whenever it wanted to sound convincing. The model didn’t learn to lie despite the training data. It learned to lie from it.
The autopsy, in numbers
We re-judged the entire v1 dataset with a calibrated scoring model. Nearly 9 in 10 pairs were filler — generic, padded, or stat-stuffed. Only 682 of 13,500 pairs survived into the new dataset as gold examples. We also auditioned teacher models for the regeneration and fired one large, well-known model after it passed the quality bar on just 0.4% of its outputs. The judge doesn’t care about the logo on the teacher.
What v2 was trained on instead
- Structured briefs, not bare instructions — company, ICP, pain points, tone, hard constraints, CTA — so the model learns to follow a real brief’s constraints.
- An anti-fabrication rule: 60% of training briefs explicitly declare “usable facts: none — no invented statistics.” The model learns that a brief without numbers gets copy without numbers.
- Per-pair quality gating: every candidate pair scored by an LLM judge; the kept set’s median score was 9/10, against the old dataset’s 2.
- Leak scrubbing, verified — client and brand names stripped before training and tested absent after it.
The gate: blind eval, then real work
| Metric (50 blind-judged prompts) | Base model | Marketing v2 |
|---|---|---|
| Overall quality (0–10) | 4.79 | 7.16 |
| Head-to-head preference | — | 86% |
| Fabricated statistics (per 50 outputs) | 13 | 4 |
| Truncations / format breaks | 12 | 0 |
Numbers weren’t the final gate, though. The acceptance test was a real deliverable: the model drafted new landing copy for neuralinfrastructure.ai from our positioning brief, judged against the human-written copy it would replace. The drafts were assessed as a significant improvement, refined by a human editor, and shipped — the hero on our homepage came out of that session. A model that only wins benchmarks isn’t done; a model whose work ships is.
What this cost, and what we learned
About $110 end to end — data regeneration and judging, a ~$15 training run, and evaluation GPU time — over two days. The lessons that carried into our finance model: model vices are dataset vices (fix the diet, not the model); measure the failure mode you fear, explicitly, before and after; and give every model one piece of real work as its final exam. The human editor’s notes from the website session didn’t get discarded either — they’re the preference signal for the next training round.
Frequently asked questions
Why do fine-tuned models make up statistics?
Because their training data taught them to. If synthetic training examples reach for authoritative-sounding numbers, the model learns that numbers signal quality — and reproduces the pattern with invented figures. Fine-tuning amplifies whatever the dataset rewards.
How do you stop an LLM from inventing numbers in marketing copy?
Train the behaviour explicitly: our briefs carry a ‘usable facts’ field, and 60% of training examples declare that no statistics are available — teaching the model that no-number briefs get no-number copy. Then count fabrications in a blind eval, before and after.
How much does it cost to fine-tune a marketing LLM?
Our v2 cost about $110 end to end: data regeneration and per-pair judging, roughly $15 of LoRA training compute on a 32B base, and evaluation time — over two days of work.
What is a dogfood acceptance test for an AI model?
A real deliverable, judged like real work, as the release gate. Ours: the model wrote the landing copy for our own website from our positioning brief. It was judged better than the copy it replaced, human-edited, and shipped — that’s when the model was done.
The model that wrote our website can write for yours.
Marketing v2 is open-weight, evals attached, and working inside our AI employees today.
See the models