-
A fine-tuned 2B or 4B model beats Azure Document Intelligence on half the forms, and matches it on the rest
-. Azure charges - a thousand pages; the fine-tune costs - on a rented gaming GPU.
The head-to-head- pages processed, every field graded against a known answer
A small vision model you own beats Azure's document AI on half the forms tested, and matches it on the rest, for a fraction of the price
Every organisation that still takes paper has someone typing forms in. There are three ways to hand that to a machine: pay a frontier AI model by the page, buy a managed document service, or fine-tune a small open model and run it yourself. They are rarely measured on the same pages. This project measured them. Twelve readers, from Claude, GPT and Gemini to Azure Document Intelligence, classic OCR pipelines, and a 2B and a 4B Gemma model fine-tuned here, read the same - public pages, many of them again and again: clean and degraded, development and test, three training seeds, second reads. Every field they wrote, on every one of those - page reads, was graded by code against the answer the dataset records.
-
-
-
-
than Azure Document Intelligence, for the same pages
-
-
of fields right from the best frontier model
-
-
pages processed and graded, field by field
-
Every figure on this page is read from the project's results files, written by the same code that writes the repository's tables, with a 95% interval behind each one and the date of every price. See how it's built.
What we found
The short version, before any chart. Each card links to the evidence further down.
-
The frontier vision models still read best, and which one depends on the form
-. No single vendor wins everywhere, so the model has to be picked per document type, on your own pages.
Accuracy against cost-
A bad scan barely moves a vision model, and breaks OCR
-. Reading the pixels directly is the single biggest robustness gain on offer.
The degradation curve-
Fine-tuning is what makes a small model usable
-, from about - of training on one rented card.
How it was trained-
On two form types it invents fewer values than any frontier model
-. For a form a person signs off, a blank is cheaper than a confident wrong value.
Where it is strong-
The cheapest frontier model is close on clean, printed forms
-. Paying for the biggest model is not where the accuracy is.
The ceilingAnd the limit. No reader reached 99% accuracy on the fields it was confident about, so at that standard a person still checks every field, whichever pipeline you buy. What these models replace is the typing, not the checking. Why, and what was tried.
The fine-tuned model, head to head
For each kind of form, the fine-tune the release gate compared (the better of the 2B and the 4B on the development pages) beside Azure Document Intelligence, the cheapest frontier model and the most accurate one. Left: the share of fields right, with its 95% interval. Right: what a thousand pages cost, on a log scale, so each gridline is ten times the last.
Where it earns its place. -
Where it falls short. -
Why it is a fair fight with Azure
- Same pages, same grader. Both read the same test documents, at the same resolution, and one programmatic grader scored every field of both.
- Both were trained on the forms. Azure's custom neural models were trained on 100 labelled documents of each registration and ad-buy training split and 30 of each tax-form face; the fine-tunes on -. On registration and ad-buy forms neither saw a page of the form design it was tested on.
- Azure was not handicapped on receipts. CORD's receipts went to Azure's own prebuilt receipt model, untrained, at its lower - price.
- The prices are list prices, dated. Azure's from its retail price API (-), the fine-tune's from the measured GPU time at - (-).
Every reader, accuracy against cost
Each mark is one reader on one kind of form: its share of fields right against what a thousand pages cost it, with both 95% intervals as whiskers. Up and to the left is better. Pick a form type.
Point at a mark for its figures.
How it was fine-tuned
Two open-weights bases from Google, Gemma 4 E2B and E4B, each fine-tuned per kind of form with QLoRA: the base model's weights are frozen and stored in 4 bits, and only small low-rank "adapter" matrices beside its attention and feed-forward layers are trained, on the page images directly. That is what lets a vision model train on one 24 GB card in minutes rather than on a cluster.
What fine-tuning bought, per form type
The same base model before and after, on the same test pages: the share of fields right.
The recipe
How long each adapter took to train
From scanned page to served model
- 1Pair each page with its answer. The page image, and the fields the dataset records for it as the exact JSON the grader marks. Form designs held out for testing never reach training, and a held-out document refuses the training set.
- 2Train on the answer only. One turn of page images and the same instruction every other reader is sent, one turn of the JSON. The loss falls on the answer's tokens, so it learns to read, not to recite the prompt.
- 3Keep the vision tower whole. The image encoder, its projector and the output layer stay at full precision; the adapter trains the projector and the language layers. A first run that quantised the tower failed at its first step.
- 4Checkpoint from the first steps. Rented cards can be taken back, so every checkpoint goes to object storage and a run resumes from its newest.
- 5Serve it plainly. vLLM on one RTX 3090 at temperature 0. Forcing the output into the schema's grammar was tried and rejected: it pushed the models off the format they were trained on.
- 6Grade it once, publish it. Each test set read once, graded by the same code as every other reader, and the adapters published with cards that list their failure modes.
Serving it: what one card does
Each fine-tune was served on its own on one rented - at - an hour, behind the same gateway as the frontier APIs, and read by a closed-loop client with 1, 8 and 32 requests in flight for three minutes each. Pages a second climb with the load until the card is full; the time to read one document climbs too.
In practice. A 2B fine-tune and vLLM fit on one consumer 24 GB card with room for 32 requests at once. No page leaves the building: for records that cannot go to a vendor, that matters more than the price. The card is paid by the hour whether it is busy or not, so what a page really costs depends on how full you keep it, which is what the calculator below is for.
Your break-even
An API is paid by the page. A card is rented by the hour, whether it reads one page or its fill. Below some volume the API is cheaper; above it, the card. Choose the model, put in what you pay, and how hard you would run the card; the measured throughput does the rest.
Price the API at what a reader cost on these forms
-pages a month and the card is the cheaper
-per 1,000 pages on the card, at this load
-a month paying by the page
-a month on rented cards
What this leaves to you. The rent is the only cost counted unless you add one. The throughput is what the card was measured serving on these forms with 32 requests in flight; longer documents or another card will serve a different number. Training is a one-off, minutes to a few hours of rented time here, and labelled examples are the real cost if you have none. And the calculator says nothing about accuracy: the charts above do.
What a bad scan does
The same pages, degraded in four reproducible steps of noise, blur, skew and lost contrast, calibrated to what they do to OCR's word accuracy. Level 0 is the clean page. The vision models read the pixels and barely notice; the pipelines that read OCR's text lose almost everything.
Point at the chart for each reader's figures at that level.
Watch them read a page
Real test pages, with every field a reader wrote drawn where the form prints it: green when the grader marked it right, red and dashed when wrong. Pick a reader and slide the scan quality. Registration forms name individuals, so they are never shown here; they appear as numbers only.
Form type
Every reader on these pages, at this scan quality
Surprises
Things the measurements said that we did not expect, each with its evidence.
The dearest model buys accuracy only on some forms
Claude Opus 5.5, Anthropic's largest model, against Claude Sonnet 5.5 on the same subset of test documents. -
A new form design hurts the fine-tune, not the frontier model
- A fine-tune learns the layouts it was shown; a frontier model reads the words. Test on designs it has never seen, or the number flatters it.
Reading twice is not reading the same
- A gap of a point or two between two readers on one form type is inside that noise.
One seed learned the wrong format
Two of the extra training runs, same data and recipe but a different random seed, learned to wrap their answers in the prompt's schema instead of answering. Inside, the values were often right; graded as written, they scored under 10%. Serve one model without a held-out check and you would not see it.
The untuned 4B already reads registration forms
- Fine-tuning helped everywhere else, by up to - points: measure the base before training it.
Azure is careful, not accurate
- It leaves a field blank rather than guess, which keeps its invented values among the lowest of any reader, and costs it the fields it could have read.
What this does not show
Nothing reached 99% unattended. To let a field through without a person, a reader has to know when it is right. Two signals were tried: whether a frontier model's read of the image agrees with its read of the OCR text, and the fine-tune's own token probability. Set on the development pages to keep accepted fields 99% right, both sent - of test fields to a person. The models save the typing; a person still checks.
- Old, public, printed forms. FARA registrations from the 1960s on, FCC ad-buy orders, Indonesian receipts and NIST's synthesised 1988 tax forms. No handwriting, no long documents, no form whose type is unknown.
- Small test sets on two form types. CORD's 100 test receipts and NIST's 500 pages over twenty form faces are too few for the release gate to decide field by field at three points, so its verdict there is "not decided", not "pass".
- The release gate blocks the fine-tunes on registration and ad-buy forms against the frontier models: -. They are cheaper, not equivalent.
- Prices move. Every price here carries the day it was read. The calculator is there so you can put in today's.
- Training is not in the cost per page. Neither the fine-tune's rented hours nor Azure's - training hours (- past its free hours) are in the per-page figures; both are one-off costs to spread over the months you run.