- pages read, every field graded against a known answer
Every organisation that still takes paper has someone typing forms in. There are three ways to hand that to a machine: pay a frontier AI model by the page, buy a managed document service, or fine-tune a small open model and run it yourself. They are rarely measured on the same pages. This project measured them. Twelve readers, from Claude, GPT and Gemini to Azure Document Intelligence, classic OCR pipelines, and a 2B and a 4B Gemma model fine-tuned here, read the same - public pages, and every field they wrote was graded by code against the answer the dataset records.
Every figure on this page is read from the project's results files, written by the same code that writes the repository's tables, with a 95% interval behind each one and the date of every price. See how it's built.
The short version, before any chart. Each card links to the evidence further down.
-
On every form type measured, the fine-tune gets as many fields right as Azure's custom-trained model, or more: -. Azure charges - a thousand pages; the fine-tune costs - on a rented gaming GPU.
The head-to-head-
-. No single vendor wins everywhere, so the model has to be picked per document type, on your own pages.
Accuracy against cost-
-. Reading the pixels directly is the single biggest robustness gain on offer.
The degradation curve-
-, from about - of training on one rented card.
How it was trained-
-. For a form a person signs off, a blank is cheaper than a confident wrong value.
Where it is strong-
-. Paying for the biggest model is not where the accuracy is.
The ceilingAnd the limit. No reader reached 99% accuracy on the fields it was confident about, so at that standard a person still checks every field, whichever pipeline you buy. What these models replace is the typing, not the checking. Why, and what was tried.
For each kind of form, the fine-tune the release gate compared (the better of the 2B and the 4B on the development pages) beside Azure Document Intelligence, the cheapest frontier model and the most accurate one. Left: the share of fields right, with its 95% interval. Right: what a thousand pages cost, on a log scale, so each gridline is ten times the last.
Where it earns its place. -
Where it falls short. -
Each mark is one reader on one kind of form: its share of fields right against what a thousand pages cost it, with both 95% intervals as whiskers. Up and to the left is better. The line joins the readers nothing beats on both counts. Pick a form type.
Point at a mark for its figures.
Two open-weights bases from Google, Gemma 4 E2B and E4B, each fine-tuned per kind of form with QLoRA: the base model's weights are frozen and stored in 4 bits, and only small low-rank "adapter" matrices beside its attention and feed-forward layers are trained, on the page images directly. That is what lets a vision model train on one 24 GB card in minutes rather than on a cluster.
The same base model before and after, on the same test pages: the share of fields right.
Each fine-tune was served on its own on one rented - at - an hour, behind the same gateway as the frontier APIs, and read by a closed-loop client with 1, 8 and 32 requests in flight for three minutes each. Pages a second climb with the load until the card is full; the time to read one document climbs too.
In practice. A 2B fine-tune and vLLM fit on one consumer 24 GB card with room for 32 requests at once. No page leaves the building: for records that cannot go to a vendor, that matters more than the price. The card is paid by the hour whether it is busy or not, so what a page really costs depends on how full you keep it, which is what the calculator below is for.
An API is paid by the page. A card is rented by the hour, whether it reads one page or its fill. Below some volume the API is cheaper; above it, the card. Choose the model, put in what you pay, and how hard you would run the card; the measured throughput does the rest.
What this leaves to you. The rent is the only cost counted unless you add one. The throughput is what the card was measured serving on these forms with 32 requests in flight; longer documents or another card will serve a different number. Training is a one-off, minutes to a few hours of rented time here, and labelled examples are the real cost if you have none. And the calculator says nothing about accuracy: the charts above do.
The same pages, degraded in four reproducible steps of noise, blur, skew and lost contrast, calibrated to what they do to OCR's word accuracy. Level 0 is the clean page. The vision models read the pixels and barely notice; the pipelines that read OCR's text lose almost everything.
Point at the chart for each reader's figures at that level.
Real test pages, with every field a reader wrote drawn where the form prints it: green when the grader marked it right, red and dashed when wrong. Pick a reader and slide the scan quality. Registration forms name individuals, so they are never shown here; they appear as numbers only.
Things the measurements said that we did not expect, each with its evidence.
Claude Opus 5.5, Anthropic's largest model, against Claude Sonnet 5.5 on the same subset of test documents. -
- A fine-tune learns the layouts it was shown; a frontier model reads the words. Test on designs it has never seen, or the number flatters it.
- A gap of a point or two between two readers on one form type is inside that noise.
Two of the extra training runs, same data and recipe but a different random seed, learned to wrap their answers in the prompt's schema instead of answering. Inside, the values were often right; graded as written, they scored under 10%. Serve one model without a held-out check and you would not see it.
- Fine-tuning helped everywhere else, by up to - points: measure the base before training it.
- It leaves a field blank rather than guess, which keeps its invented values among the lowest of any reader, and costs it the fields it could have read.
Nothing reached 99% unattended. To let a field through without a person, a reader has to know when it is right. Two signals were tried: whether a frontier model's read of the image agrees with its read of the OCR text, and the fine-tune's own token probability. Set on the development pages to keep accepted fields 99% right, both sent - of test fields to a person. The models save the typing; a person still checks.