OCR and Document AI: From Tesseract to Vision-Language Models

Most business data still arrives as documents: invoices, contracts, forms, receipts, ID cards. Getting reliable text and fields out of them takes more than character recognition. This article walks from the classic OCR pipeline to vision-language models that read a page directly, and explains how to tell whether any of them is getting it right.

What OCR does and why documents are hard

Optical character recognition (OCR) converts an image of text into machine-readable characters. Before running it, check whether you need it at all: a PDF exported from a word processor or an invoicing system already contains a text layer, and libraries such as pdfplumber or PyMuPDF extract it exactly. OCR is for scans, photos, faxes and image-only PDFs. Many real archives mix both, so test page by page.

Recognising clean printed characters is largely solved. Documents are hard for other reasons:

• Image quality. Skewed scans, blur, low resolution, compression artefacts, shadows and curved pages in phone photos.
• Layout. Multiple columns, sidebars, headers and footers, text in boxes. Reading the words in the wrong order turns a correct transcription into nonsense.
• Tables. Rows, columns, merged cells, tables without ruling lines and tables that continue on the next page.
• Handwriting, stamps and checkboxes. Huge variation between writers, and marks that matter but are not text.
• Scripts and languages. Right-to-left scripts, thousands of Chinese characters, and documents that mix languages on one page.

And the text is rarely the goal. People want the invoice total, the due date or the policy number, which is why OCR is usually the first step of a larger document AI system.

Classic OCR: clean, detect, recognise

A traditional OCR engine runs three stages. Pre-processing converts the image to greyscale, separates ink from background (binarisation), straightens skewed pages and removes noise. Text detection finds where the text is and returns a bounding box for each line or word. Recognition crops each line and converts it into characters. Modern recognisers read a whole line at once with a neural network trained with CTC, the same alignment trick used in speech recognition. The output is text plus a box and a confidence score for each word.

Tesseract

Tesseract is the best-known open-source engine. It was developed at HP between 1984 and 1994, released as open source in 2005 and developed at Google for many years; it is now maintained by its community. Version 4 added a recogniser based on LSTM neural networks that read whole text lines, which considerably improved accuracy over the original character-shape classifier. It recognises more than 100 languages, runs on a CPU and is easy to call from Python:

# Requires the tesseract binary (apt install tesseract-ocr, brew install tesseract)
# and: pip install pytesseract pillow
from PIL import Image
import pytesseract

img = Image.open("scan.png")

# --oem 1: LSTM engine only; --psm 6: treat the page as one uniform block of text
print(pytesseract.image_to_string(img, lang="eng", config="--oem 1 --psm 6"))

# Word-level boxes and confidences, useful for flagging doubtful words
data = pytesseract.image_to_data(img, lang="eng", output_type=pytesseract.Output.DICT)
for word, conf in zip(data["text"], data["conf"]):
    if word.strip() and float(conf) < 60:
        print(f"check: {word!r} (confidence {conf})")

Tesseract does well on clean, printed, single-column pages scanned at around 300 DPI. It struggles with phone photos, complex layouts and handwriting, and the page segmentation mode (--psm) often matters more than any other setting. It returns text and boxes, not structure: tables come out as loosely spaced lines.

Deep-learning toolkits and transformer recognisers

Newer open-source toolkits replace each stage with a trained network. Baidu's PaddleOCR combines a neural text detector, a classifier that corrects text orientation and a line recogniser, and adds modules for layout analysis and table recognition. docTR and EasyOCR offer similar detection-plus-recognition pipelines in PyTorch. These toolkits handle photos, rotated text and mixed layouts much better than Tesseract, at the price of more setup and, for speed, a GPU.

TrOCR, from Microsoft, rebuilt the recogniser itself as a transformer: an image encoder reads the cropped line and a text decoder generates its characters one token at a time, like a small language model. It is strong on handwriting, but it still reads one line at a time, so it needs a detector in front of it.

Layout, reading order and tables

Layout analysis divides a page into regions and labels them: title, paragraph, list, table, figure, caption, header, footer. Detectors for this are trained on annotated page collections such as PubLayNet and DocLayNet. A second step decides reading order. Simple top-to-bottom, left-to-right sorting fails on two-column articles and on forms where labels sit beside their values.

Tables need their own model for table structure recognition, which identifies rows, columns, header cells and merged cells and outputs HTML, Markdown or CSV. Tables without ruling lines and tables that span pages remain the hardest case. Open-source converters such as IBM's Docling and PaddleOCR's structure modules bundle layout, reading order and table models into one tool that turns a PDF into Markdown or JSON, a common first step before indexing documents for retrieval.

Document AI: from text to fields

Document AI is the umbrella term for the tasks built on top of OCR and layout:

• Classification: is this page an invoice, a payslip, a contract or a passport?
• Key-information extraction: pull named fields such as supplier, total and due date.
• Key–value linking: on forms, connect each printed label to the value written next to it.
• Document question answering: answer a free-form question about a page.
• Splitting: separate a 40-page scan into the individual documents it contains.

The pre-LLM approach to extraction was rules: regular expressions and positional templates such as "the total is the number to the right of the word Total". They are cheap and transparent, and they break as soon as a supplier changes its layout. Layout-aware models such as LayoutLMv3 made extraction learnable. The model receives the OCR words, the position of each word on the page and the page image, and is fine-tuned to tag each word with a field label. It is accurate and cheap to run, but it depends on the OCR step and needs labelled examples for every document type.

OCR-free models and vision-language models

Donut (Document Understanding Transformer) removed the separate OCR step. An image encoder reads the whole page and a text decoder writes the answer directly as a token sequence that converts to JSON. It is pretrained to read the text in document images and then fine-tuned for a task such as receipt parsing. With no OCR stage, OCR errors cannot propagate, but you still need training data for each task.

Vision-language models (VLMs) take this further. A general-purpose multimodal LLM reads a page image and follows instructions: "extract these fields as JSON", "convert this page to Markdown", "which date is the payment deadline?". They work zero-shot on layouts they have never seen, read handwriting and stamps, and use context to tell the issue date from the due date. As of 2026 there are also smaller open-weight VLMs trained specifically to convert pages into Markdown, which can be run on your own hardware.

VLMs have real costs. Each page costs far more compute than classic OCR, pages are resized or tiled to fit the model's input, so very small print can be lost, and most models do not return reliable coordinates for what they read. Their biggest risk is covered in the evaluation section below.

Structured extraction with a schema

The most useful pattern is to define the output you want as a JSON Schema and ask the model, either a VLM on the image or an LLM on OCR text, to fill it. Many APIs can constrain generation so the response always parses against the schema. Two design choices matter: make absent fields nullable, so the model is not pushed to invent a value, and keep field descriptions precise.

{
  "type": "object",
  "properties": {
    "supplier_name":  { "type": "string" },
    "supplier_vat_id": { "type": ["string", "null"], "description": "VAT number exactly as printed, or null" },
    "invoice_number": { "type": "string" },
    "issue_date":     { "type": "string", "format": "date" },
    "due_date":       { "type": ["string", "null"], "format": "date" },
    "currency":       { "type": "string", "enum": ["EUR", "GBP", "USD"] },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "description": { "type": "string" },
          "quantity":    { "type": "number" },
          "unit_price":  { "type": "number" },
          "amount":      { "type": "number" }
        },
        "required": ["description", "amount"]
      }
    },
    "net_total":   { "type": "number" },
    "tax_total":   { "type": "number" },
    "gross_total": { "type": "number" }
  },
  "required": ["supplier_name", "invoice_number", "issue_date", "currency", "line_items", "gross_total"],
  "additionalProperties": false
}

A response that matches the schema can still be wrong. Business rules catch much of what the schema cannot:

def check_invoice(inv: dict, ocr_text: str) -> list[str]:
    """Return a list of problems; an empty list means the invoice passes."""
    problems = []
    lines = round(sum(item["amount"] for item in inv["line_items"]), 2)
    if inv.get("net_total") is not None and abs(lines - inv["net_total"]) > 0.01:
        problems.append(f"line items sum to {lines}, net total is {inv['net_total']}")
    if None not in (inv.get("net_total"), inv.get("tax_total")):
        if abs(inv["net_total"] + inv["tax_total"] - inv["gross_total"]) > 0.01:
            problems.append("net + tax does not equal gross")
    if inv.get("due_date") and inv["due_date"] < inv["issue_date"]:   # ISO dates compare as strings
        problems.append("due date is before issue date")
    if inv["invoice_number"] not in ocr_text:                          # cross-check against plain OCR
        problems.append("invoice number not found in OCR text")
    return problems

Documents that fail any check go to a person, a standard human-in-the-loop step. These checks are a form of guardrails, and in production they matter as much as the choice of model. For wiring this into larger flows, see LLM automation.

Measuring accuracy

For raw transcription, the standard metrics are character error rate (CER) and word error rate (WER): substitutions, deletions and insertions divided by the length of a reference transcript. For extraction, measure what the business uses:

Build the test set from your own documents, including the worst scans, and keep it as a regression suite of evals so that a model or prompt change cannot silently lower accuracy.

Choosing an approach

Managed services include Amazon Textract, Google Cloud Document AI and Azure AI Document Intelligence (names as of 2026). Many production systems are hybrids: an OCR engine supplies text, boxes and confidences; a VLM or LLM maps them to a schema; validation rules decide what is accepted; and people review the rest. Start with the simplest tool that meets your accuracy target on your own test set, and move up only where it falls short.