Document AI: Automating Data Extraction from Unstructured Documents at Scale

Every enterprise has documents that need to be read and acted upon: invoices to be processed, contracts to be reviewed, forms to be digitised, reports to be summarised. For decades, this meant either manual data entry or fragile template-based OCR that breaks whenever a document format changes.

The combination of modern OCR, layout analysis, and LLMs has produced a new generation of document AI systems that handle format variability gracefully. This guide covers how to build production document extraction pipelines that scale.

The Document Extraction Problem

Document extraction sounds simple: read a document, extract specific fields, return structured data. The reality is more complex:

Format variability: An invoice from one vendor looks nothing like an invoice from another. The "Invoice Total" field might be labelled "Amount Due," "Total Payable," "Grand Total," or just a number at the bottom with no label.

Layout complexity: Tables, nested tables, headers and footers, multi-column layouts, footnotes, and embedded images all create parsing challenges that line-by-line text extraction cannot handle.

Handwriting and poor scan quality: Many enterprise documents are scanned paper forms. Scan quality varies; handwriting is notoriously difficult for standard OCR.

Context dependency: A contract clause requires understanding context to interpret — a dollar amount in the preamble means something different than the same amount in the penalty clause.

Scale: A large enterprise may process thousands to tens of thousands of documents per day. The pipeline must be reliable, parallelisable, and observable.

The Document AI Stack

A production document extraction pipeline has four stages:

1. Ingestion and preprocessing — format conversion, quality enhancement
2. Document parsing — text and layout extraction
3. Extraction — identifying and pulling specific information
4. Validation and output — structured output with confidence scoring

Stage 1: Ingestion and Preprocessing

Convert all documents to a normalised format before parsing. Common conversions:

• DOCX → PDF: Use LibreOffice headless for reliable conversion
• Email attachments: Extract from MIME containers; handle base64 encoding
• Images: Detect orientation and rotate; enhance contrast for poor quality scans

For scanned documents, apply preprocessing before OCR:

• Deskewing: Correct rotated or skewed scans (Pillow, OpenCV)
• Noise removal: Remove scan artifacts that interfere with character recognition
• Contrast enhancement: CLAHE (Contrast Limited Adaptive Histogram Equalization) for faded or low-contrast scans

Poor preprocessing is the primary cause of low OCR accuracy. A 5% improvement in OCR accuracy from preprocessing can increase extraction accuracy by 15–20%.

Stage 2: Document Parsing

Traditional OCR (Tesseract, PaddleOCR)

Tesseract is the most widely deployed open-source OCR engine. PaddleOCR from Baidu performs better on complex layouts and non-Latin scripts.

Both extract text position (bounding boxes) alongside character content. This positional information is critical for layout-aware processing — knowing that a value appears directly below a column header is necessary for table extraction.

Layout-aware parsing

Modern layout analysis models (LayoutLM, DiT, Donut) understand document structure semantically — they can identify table regions, form fields, headers, and body text. This is significantly more capable than pure OCR for structured documents.

Cloud document AI APIs

• AWS Textract: Strong table and form extraction, page geometry information, handwriting support
• Google Document AI: Similar capabilities with good multilingual support
• Azure AI Document Intelligence (formerly Form Recognizer): Strong for forms and invoices with prebuilt models

For documents with complex layouts or handwriting at scale, cloud APIs often outperform self-hosted solutions in accuracy while reducing infrastructure complexity. Cost: $1–15 per 1,000 pages depending on feature set.

Stage 3: Extraction

This is where modern LLMs have transformed document AI.

Rule-based extraction (legacy approach)

Regular expressions and keyword matching. Fast, deterministic, and works perfectly when document formats are fixed. Fails immediately when format changes.

Use for: highly standardised internal documents with no format variability.

Template-based extraction

Define field locations by template (e.g., "Invoice Total is always at coordinates X, Y on page 1"). Works for documents from a single source with consistent layout.

Use for: documents from a limited set of known sources where layout is stable.

LLM-based extraction (modern approach)

Pass parsed document text to an LLM with a structured extraction prompt. The LLM understands context and can identify fields regardless of their label or position in the document.

from pydantic import BaseModel
from openai import OpenAI

class InvoiceExtraction(BaseModel):
    vendor_name: str
    invoice_number: str
    invoice_date: str
    due_date: str | None
    line_items: list[dict]
    subtotal: float | None
    tax: float | None
    total: float
    payment_terms: str | None

def extract_invoice(document_text: str) -> InvoiceExtraction:
    client = OpenAI()

    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "Extract invoice data from the provided document text. "
                           "Return null for fields that cannot be found. "
                           "Dates should be in YYYY-MM-DD format. "
                           "Amounts should be numeric values without currency symbols."
            },
            {"role": "user", "content": document_text}
        ],
        response_format=InvoiceExtraction,
    )

    return response.choices[0].message.parsed

Using Pydantic models with OpenAI's structured output API guarantees that the LLM output matches your schema — the API will not return a response that fails Pydantic validation.

When LLM extraction shines:

• Variable-format documents from many sources
• Documents requiring contextual interpretation
• Complex nested structures (line items within tables within sections)
• Mixed document types in a single pipeline

LLM extraction limitations:

• Per-token cost (scales linearly with document length)
• Latency (2–8 seconds per document with GPT-4 class models)
• Non-deterministic (same document can produce slightly different outputs)
• Token limits (very long documents need chunking strategies)

Stage 4: Validation and Output

Raw LLM extraction output needs validation before use in downstream systems.

Schema validation: Pydantic handles this automatically for typed fields. Dates, numbers, and enums are checked at extraction time.

Business logic validation: Domain-specific rules that catch extraction errors:

• Invoice total should approximately equal sum of line items (within ±2% for rounding)
• Due date must be after invoice date
• Tax percentage should be within expected range (0–30%)
• Vendor name should match a known vendor if from an approved vendor list

Confidence scoring: For uncertain extractions, generate a confidence score. GPT-4 with logprobs enabled can provide token-level confidence. Alternatively, use a secondary extraction pass and compare results — agreement between two passes indicates confidence.

Human review queue: Route low-confidence extractions to a human review interface. The review decision becomes a training example for improving the extraction model or prompt.

Accuracy Measurement

Without measurement, you cannot improve a document extraction system.

Key metrics

Field-level extraction accuracy: For each target field, what percentage of documents have the correct value extracted? Measure separately — a system might achieve 98% accuracy on vendor name but only 85% on line items.

Document-level accuracy: Percentage of documents where all required fields are extracted correctly. This is the metric that matters most for downstream automation — a document with one wrong field may fail validation.

False negative rate: Fields the system marks as not found when they are present. Missing data is often worse than incorrect data for downstream processes.

Processing success rate: Percentage of documents that complete the extraction pipeline without error. Failed documents require manual processing — measure this separately from accuracy.

Building the evaluation dataset

Evaluate on a stratified sample of production documents:

• Cover all document sources and format variations
• Include both easy documents and known edge cases
• Create ground truth by having a human annotator extract fields manually for 100–500 documents
• Review ground truth for consistency across annotators

Run automated evaluation on every pipeline change using this dataset. Track metric trends over time — gradual degradation often indicates upstream data source changes.

Production Architecture

Pipeline orchestration

For high-volume document processing, orchestrate with Apache Airflow or Prefect:

Ingest documents → Preprocess → Parse (OCR/Layout) → Extract (LLM) → Validate → Route (auto-approve or human review) → Output to system of record

Each stage should be independently monitored and restartable. Document processing failures should not lose the original document — maintain a processing log with stage completion status.

Storage patterns

• Raw documents: Object storage (S3, Azure Blob, GCS) with original document preserved
• Parsed text and layout: JSON stored alongside document in object storage
• Extracted structured data: Relational database for querying and reporting
• Processing logs: Per-document audit trail with stage timestamps, error messages, and extraction results

Cost optimisation

For high-volume extraction, tier your model usage:

1. Fast path: Simple document types with standardised formats → rule-based extraction (zero LLM cost)
2. Standard path: Variable-format documents → GPT-4o mini or Claude Haiku (low cost, 80–85% accuracy)
3. Complex path: Difficult documents or low-confidence standard extractions → GPT-4o or Claude Sonnet (higher cost, 92–96% accuracy)
4. Human path: Very low confidence or exception cases → human review queue

This tiering reduces per-document LLM cost by 60–80% compared to sending everything through expensive models.

Specific Document Types

Invoices and purchase orders: Well-understood problem. Several off-the-shelf solutions (AWS Textract Forms, Google Document AI Invoice Processor) achieve 90%+ accuracy on standard invoices. Custom LLM extraction adds value for non-standard or international invoice formats.

Contracts: High complexity. Clause extraction, obligation identification, and date/party extraction are tractable. Full legal analysis requires specialised legal AI tools or human lawyers — do not overstate what extraction can do.

Insurance claims: Mix of forms and supporting documents (photos, reports). Form extraction is tractable; evaluating claim validity from unstructured supporting documents requires more sophisticated approaches.

Medical records: Highly regulated, complex layout, medical terminology. EHR-native formats are better handled by FHIR-aware tools. Scanned medical records require careful handling of PHI and strong access controls.

Getting Started

Start small and expand:

1. Pick one high-volume, high-value document type with clear success metrics
2. Collect 200–500 representative samples from production
3. Build the extraction pipeline with human review for all outputs initially
4. Measure accuracy on the evaluation dataset
5. Define confidence threshold for auto-approval (start conservatively — 95%+ confidence)
6. Gradually increase auto-approval rate as you validate accuracy

The incremental approach builds confidence in the system and creates the evaluation data needed to improve it.

---

We build production document AI systems for enterprise clients — invoice processing, contract extraction, form digitisation, and custom document types. If you are evaluating document AI for a specific use case, talk to our team.