Redefining Technology

Artificial Intelligence

What is Document Intelligence & Automation?

Document intelligence and automation is the use of OCR, layout analysis, and large language models to convert unstructured documents — invoices, work orders, contracts, forms — into validated structured data. Confident extractions post straight into the ERP untouched; low-confidence fields route to a short human queue. Best-in-class teams process invoices at 79% lower cost this way (Ardent Partners).

Atomic Loops EngineeringIndustrial AI delivery teamPublished Updated 17 min read

What is document intelligence & automation?

Document intelligence and automation is the use of OCR, layout understanding, and large language models to convert unstructured documents — invoices, work orders, contracts, claims, forms — into structured data that is validated before it enters a business system. The output is not a scanned image or raw text: it is a set of typed fields — supplier, amount, due date, clause — each carrying a confidence score and each checked against the master data the business already trusts.

The discipline sits one level above plain OCR and one below full workflow automation. OCR reads pixels into characters; document intelligence turns those characters into records; automation routes the records into an ERP, a document management system, or an approval workflow without a person retyping anything. Together, most documents are processed with no human touch and the exceptions land in a short review queue instead of a backlog.

The category exists because most enterprise information never reaches a database in a form anything can query. IDC puts 90% of all data as unstructured: PDFs, scans, email attachments, and handwritten forms, each holding numbers someone will eventually retype by hand.

Adoption has followed the economics. Documents arrive faster than people can key them: backlogs grow, data-entry errors propagate into the ERP, and skilled staff spend their day retyping what a machine can read.

How does document intelligence work?

Document intelligence works as a six-stage pipeline with a confidence gate in the middle: documents are ingested from every channel, OCR and layout analysis turn pages into structured text, an LLM extracts the target fields, validation rules score each field against master data, low-confidence cases route to a reviewer, and confirmed records post into the ERP or DMS. The gate is the design — automation for the confident cases, people for the rest.

  • Document intake — Mail, scans, supplier portals, and ERP exports route into one entry point. Coverage beats elegance: a channel that bypasses intake stays manual.
  • OCR and layout analysis — Deep-learning OCR reads the text; layout models recover the structure — tables, stamps, checkboxes, handwriting — that tells the extractor where each field lives on the page.
  • LLM extraction — A language model maps recognised text onto a typed schema: supplier, line items, dates, clauses. Unlike template OCR, it tolerates layouts it has never seen.
  • Validation rules — Every field is checked against master data — does the supplier exist, does the PO match, do line items sum to the total — and scored for confidence.
  • Confidence-routed human review — Documents passing every check flow straight through; the rest queue for a reviewer who sees which field failed, and why.
  • Delivery into the ERP or DMS — Confirmed records post into the system of record with an audit trail per field, so every number traces back to the pixels it came from.
How a document becomes a validated record

Master data is a first-class input, not an afterthought: every extracted field reaches the gate with a check behind it. Confident documents post straight through; the rest reach a person with the failing field already named.

Read this diagram as a list
  1. Document intake — mail · scans · portals
  2. Master data — suppliers · POs · GL codes
  3. OCR, layout & LLM extraction — typed fields per class
  4. Validation & confidence gate — every field scored
  5. Post to ERP or DMS — straight-through
  6. Human review queue — low-confidence fields

The validation loop is what separates a production system from a demo. Every reviewer correction does two jobs: it fixes the document in front of them, and it becomes labelled data that retrains the extractor. Over months the straight-through share climbs and the queue shrinks. Getting documents in from every channel is a data-ingestion problem before it is an AI problem — the model only sees what intake delivers.

Template OCR vs LLM extraction: what changed?

Template OCR extracts fields from fixed pixel coordinates and breaks the moment a supplier redesigns an invoice; LLM extraction reads the document the way a person does and maps it onto a schema, so one system covers a format zoo that would once have needed hundreds of templates. That difference turns document automation from a per-layout engineering project into a per-document-class delivery.

Template OCR vs LLM-based extraction
CriteriaTemplate OCRLLM-based extraction
Setup effortOne template per supplier layout — hundreds at scaleOne schema per document class
New or changed layoutsBreak silently until someone re-templatesHandled by the model; monitored, not rebuilt
Tables, stamps, handwritingPoor — fixed zones miss moved contentLayout models plus LLM context recover most cases
ValidationBolted on afterwards, if at allBuilt in: every field scored against master data
Maintenance costGrows with every supplier addedGrows with document classes, not layouts
Best forOne stable, high-volume formMixed, changing document ecosystems

The caveat is symmetrical: language models can misread with confidence, which is why the validation layer exists. No field posts to the ERP on model output alone — it posts because the supplier exists, the purchase order matches, and the totals reconcile. When a check fails, the response is a human, not a retry.

Which documents should you automate first?

Automate the class with the highest monthly volume and the widest layout variability first — supplier invoices for most businesses, bills of lading in logistics, claims forms in insurance. Volume decides how quickly the pipeline pays for itself; variability decides whether extraction is worth building at all, because a single stable form can often be handled by rules alone.

Which document class to automate first

ThousandsMonthly volumeDozens

Cheap, fast win

  • One standard form at scale
  • Schema plus strict validation rules
  • Highest straight-through rates of any class

Start here

  • Supplier invoices, delivery notes
  • Hundreds of layouts, one schema
  • Payback measured in months

Leave it manual

  • Automation costs more than the keying does
  • Revisit when volume grows

Second wave

  • Contracts, leases, due-diligence packs
  • Value is risk avoided, not minutes saved
  • Same pipeline, different schema

One stable formLayout variabilityHundreds of layouts

Two axes give four honest answers, and only one of them is “start here”. The bottom-right quadrant is real value too — it is just measured in risk avoided rather than minutes saved.

Rank inside the winning quadrant by documents per month × minutes per document, then cut the list at one. Counting document types instead of documents is the usual scoping error: a business listing 40 “types” normally finds three carry 80% of the volume, and the other 37 are a maintenance bill dressed as coverage.

The low-volume, high-variability corner earns its place differently. Nobody saves minutes pulling renewal dates out of 4,000 contracts — but the auto-renewal nobody diarised and the indexation clause nobody applied are expensive, and both are findable only once the archive is a register. Scope that class second, on the pipeline the first one paid for.

What the evidence says

$9.84

average all-in cost to process one invoice manually

Source: Ardent Partners

79%

lower per-invoice cost at best-in-class AP teams

Source: Ardent Partners

25,000 hours

avoidable rework a finance team can save each year

Source: Gartner

What straight-through processing rate should you target?

Agree a straight-through target near 70% for a first high-volume document class — the level we scope invoice pipelines to — and measure it per class, never as a blended average. Straight-through processing (STP) is the share of documents fully handled with no human touch, and it is the number that converts extraction quality into money: every document above the line costs seconds of compute, every one below costs minutes of a person.

The spread between teams doing identical work is the clearest argument for automating it: the same process, on the same documents, costs nearly five times more at the bottom of the benchmark than at the top.

Total cost to process one invoice, by performance quartile

APQC's benchmark of 1,485 organisations: $10 or more per invoice in the bottom quartile, $5.83 at the median, $2.07 or less in the top quartile. The gap is process design and automation, not headcount.

Source: APQC benchmarking database, via CFO.com (opens in a new tab)

View the data
ItemTotal cost per invoice (USD)Note
Bottom quartile10$10 or more per invoice processed
Median organisation5.8$5.83 — a little over double the top quartile
Top quartile2.1$2.07 or less — nearly five times cheaper than the bottom quartile

The arithmetic is concrete. An accounts-payable team handling 100,000 invoices a year at the $9.84 average carries a $984,000 processing bill; a 70% straight-through rate removes the manual cost from 70,000 of those documents — roughly $690,000 a year — before the downstream corrections that never happen because validation caught the error at intake. In our deployments handling time falls from about nine minutes per document to one, and data-entry error rates drop by roughly 90%.

$984k

annual bill for 100,000 invoices at the $9.84 average

$690k

manual cost removed by a 70% straight-through rate

9 → 1 min

handling time per document in our deployments

Report three KPIs per class from the first week of production: straight-through rate, field-level accuracy against a labelled sample, and review latency. Read them together. A rising straight-through rate with stable accuracy means the thresholds have earned their looseness; a rising rate with falling accuracy means the gate is open too far.

How to automate your first document class

The fastest route to production is one document class automated end to end behind a confidence gate — not five in parallel. A first class takes four to six weeks; a first production release inside 90 days is the default we scope to. The pipeline it produces is an end-to-end AI system in miniature — intake, models, validation, integration, monitoring — and every later class reuses it.

  1. Pick one document class by volume and pain (week 1)

    Choose the highest-volume class with the clearest downstream use — supplier invoices, most often. Record the baseline it will be judged against: cost per document, handling time, error rate, backlog size. Agree with operations which fields truly matter before any tuning starts.

  2. Define the extraction schema and validation rules (weeks 1–2)

    Write the typed schema the business will own — which fields, which formats, which are mandatory — and the checks that make a field trustworthy: supplier exists, PO matches, line items sum to the total. Confidence thresholds are a business decision, not an IT default.

  3. Wire every intake channel (weeks 2–3)

    Mail, scans, supplier portals, ERP exports — each routes into the one intake. A channel that bypasses it stays manual forever, and its documents vanish from the rate you report.

  4. Shadow-run against a labelled sample (weeks 3–5)

    Extract in parallel with the manual process and measure field accuracy against labelled documents. Tune for precision first: an extraction error that reaches the ERP erodes trust at roughly ten times its actual size.

  5. Go live behind the confidence gate (weeks 5–6)

    Confident documents post straight through with an audit trail per field; the rest land in a review console showing which check failed and why. Corrections feed back as training data from day one.

  6. Expand class by class

    Compare the measured straight-through rate and handling time against the week-1 baseline, then add the next class on the same pipeline. Scaling is a scheduling decision, not a rebuild.

What happens after the first document class?

After the first class goes live the work stops being modelling and becomes cataloguing: the pipeline exists, so every additional class is a schema, a rule set, and an integration mapping rather than a project. The marginal cost per class falls sharply: a second class typically lands in two to three weeks, against four to six for the first.

From first document class to a document platform

  1. Weeks 1–6

    First class in production

    One class — usually supplier invoices — automated end to end behind a confidence gate, with straight-through rate, field accuracy, and review latency reported from the first live week.

    Decision: measured straight-through rate against the week-1 baseline.

  2. Months 2–4

    Second and third classes

    Delivery notes, remittance advices, purchase orders. Intake, review console, and ERP integration already exist, so each class is a schema and a rule set instead of a build.

    Decision: is the marginal cost per class actually falling?

  3. Months 4–8

    Enrichment and entity matching

    Extracted records are matched against master data — suppliers, contracts, cost centres — and the inconsistencies the business has always carried become countable and fixable.

    Decision: who owns the master-data corrections this surfaces?

  4. Months 8–12

    The archive becomes queryable

    Contracts, claims, and correspondence join the same pipeline, and the extracted registers become an asset analytics and search can use rather than a folder tree nobody can interrogate.

    Steady state: a new class onboards against a template, not a business case.

Each phase ends with a measured decision rather than a deliverable — the next class is funded by the numbers the last one produced.

The phase most teams underestimate is enrichment. Matching extracted records to master data exposes every inconsistency it has always held — three spellings of one supplier, cost centres closed two years ago, contracts filed against the wrong legal entity. That is not an extraction failure; it is the first time anyone has seen the whole set at once. Give the corrections a named business owner before the matching runs.

The compounding return is that structured document data becomes an input to everything else. Invoice lines feed spend analytics, work orders label predictive maintenance models, and contract terms populate obligation registers. The pipeline stops being an accounts-payable tool and becomes the layer that turns documents into data for the business.

Where document automation fails

Document automation fails at the edges of the system, not in the model: a channel that never routes into intake, a threshold nobody in the business agreed, a review queue with no owner. Past week three of a delivery, extraction quality is rarely the blocker — integration and ownership are, and both are visible up front.

  • The pipeline extracts but posts nowhere — A perfect JSON payload that a person copies into the ERP has moved the typing, not removed it. Integration into the system of record is the deliverable; extraction is one step inside it.
  • One intake channel is left out — The supplier portal finance logs into manually, the shared mailbox one team owns. Those documents stay manual and vanish from the straight-through rate, which then reports a number nobody can reconcile.
  • Accuracy is measured on the easy documents — A sample of clean, born-digital PDFs produces a flattering figure and a system that fails on the fax. Build it from the real mix, worst scans included.
  • Thresholds are set by IT — How confident a field must be before it posts unattended is a risk decision about money and audit, not a hyperparameter. Unowned by the business, the first surprise in the ledger stops the programme.
  • Nobody owns the review queue — Exceptions arrive from day one. Without a named owner, the queue becomes the new backlog and straight-through processing gets blamed for it.

Exceptions themselves are not failure. Ardent Partners puts the average invoice exception rate at 18.4%: nearly one in five invoices carries something no model was going to resolve unattended — a price mismatch, a missing purchase order, a duplicate submission. The goal is not zero exceptions; it is that every exception reaches a person with the failing check named.

Work the list honestly. A “no” against master data or the labelled sample is not a reason to wait — it is the first two weeks of the project, and far cheaper to find now than in week five.

Key terms

Intelligent document processing (IDP)
The industry term for the whole pipeline: capture, OCR, layout analysis, extraction, validation, and routing into a business system. Document intelligence names the same discipline by what it produces — validated records — rather than by software category.
Straight-through processing (STP)
The share of documents processed end to end with no human touch. Measured per document class and never blended, it is the KPI that turns extraction quality into money; a first invoice class is typically scoped near 70%.
Layout analysis
The model layer that recovers a page's structure — tables, columns, stamps, checkboxes, signature blocks — so an extractor knows a number in a table's third column is a line-item amount, not the invoice total.
Confidence gate
The routing rule deciding whether a document posts automatically or queues for review, based on per-field confidence scores and validation results. Its thresholds are a business risk decision, owned by finance or operations rather than IT.
Three-way match
The accounts-payable check reconciling an invoice against its purchase order and goods-receipt note before payment. It is the canonical validation rule in invoice automation: extraction supplies the fields, the match decides whether they can be trusted.
Master data
The authoritative business records extraction is validated against — suppliers, purchase orders, contracts, cost centres, GL codes. Extraction alone produces plausible text; extraction checked against master data produces records a system of record will accept.

Frequently asked questions

The questions finance, operations, and legal teams ask before automating a document workflow.

What is the difference between OCR and document intelligence?

OCR converts pixels into characters; document intelligence converts characters into validated business records. An OCR engine can read '4,820.00' off a scan. A document intelligence system knows that figure is the invoice total, checks it against the purchase order and the line-item sum, scores its confidence, and decides whether a human needs to see it before it posts to the ERP.

How accurate is LLM document extraction?

Accuracy is only meaningful per field, per document class, measured against a labelled sample — printed invoices score far higher than handwritten forms or degraded scans. The engineering answer is not a perfect model: it is validation against master data plus confidence routing, so residual errors are caught in review instead of posting to the ERP.

Can document automation handle handwriting and poor-quality scans?

Yes, with a per-class accuracy figure rather than a promise. Layout models read stamps, checkboxes, and table structure reliably; handwriting and low-resolution faxes score materially lower and belong behind a tighter confidence threshold, not outside the pipeline. Measure field accuracy on a labelled sample that includes your worst documents, then set the gate so hard cases route to a person instead of posting quietly.

Which documents should you automate first?

The single class with the highest volume and the clearest downstream use — supplier invoices for most businesses, bills of lading in logistics, claims forms in insurance. One class automated end to end, with a measured straight-through rate, beats five classes half-done: it proves the confidence gate, the review workflow, and the ERP integration that every later class reuses.

How long does document automation take to implement?

Four to six weeks for the first document class, with a first production release inside 90 days as the default goal. The critical path is rarely the model — it is agreeing which fields matter, wiring every intake channel into the pipeline, and connecting validation rules to master data. Later classes reuse all of that and arrive faster, typically in two to three weeks.

What is a good straight-through processing rate?

For a first document class, agree a target near 70% — the level our invoice pipelines are scoped to — and treat anything above it as threshold tuning, not a finish line. Best-in-class AP organisations straight-through-process 1.8 times more of their invoice volume than their peers (Ardent Partners), and that gap is most of the cost difference between them.

Does document automation replace data-entry teams?

It changes the work from typing to exception handling. The review queue holds only low-confidence documents, and each reviewer decision does two jobs: it fixes the document and becomes labelled training data that raises the straight-through rate. The queue shrinks over time, but the judgement work — vendor disputes, anomalies, new document types — stays human.

Ask us a question directly

Sources

  1. IDC, “Untapped Value: What Every Executive Needs to Know About Unstructured Data” (2023) (opens in a new tab)
  2. Deloitte, Automation with Intelligence (2020) (opens in a new tab)
  3. Ardent Partners, State of ePayables (opens in a new tab)
  4. APQC benchmarking database, via CFO.com (opens in a new tab)
  5. Gartner (opens in a new tab)

Related reading

Put your document backlog on a 90-day path to production

A 30-minute consultation maps your document classes, master data, and integration points into a scoped plan — first class automated in four to six weeks, with a straight-through target agreed up front.

Book a consultationExplore the document intelligence service

Last updated: