Purpose and development
What document understanding preserves
The center panel of Abraham’s Formatting Strategy slide at 11:31 shows how HTML records the relationship. In the markup, Q spans two rows, so it still occupies the first column in the second header row. Rev spans the next two columns. That places the second row’s US and EU headers under Rev. The slide also shows a simple quarter-and-revenue table in Markdown at left and a prose description for retrieval at right.
| Operation | Concrete input | Output | Verification |
|---|---|---|---|
| Native text extraction | Encoded text on a PDF invoice | Strings and available page positions | Compare copied characters with the visible print. |
| OCR | A scan of a printed price | A transcript of the digits and decimal mark | Inspect the marks at the recognized location. |
| Layout and table reconstruction | Positioned words, alignment and ruling lines | Cells, rows and header spans | Trace each regional column to its parent heading. |
| Field extraction | An invoice and a line-item schema | One record per item with its description, quantity and price | Check that all three values belong to that item. |
The same table can serve two consumers. A question about how revenue changed over time might share few words with a grid of numbers. Abraham describes indexing a natural-language description to help find the table, then supplying its structural HTML when a model needs to read the values. Liu makes a separate decision about effort. Prepare a collection economically for bulk lookup, then return to the source for detailed extraction and citations when a particular field matters. A system can use both designs.
For a searchable archive, recovering a distinctive phrase may be enough. For data entry, an invoice total and the balance after a deposit answer different questions even when both amounts are read correctly. Source fidelity means agreement with what the document says, including the labels and qualifications that affect its meaning. Faithful conversion can preserve an author’s factual mistake just as accurately as a correct statement.
From printed characters to document models
An early commercial reader had a narrowly defined job. Farrington’s machine, sold in 1956, read printed numbers on credit-sales invoice cards and punched the information into those same cards for existing equipment. Controlling the input was one way to make recognition practical. OCR-A and OCR-B were typefaces developed for optical reading in the 1960s. ISO 1073 standardized their character shapes in 1976. Exception handling also belonged to the machinery of data entry. A 1969 IBM guide describes the 1287 retaining successfully read characters and displaying unreadable ones for keyboard correction. Smithsonian collection record · OCR font history · IBM guide
A blind reader choosing a book had no control over the printer’s typeface. Kurzweil Computer Products combined a flatbed scanner, recognition across typefaces, known as omnifont recognition, and speech synthesis in a reading machine announced in January 1976. Imperfect print posed another problem. During Tesseract’s development at HP from 1984 to 1994, its recognizer used clearly read words on a page to adapt a classifier before revisiting uncertain words. Page layout came from a separate component. HP released the engine as open source in late 2005. Kurzweil’s account · Ray Smith’s account of Tesseract
Digital document interchange developed along a parallel path. At Adobe, John Warnock’s 1991 Camelot proposal addressed the difficulty of reproducing an application document on a computer with different software and fonts. PDF could package font shapes and explicit page drawing operations. That gave software access to encoded page content and made recovering characters from a scan unnecessary for some files. Reproducing a two-column page still left open how a program should read it aloud. Warnock’s oral history
Learning from complete transcripts reduced the need to mark individual character boundaries. Graves and colleagues introduced their sequence-alignment method in 2006 and tested it on speech. In 2013, Breuel and colleagues applied it to printed lines, including touching characters. Interpreting fields posed a further problem. Yiheng Xu and colleagues at Microsoft trained LayoutLM with recognized words and their page positions in a 2019 preprint published at KDD in 2020. Geewook Kim and colleagues at NAVER developed Donut to generate structured content from page images, first reported in 2021 and published at ECCV in 2022. These systems start with different evidence. Choosing among them begins with inspecting the file. Printed-line recognition · LayoutLM · Donut
Inputs and representation
Inspect the available content
A born-digital document originates in software. Its file may contain text, drawings, form fields or tags that describe logical structure. A scan supplies pixels. A searchable scan can look identical to the original image while supplying an invisible OCR transcript when someone selects text. Even one page can be mixed. If a typed heading sits above a scanned form, selecting the heading tells you little about the entries below it. Compare the text available in the target region with the rendered page before choosing an extraction path.
A glyph is the shape a font draws. A PDF can draw the right glyph and return the wrong Unicode character when copied because drawing and character extraction use different mappings. Inspect a disagreement before replacing the text. OCRmyPDF’s release notes describe a defect that left an old OCR layer inside an embedded page object and added a second reading. Replacing text can also invalidate tags pointing to the old content. When usable native text covers only part of a page, retain it and recognize the uncovered writing. PDF mapping diagnosis · OCRmyPDF processing modes
Choosing an extraction path within a page
ExampleOne searchable page can require different operations in different regions.
Read the diagram as text
- Inspect region against render. Check both coverage and character fidelity.
- Keep encoded text.
- Recognize the writing.
- Recognize gaps and combine. Retain the location of each reading.
- Inspect mappings and layers.
- Inspect region against render → Keep encoded text: Faithful coverage.
- Inspect region against render → Recognize the writing: Image-only writing.
- Inspect region against render → Recognize gaps and combine: Partial faithful coverage.
- Inspect region against render → Inspect mappings and layers: Conflicting readings.
Native structure can extend beyond the visible body. WordprocessingML encodes paragraphs and runs of text with shared properties, while comments, headers and footnotes can occupy separate package parts. Inventory the pages and parts in scope against an identified source version. Record unsupported or unreadable material so that a missing part cannot pass downstream as an empty one. Preserving the source snapshot establishes the start of its provenance, the record of origins and transformations. WordprocessingML structure · Origins, times and permitted use
Retain document relationships
An intermediate document representation gives software identifiable objects to work with before it exports strings or records. A page can contain regions, a region can contain lines, and a line can contain words. Containment answers where an item belongs in this hierarchy. Consider a page with two pictures and a caption. All three share the same page parent. Following a separate link from the caption to one picture tells the program which picture it describes. A model-generated description can be attached to the same picture while remaining distinguishable from the author’s printed caption.
A bounding box is a rectangle locating an item in a specified coordinate frame. DoclingDocument can attach boxes to items, order the children of its body tree in reading sequence, and keep repeating page furniture such as footers in a separate tree. An item reference addresses this conversion. Retaining the source version and spatial location makes it possible to inspect the item even if another conversion gives it a different identifier.
Which picture does the caption describe?
ExampleA relationship between siblings carries meaning that their shared parent cannot express.
Read the diagram as text
- Page.
- First picture.
- Second picture.
- Printed caption.
- Page → First picture: contains.
- Page → Second picture: contains.
- Page → Printed caption: contains.
- Printed caption → Second picture: describes.
Plain text exports one sequence. Markdown can preserve headings and simple tables. CSV supplies rectangular records. Keep the original, observed text, normalized values, extraction method and completion state available behind those exports. Established OCR formats illustrate several ways to retain more. hOCR embeds spatial text in HTML. ALTO XML can record order and reference processing steps for individual elements. PAGE XML represents polygonal regions and can group their identifiers in ordered or unordered sets. hOCR specification · Library of Congress ALTO documentation · PAGE XML example
Text and page structure
Prepare images and preserve coordinates
Rasterization renders a page into pixels at a chosen resolution. Inspect the smallest mark that can change the reading, such as a decimal point. Rotate a sideways page to its intended orientation. Deskew tilted lines so the recognizer can group their letters. Thresholding classifies pixels as foreground or background and can separate ink from uneven paper, but it can also remove faint strokes. Dewarping straightens print on a curved surface by moving different parts of the page different distances. Retain both the source render and the image actually sent to recognition. Tesseract’s input-quality guidance
Returning a highlight to the source
In the two pixel frames, follow region R. The crop starts at (40, 20) on the source. Subtract that origin from R’s upper-left corner at (50, 30) to get (10, 10) within the crop. Doubling the crop places the corner at (20, 20) in the recognition image. The opposite corner moves from (70, 40) to (60, 40). To highlight the result on the source, divide both recognized coordinates by two and add the crop origin. If padding was added after resizing, subtract the padding first.
One region before and after cropping and resizing
ExampleThe crop origin moves while the region doubles in pixel dimensions.
Source render
A 50 by 30 pixel crop encloses R.
- 1. Crop boundary
- 2. Region R
Read coordinates and regions as data
X: 0–110 pixels; Y: 0–80 pixels, increasing down. Equal scale on both axes.
(40, 20); (90, 20); (90, 50); (40, 50); (40, 20)
(50, 30); (70, 30); (70, 40); (50, 40)
Crop origin: (40, 16)
R: (60, 36)
Recognition image
The crop now starts at (0, 0) and measures 100 by 60 pixels.
- 1. Crop boundary
- 2. Region R
Read coordinates and regions as data
X: 0–110 pixels; Y: 0–80 pixels, increasing down. Equal scale on both axes.
(0, 0); (100, 0); (100, 60); (0, 60); (0, 0)
(20, 20); (60, 20); (60, 40); (20, 40)
R: (40, 31)
Use the actual transform for each page. Rotation requires mapping all four corners of a word box. Nonlinear dewarping requires a spatial map. Coordinates expressed as fractions of a frame need that frame’s dimensions. A PDF highlight may require conversion from image pixels to page points, accounting for the visible crop and page rotation. A location can map back accurately even when resizing has changed the pixel samples or cropping has discarded context. PyMuPDF page geometry · Computer Vision
To test whether smoothing helps recognition, compare the transcripts with and without smoothing on the same source images. Character error rate counts the character substitutions, deletions and insertions needed to align a transcript with a reference, divided by the reference length. On 25 images with Tesseract 4.1.0 in its default configuration, the DewarpNet authors reported a rate of 0.249 without added blur and 0.3136 with it. Smoothing made the text harder for that recognizer to read on those images. DewarpNet evaluation notes
Recognize character sequences
Detection locates writing, grouping supplies a word or line image, and recognition assigns it a transcript. A character classifier that receives two touching letters as one cutout must work around an error made before classification. A sequence recognizer can learn from the transcript of a whole line without an annotated cut between each pair of letters. It still needs an appropriate region and enough detail to distinguish the marks.
Figures 1 and 2 in Shi, Bai and Yao’s paper show a cropped word becoming a sequence of columns of visual features. Their convolutional recurrent neural network (CRNN) uses neighboring columns on both sides to interpret each position. It can assign a character or a special blank label at each position. Connectionist Temporal Classification (CTC) trains the recognizer by summing probabilities over paths of labels compatible with the target transcript. A wide character can occupy several successive positions without requiring a person to specify its width. CRNN figures · Original CTC paper
Two runs produce two digits
ExampleRemoving a blank too early would turn two digit runs into one.
Two adjacent positions predict the first digit.
Read the diagram as text
- 1 · 1 · blank · 1. One possible label path across the image.
- 1 · blank · 1. One label per consecutive run.
- Transcript 11.
- 1 · 1 · blank · 1 → 1 · blank · 1: collapse adjacent repetitions.
- 1 · blank · 1 → Transcript 11: remove blank.
- Position labels. Two adjacent positions predict the first digit. Active: 1 · 1 · blank · 1. New: 1 · 1 · blank · 1.
- Consecutive runs. The blank keeps the last digit in its own run. Active: 1 · 1 · blank · 1, 1 · blank · 1. New: 1 · blank · 1.
- Transcript. The two digit runs now contribute one character each. Active: 1 · 1 · blank · 1, 1 · blank · 1, Transcript 11. New: Transcript 11.
To read a possible path, collapse consecutive identical labels and then remove blanks. In the figure, the blank between two runs of 1 allows the transcript to contain two identical digits. A lexicon, a set of permitted strings, can help choose among uncertain readings. A dictionary of ordinary words can also exclude the correct product code. When a result is implausible, inspect the crop for blur, compression damage or unusual print, and check whether the recognizer supports the script and handwriting on the page. Even an exact reading of a code leaves its role to be determined.
Find regions and their roles
Physical segmentation groups marks into regions. Role assignment identifies what each region does, such as introducing a section, presenting a table or qualifying a figure. A mistaken boundary can send the wrong material to the next stage. Group a paragraph with a neighboring table and its sentences become candidates for cell contents. Drop a small footnote region and even a correctly located reference mark has nowhere to point.
Top-down partitioning starts with a page. Ha, Haralick and Phillips’s 1995 recursive X-Y cut projects the rectangles enclosing connected marks onto each axis. A sufficiently large empty interval permits a horizontal or vertical cut, and the procedure repeats inside each resulting rectangle. On the partitioned pages in the paper, look for whitespace extending all the way across the region being divided. A gutter that stops at a spanning heading cannot separate the columns in a single cut through that heading. Recursive X-Y cut
Bottom-up grouping starts with neighboring marks. Mao and Kanungo’s implementation study contrasts Docstrum’s ability to group nonrectangular arrangements with its difficulties when characters are sparse or irregularly spaced. Learned detectors can use visual features to predict region boundaries and roles, but even their reference labels require judgment. In IBM’s DocLayNet Figure 4, inspect the choice between enclosing subfigures together and identifying them separately. An annotation convention settles what the detector should learn. Keeping nontext and unknown regions addressable leaves a later stage somewhere to attach a caption. Segmentation study · DocLayNet Figure 4
Recover reading order and continuity
Reading order specifies the precedence that preserves a document’s meaning. In W3C’s two-column PDF example, follow a line to the gutter. The sentence continues on the next line of the same column. Sorting all words by height and then horizontal position would instead cross into the other column. The words stay correct while the sentence changes. Tagged PDFs can supply a reading order directly. Following the tags across the same boundary tests whether conversion preserved the author’s sequence. W3C reading-order example
A magazine page may contain two independent articles. Each opening precedes its own continuation, but the reader can choose either article first. A partial order records just the precedence relations that must hold. A plain-text export selects one traversal. Keep an explicit link from a note to the passage it qualifies so the association survives whichever traversal the exporter chooses. Reading-order evaluation
Precedence within independent articles
ExampleA note can qualify a passage without determining which article comes first.
Read the diagram as text
- Article A opening.
- Article A continuation.
- Article B opening.
- Article B continuation.
- Note.
- Article A opening → Article A continuation: precedes.
- Article B opening → Article B continuation: precedes.
- Note → Article A continuation: qualifies.
Within a line, text direction poses a different problem. Arabic or Hebrew can include left-to-right digits and Latin text. Unicode distinguishes a string’s logical character order from the order of glyphs on display. Across pages, a logical paragraph can continue past a footer and a repeated header. A readable export can omit that furniture from the paragraph and join a word split by a line-ending hyphen. Preserve links to both original fragments when making the join. Unicode bidirectional algorithm · Tagged PDF guide
Traceability
Locate supporting source material
A source location addresses material in a particular document version. An evidence span identifies the text or regions that support a reading. Consider a duration exported from a table. One region contains the digits, while a header above several columns supplies the unit. For verification, open both regions on the archived page and check that the header governs this cell. A link to today’s file at the same URL or a search for the first occurrence of the digits can resolve to different evidence.
| Address | Information needed to resolve it |
|---|---|
| Physical page index | Source version and numbering convention. A display label can repeat, use Roman numerals or differ from the number printed on the page. |
| Box or polygon | Page or image frame, units, origin and any crop or rotation. An oriented quadrilateral can follow tilted text more precisely than an axis-aligned box. |
| Text span | Exact text representation, counting unit and boundary convention. W3C’s position selector includes the start and excludes the end. |
| OCR word ID or model token position | The conversion or tokenized input in which the identifier resolves, plus a mapping back to source text. Tokenization |
Two locations support one duration
ExampleInspecting the number resolves only one branch of the field’s evidence.
Read the diagram as text
- Duration field.
- Numeric text span.
- Unit text span.
- Data cell region.
- Governing header region.
- Page in archived source. Identified by source version and physical index.
- Duration field → Numeric text span: magnitude read from.
- Duration field → Unit text span: unit read from.
- Numeric text span → Data cell region: located in.
- Unit text span → Governing header region: located in.
- Data cell region → Page in archived source: region of.
- Governing header region → Page in archived source: region of.
Expanding fi into fi makes two output characters correspond to one source character. Joining a word across lines can give one output word two source regions. ICU’s edit map records corresponding ranges even when the interior of a replacement has no exact character-level alignment. For reordered blocks, retain the original segments in the new traversal order. Google Document AI’s document resource illustrates a useful separation between text anchors into a string and page anchors into polygons. A model can propose either kind of anchor. Inspecting the addressed material establishes what it supports. ICU edit maps · Document AI resource
Document architectures
Combine text, layout and images
A converter can obtain a word’s spelling from one source and its cell assignment from another. In Docling’s original native-PDF pipeline, one branch extracts encoded words with coordinates. Another renders the page, identifies table regions and predicts cells, spans and header roles using TableFormer. At the point where the branches meet, post-processing assigns words to predicted cells by position. Document assembly then incorporates the populated table alongside other elements in reading order. If a price lands one column over, inspect the predicted boundary and the word assignment before rerunning recognition. Docling technical report
In Cedric Clyburn’s recorded notebook, Clyburn of Red Hat converts Docling’s own research paper, inspects its pages and renders extracted tables separately. To check a table, choose a value on the source page, identify its row and column heading, then perform the same lookup in the extracted table. Finding the digits somewhere in the Markdown export would test a weaker claim.
Assigning native words to predicted cells
A correct spelling can enter the wrong cell where the text and image branches meet.
Read the diagram as text
- PDF.
- Positioned words.
- Rendered page.
- Page regions and roles.
- Table structure recognition. TableFormer predicts cells, spans and roles.
- Assign text to cells.
- Document assembly.
- PDF → Positioned words: extract text and coordinates.
- PDF → Rendered page: render page pixels.
- Rendered page → Page regions and roles: predict regions.
- Page regions and roles → Table structure recognition: table boundaries.
- Rendered page → Table structure recognition: pixels inside boundaries.
- Table structure recognition → Assign text to cells: predicted cells.
- Positioned words → Assign text to cells: spellings and positions.
- Assign text to cells → Document assembly: populated table.
- Page regions and roles → Document assembly: regions and roles.
Learning from words and positions
Layout-aware models learn how position changes the interpretation of words. During pretraining, when a model learns representations for later tasks, the original LayoutLM used recognized words and scaled page coordinates. Its architecture added separately obtained image features downstream. Its form experiment labeled spans. Correctly labeling a name and an address still leaves their association to be checked. LayoutLMv2 brought imagery into pretraining while continuing to consume OCR words. Its tasks included aligning text lines to image regions and detecting mismatched text and images. In a comparison of base models trained for one pass on the same corpus, adding visual information and the additional objectives improved document-question-answering validation scores. LayoutLM Figure 2 · LayoutLMv2 experiments · Pretraining and Midtraining
Generate structure from page images
Donut takes a page image and generates a sequence encoding content and structure. Figure 3 pairs a receipt with emitted item records. This route removes the external OCR dependency. Reading and field assignment still occur within the model. In one published failure, Donut returned a different visible number for the publication number. Its attention visualization suggested where the decoder looked, without certifying field coordinates. Larger images helped with tiny text at additional computational cost. Donut figures and experiments
A vision-language model processes images and language together. IBM’s Granite-Docling accepts whole-page and regional instructions and can generate DocTags, structural markers convertible into a DoclingDocument. Working on a region can give a small field more pixels, but a tightly cropped value may lose its governing header. Google’s Gemini 3 PDF interface supplies native PDF text alongside visual page information. Compare routes on the actual document types by inspecting what each sees, which intermediate readings can be checked, and whether its returned locations resolve to the necessary context. Granite-Docling model card · Gemini document inputs · Multimodal Models and Applications
Tables and records
Recover table meaning
Table detection finds the area. Structure recognition identifies rows, columns, empty cells and cells spanning several positions. Rules drawn on the page can suggest boundaries, while a borderless table can express the same grid through alignment. Transcription supplies cell contents. Functional analysis identifies which cells act as headers for others. Microsoft’s Table Transformer exposes the separation directly. Models detect tables and recover their structure from images, but the exporter needs separately supplied text from native extraction or OCR. PubTables-1M also addressed a problem in training references, where annotations sometimes split one spanning header into several cells. Table Transformer · PubTables-1M
In the Journal Article Tag Suite (JATS) example, columns inherit their unit from a spanning header. To interpret 151/597, follow the cell’s footnote reference. JATS table and footer · W3C multi-level headers
Inherited units and a referenced qualification
Header spans apply across columns. Note references apply to individual cells.
Read the diagram as text
- Header giving days as the unit.
- First subordinate column.
- Second subordinate column.
- 151/597. A cell elsewhere in the table.
- Data unavailable for one trial. A note in the table footer.
- Header giving days as the unit → First subordinate column: supplies unit.
- Header giving days as the unit → Second subordinate column: supplies unit.
- 151/597 → Data unavailable for one trial: references note.
A repeated header after a page break can introduce more rows of the same table. Microsoft’s template guidance labels rows across expected page breaks as one table and calls for training examples covering those variations. Check unfamiliar continuations against the pages. Preserve empty cells that hold a column position, cell regions, header spans and note links. A simple grid can go directly to Markdown. A complex table can use HTML spans as in Reducto’s example. For CSV, expand inherited headings into column names and carry qualifications separately so a reader of the export can recover them. Template model guidance
Extract fields with context
Field extraction assigns evidence to a named role within a scope. An Address label can appear once under supplier and again under customer. Position, section membership and the label together identify the requested address. Amazon Textract distinguishes CHILD links, such as a line containing words, from VALUE links connecting a key with its value. A selection mark in a rating grid takes its meaning from both the row question and the answer column. Detecting an unchecked box supports a negative selection. Failing to detect the box leaves its state unknown. Textract relationships · Selection elements
For repeated records, extract related fields together or retain enough evidence to join them. Ben Kus describes Box dividing large extraction requests while keeping contract parties with their addresses. Independently sorted lists could contain every correct name and every correct address, yet pairing them by index could assign a party someone else’s address. A complete line item likewise requires the description, quantity and price to refer to the same product.
What the application added
| Operation | Example | Retain |
|---|---|---|
| Normalization | Google’s French receipt date 11/12/2022 becomes 2022-12-11. | Literal span, location and date convention. Decimal separators and units also need an explicit interpretation. Leave alternatives open when the convention is unknown. |
| Correction | A reviewer changes a misread character in an identifier. | Original reading, visible mark, amendment and reviewer. |
| Calculation | Subtract a deposit from the invoice total to calculate the balance. | Both amounts, their units and source regions, and the calculation used. |
| Enrichment | An external lookup supplies an address missing from a receipt. | Printed mention and separate origin of the added address. |
Check records against their sources
A schema can require a numeric amount for each line item. Peppol rule BR-CO-13 checks a relationship among those amounts. To obtain the invoice total excluding value-added tax, add the net amounts for all lines, subtract document-level allowances and add document-level charges. Comparing the result with the total read independently from the page can expose a missing line. If the extractor instead fills the total field from its own shortened list, the two figures will agree. Even an independently read total cannot expose a swap of two line net amounts because the sum stays the same. Check each amount against its source row to verify which item it belongs to. Peppol BR-CO-13 · What a schema promises · Structure is not domain correctness
| Check | Evidence inspected | What can pass unnoticed |
|---|---|---|
| Structure | Record fields and types | An amount copied from another row |
| Reconciliation | Amounts supplied to the arithmetic rule | Exchanging two amounts |
| Association | Source row, label and value together | Another row omitted entirely |
| Coverage | Expected pages and rows against accounted-for pages and rows | A misread digit on an accounted-for row |
Keep the reason when a field cannot be accepted. An inspected blank, illegible ink, conflicting entries, an inapplicable field and a page awaiting processing call for different actions. The Text Encoding Initiative’s transcription guidelines distinguish a gap, a reading retained with uncertainty and words supplied by an editor. Preserve that distinction when a reviewer proposes a value the original cannot settle. Agreement between models that read the same faulty transcript adds no new observation of the disputed mark. TEI transcription guidelines
Quality and recovery
Measure the required fidelity
To evaluate a transcript, align it with a reviewed reading. A missing character can then count as one deletion without counting every subsequent character as wrong. Character error rate (CER) and word error rate (WER) use the same calculation at different units. Specify the character or word boundaries and apply the same normalization to both sides. Stripping punctuation before scoring prices would forgive the loss of a decimal point. JiWER alignment and transformations
| Property | Reference and comparison |
|---|---|
| Reading order | Match predicted regions to reviewed regions by overlap to account for splits and merges. Compare precedence and unordered relations between corresponding regions. |
| Table structure | Grid Table Similarity (GriTS) compares cell arrangement, content and location separately while respecting two-dimensional order. |
| Field values | Precision divides matching predictions by all predictions. Recall divides recovered required pairs by all reference pairs. Under exact field-value matching, a wrong value gives one unmatched prediction and one missed reference. |
| Source and item assignment | Check the field type and localized evidence, then check the item to which the field belongs. DocILE measures localization and grouping separately from transcription and normalization. |
| Completeness | Compare delivered pages, regions and required items with the input inventory. Keep skipped and failed pages in the coverage denominator. |
Review examples across the layouts, scripts and capture conditions expected in use. Split entire documents between training and evaluation. DocLayNet’s layout detector scored higher when other pages from the same documents appeared in training. Whole-document separation still permits a familiar template to appear on both sides, so report performance on unseen layouts separately. For an operational measure, count the units the consumer can use. In the CHICA clinic pilot, barcodes identified patient records and templates located form entries. Digit recognition accuracy was 98.6%, while 125 of 221 scanned forms could proceed without verification. The first denominator counts digits and the second counts forms. CHICA study · Evals and Benchmarks
Choose what to accept
Extraction confidence is a score for a specified prediction. Reading 40.00, associating it with a total label and getting every critical invoice field right are three different events. A column shift can spoil several associations together while leaving the words clear. Before interpreting an engine score or a model’s self-assessment as a probability, check its calibration against the event of interest on comparable labeled cases. Among cases assigned a calibrated probability near 0.9, roughly nine in ten should be correct. Machine Learning Fundamentals
An acceptance threshold divides cases between automatic release and referral. Coverage is the fraction of all eligible cases released. With correctness scored as pass or fail, selective risk is the fraction of released cases that are wrong. Risk is undefined when none are released. A stricter policy can reduce risk by referring more work. Compare policies at a common risk or coverage, taking into account both the cost of a wrong release and the available review capacity. Selective classification · Evals and Benchmarks
Basware held its extractor fixed and trained a separate confidence estimator for joint correctness of five fields: invoice number, issue date, subtotal, tax and total. The authors calibrated the estimator separately and tested on 24,000 held-out single-page, machine-readable invoices with known templates. At the production system’s undisclosed target precision, estimated automatic coverage increased from 65.6% to 73.2%. The change concerned which invoices qualified for release on that test set. Unknown templates and multipage invoices fell outside the comparison. Basware study
Make the referral reason specific. Amazon’s documented Textract review integration distinguished uncertainty about the words from uncertainty about the key-value pair. A missing critical key could trigger a search for the missing entry, while sampling regardless of confidence allowed review of apparent successes. That integration is closed to new customers. If the page itself supports competing readings, pass the candidates and their context to the reviewer. Adjusting a threshold cannot supply evidence missing from the source. Review activation conditions
Recover incomplete and disputed results
Before rerunning a job, distinguish incomplete delivery from incomplete processing. In Amazon Textract’s asynchronous API, NextToken requests another batch of available result blocks. A partial-success status and warnings about particular source pages describe a processing problem. Fetch the remaining batches before identifying what is missing. A warned page can still require investigation after the last batch arrives. GetDocumentAnalysis contract
For a disputed price, locate the OCR word selected by the field extractor. If it points to an adjacent number, revisit grouping or field assignment. If it points to the price but omits the decimal point, compare that location in the source render and the image sent to OCR. A point visible only in the render suggests changing preparation. A point clearly visible in the prepared image directs attention to recognition. If neither image contains enough detail, obtain a better capture or retain an unresolved reading. Each retry now changes an operation or supplies evidence relevant to the discrepancy. OCRmyPDF inspection artifacts
Review and correction
A reviewer needs the source version, candidate readings, referral reason and enough surrounding context to resolve the question. Figure 2 of the Vanderbilt assessment-form study places the scan beside an editable form. Selecting a field highlights the corresponding location in both views, so a reviewer can see which question a mark answers. Review time also depends on the work requested. Once transcription of handwritten comments was dropped, reported mean annotation time fell from 188 to 156 seconds, with observations above 401.5 seconds excluded from the timing analysis. Compare review policies at a fixed task scope. Vanderbilt Figure 2 and study
Record the reviewer’s decision and responsibility alongside the original reading. A reviewer may settle which item owns a value or determine that the page leaves it ambiguous. Rerun checks affected by a correction. Changing a line amount affects reconciliation, while changing a spanning header can affect every record exported under it. Keep completion and review state at page, region and field level so a local correction cannot silently mark a whole document complete. Design Engineering and AI Interfaces
Downstream use
Deliver context with the records
Export 151/597 with its row identity, governing headings and unavailable-data note. Include locations for that evidence in the processed source version, any interpretation or normalization applied, and the scope of verification. A partial-output contract can release a verified item while leaving a document-wide total pending. The converse is possible too. If every count is verified exactly once, their sum can be released even while the assignment of individual counts to recipients awaits review.
A chunk is a bounded unit supplied for retrieval. An index is a derived structure for finding such units. Docling can form chunks from the document hierarchy and repeat table headers so rows can be interpreted outside the full table. Inspect the actual chunk for qualifications and source references. Reducto’s description for lookup and HTML for reading can point to the same table. Liu’s return to the source adds a way to inspect a consequential field in context. Finding the unit and composing an answer from it are downstream responsibilities. Docling chunking · Choosing Searchable Units · Retrieval-Augmented Generation
Carry permission metadata to the consumer that enforces access. Retain lineage links from records, chunks and index entries to their source elements and extraction version. When a reviewer corrects a shared header, those links identify which independent-looking rows need to be regenerated or withdrawn. Maintain fitness through change · Privacy and Data Governance
Reference
Frequently asked questions
Does a searchable PDF still need OCR?
Check the target region. A searchable heading may sit above an image-only table, and a PDF can display correct glyphs while copying incorrect characters. Retain faithful encoded text. Recognize uncovered image text and investigate conflicts against the rendered page.
Can OCR-free extraction choose the wrong number?
Yes. An OCR-free model reads the page image directly without a separate OCR transcript. In a published example, Donut filled a publication-number field with a different number visible on the same page. To verify the field, locate the publication-number label on the page and compare its value with the extracted number.
Why can a table value need several source locations?
The cell establishes the literal value. A spanning header can supply the unit, and a separate note can qualify the reading. Retain the regions and their links so a reviewer can check which header or note applies.
Can a partially processed document supply a usable record?
Yes, when the consumer accepts that unit and all evidence required for it has been checked, including item membership. Fetch outstanding result batches and identify failed pages separately. Release a verified record with its scope explicit and withhold results that depend on missing material.
Open questions
How well do document acceptance policies transfer to unfamiliar templates? A layout change can break several field associations together while preserving clear words. Measuring both automatic coverage and errors among released documents on unseen layouts and multipage inputs would show whether a threshold still supports the intended use.
When is a table on the next page a continuation? Repeated headers support that interpretation but can also introduce a distinct table. Reviewed cross-page links on unfamiliar layouts, with ambiguous joins marked separately, would make it possible to measure how often an export combines unrelated records or drops continuing rows.
Which referrals reduce review effort at a fixed level of residual error? Policies can appear faster by narrowing the work or referring fewer cases. Comparing time, reviewer disagreement and errors left in released records on the same forms and fields would distinguish genuine savings.

























