AI Engineer World's Fair 2026

Structuring the Unstructured: Advanced Document Parsing for AI Workflows

Loading slides…

Read the talk

Structuring the Unstructured: Advanced Document Parsing for AI Workflows

Turn PDFs into usable AI context by preserving layout, extracting tables and pictures, enriching diagrams, and retrieving through document structure before exposing conversion tools to agents.

From a talk by Cedric Clyburn

Before you start: Basic Python and familiarity with retrieval-augmented generation will help you follow the notebook examples.

What context disappears inside a PDF?

How do you give an AI application useful context when the information lives in PDFs, presentations, contracts, scanned pages, diagrams and tables? A harness can manage an LLM’s context, but it cannot recover information that an ingestion pipeline has already discarded. Cedric Clyburn, introducing himself as an open source engineer at Red Hat, starts with this practical gap between having documents and being able to use them.

Enterprise documents are often scattered across dozens of systems. Processing them may also mean accepting a proprietary service or sending private data to another organization’s servers. The task is therefore more specific than extracting text: preserve the relationships in graphs, tables and images while converting them into representations such as Markdown or JSON. Docling, introduced here as an open source Linux Foundation project, supplies local tools for extraction, parsing and chunking.

Completed overview slide with a PDF icon, proprietary service names, chart and text examples, and Docling imagery beneath extraction, parsing, and chunking text.Open full source frame
Unstructured documents, proprietary parsing services, and the extraction tasks ahead.
0:000:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

An extraction error can become someone else’s evidence

Technical documentation, meeting minutes and invoices can become retrieval context or training data for a specialized model. In either case, document processing affects what the model can answer correctly. Better inference hardware or a different model cannot reliably reconstruct a table relationship that disappeared before the data reached it.

Clyburn cites a report of twenty scientific papers containing a nonsensical term, attributing its origin to words incorrectly joined across columns in an old scanned article. The matching reporting on “vegetative electron microscopy” treated that OCR explanation as a hypothesis; a follow-up investigation also considered Farsi typo or translation explanations. The precise origin remains uncertain, but the example illustrates the propagation problem: questionable text enters papers, and those papers become material that others cite.

A layout-aware view makes the proposed failure easy to understand. In Clyburn’s local Docling view, the two words occupy separated regions and should not be combined. Spatial structure is part of the evidence, not merely a formatting preference. Preserving it gives downstream systems a better basis for extraction and validation.

2:042:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:04 · section reference included

Cheap text extraction loses relationships

The next example puts a PDF containing text sections, a table, an image and captions beside a simple parser’s Markdown output. The parser is fast and inexpensive to run on a CPU, but text is truncated or merged, and the table becomes a linear stream. A question about a particular table entry now requires reconstructing relationships that the output no longer expresses. Unwanted page headers remain, while the image’s content is missing.

PDF page beside extracted text annotated with undesired headers, misunderstood tables, missing image content, line-wrap errors, and broken multicolumn order.Open full source frame
A simple PDF parser loses tables, images, and reading order.

A frontier model can produce more convincing Markdown, but its economics and output stability become consequential across a large collection. Clyburn uses an illustrative price of $30 per million output tokens to explain how processing costs accumulate across many PDFs. He also raises a hypothetical model change from version 5.1 to 5.2: even when the output looks accurate, a replacement model can change how it expresses structure. Nondeterminism and hallucinations add another source of variability.

3:544:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:54 · section reference included

Recover structure locally before choosing an output

Docling provides a CLI and Python library that combine OCR with specialized vision models to recover document structure. Its output is a Pydantic document object, which can then be exported as Markdown or JSON. In the table example, Clyburn renders the recovered table as HTML; because the table’s structure survives extraction, an application can work with a selected column instead of interpreting flattened text.

Clyburn’s Red Hat team uses Docling with thousands of product-documentation PDFs. Installation begins with pip install docling, and conversion can cover documents and websites while retaining layout information. Running locally also accommodates private data and air-gapped environments without requiring a hosted conversion service. The useful distinction among the approaches is what each preserves and what the application must manage afterward.

ApproachUseful propertyMain concern in the walkthrough
Simple PDF parserFast, cheap CPU extractionLost reading order and table relationships
Frontier modelRicher interpretationToken cost and variable output
Docling pipelineLocal, structured conversionRecover layout before downstream use

The middle ground is to make document structure an explicit intermediate representation, rather than asking every downstream model call to rediscover it.

For scale, Clyburn points to Leandro von Werra’s work preparing Common Crawl PDFs for FinePDFs. Preprocessing and structural extraction help remove unwanted material before training-data export. Clyburn reports roughly 50× lower processing cost in the FinePDFs example, emphasizing that the Docling path runs on CPUs without requiring a GPU. The published cost breakdown compares different routed workloads: customized CPU Docling handles extractable PDFs, while GPU RolmOCR handles scanned or truncated PDFs. That makes the figure a pipeline-specific cost comparison, not a same-document benchmark.

5:406:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:40 · section reference included

Descriptions and fields serve different needs

Once document content is accessible, two further operations become useful:

  • Image enrichment: A vision-language model describes and annotates an image, adding context that a RAG application or human reader can use. This can help explain an organizational diagram even after the subject-matter expert who understood it has left. The model can be local or proprietary.
  • Field extraction: An invoice workflow asks for the bill number, total price and sender name, returning those fields in a Pydantic structure. It does not need unrelated headings or every line from the document.

The first operation expands the available description; the second narrows the output to a requested schema. Clyburn then moves into workshop notebooks to demonstrate the underlying conversion steps.

8:298:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:29 · section reference included

Convert a PDF, then inspect the document object

The notebook starts with Docling’s own research-paper PDF, opened from an online source. It contains titles, subtitles, pictures, captions and tabular content—the elements a plain text extraction can lose. The first operation is straightforward: import DocumentConverter, convert the source, and export the resulting document to Markdown.

For a local copy saved as paper.pdf, the same conversion pattern is:

python

from pathlib import Path
from docling.document_converter import DocumentConverter

source = Path("paper.pdf")
result = DocumentConverter().convert(source)
document = result.document

markdown = document.export_to_markdown()
Path("paper.md").write_text(markdown, encoding="utf-8")
print(markdown)

Clyburn converts an eight-page PDF to Markdown in the notebook. The displayed result includes the paper’s title, authors, abstract and section headings; he does not give a numerical runtime.

VS Code notebook showing a Markdown export cell and output containing the paper title, authors, abstract, repository link, and introduction heading.Open full source frame
The converted Docling paper appears as Markdown in the notebook.

Markdown is only one view of the result. The Pydantic object also exposes pages, tables and the relationship between content and its source page. Keeping that object lets the application inspect structure before exporting it as Markdown, HTML or a dictionary:

python

print(f"Pages: {len(document.pages)}")
print(f"Tables: {len(document.tables)}")

html = document.export_to_html()
structured_data = document.export_to_dict()
Path("paper.html").write_text(html, encoding="utf-8")

Keep the structured document as the working representation; export text when a consumer needs text.

9:459:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:45 · section reference included

Inspect tables, pictures and their positions

Table extraction makes the value of that representation concrete. Each recovered table can become a data frame, which Jupyter can display directly and an application can query without first rebuilding rows and columns from prose. Clyburn exports eight tables from the demonstrated source PDF into data frames.

python

from IPython.display import display

for index, table in enumerate(document.tables, start=1):
    frame = table.export_to_dataframe(doc=document)
    print(f"Table {index}")
    display(frame)

This preserves a useful distinction: a table can be available to retrieval while still remaining tabular data.

Pictures require an additional PDF-pipeline setup. Clyburn configures image scaling before conversion, then inspects the extracted picture content. The notebook brings together the source image, its caption and embedded text elements. That gives a retrieval application more than an isolated image file: it retains the surrounding information needed to ask what the picture depicts.

The layout visualizer then exposes bounding boxes for section headers, text, subtitles and pictures. These coordinates show where extracted elements came from and provide a way to inspect whether the parser has separated the page correctly. Clyburn proposes using this spatial information to support removal of customer personally identifiable information before ingestion. Coordinates support such a workflow, but visualizing them does not itself detect sensitive information or complete redaction.

11:2011:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:20 · section reference included

Add a description beyond the original caption

Extracting a caption and describing an image are different operations. The next pipeline uses a locally running Granite vision model to request a detailed description of the picture. Ollama exposes the local model through an OpenAI-compatible completions interface; a third-party model is another option.

The example returns to the Docling pipeline diagram extracted earlier. Previously, the notebook had the original caption. After enrichment, it also has a generated description of what the diagram shows. This adds retrievable context for questions about visual content while preserving the distinction between text present in the source and interpretation supplied by a model.

13:1113:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:11 · section reference included

Use the document outline as the retrieval index

With the document parsed and enriched, retrieval does not have to begin with fixed chunks. In the demonstrated chunkless RAG pattern, an LLM reads a document outline, chooses a relevant section, and retrieves that section’s full text from the Docling document. It can repeat the operation in an agentic loop. This path uses no chunker, embedding model or vector database.

Retrieval approachIndexSelection mechanism
Vector retrievalEmbedded chunksSemantic similarity
Outline retrievalMarkdown headings and section summariesLLM chooses sections

Instead of searching thousands of stored vectors, the agent navigates a compact description of the document’s structure. A getting-started section, for example, provides the route to text explaining that Docling can be installed from PyPI. The outline points to the evidence; the full section supplies it.

The first query is: “What are the main AI models used in Docling?” The retrieval procedure is:

  1. Read the available section outline.
  2. Select the section likely to discuss the models.
  3. Fetch its full text and assess whether it answers the question.
  4. Answer from that material, or retrieve another section if evidence is missing.

For the Docling-models question, Clyburn allows about five iterations over twenty available sections, and the demonstration returns an answer in one iteration. The significant behavior is that the answer comes from navigating document structure rather than consulting a vector database.

The next example increases the search space. Clyburn’s parsed IBM 2025 annual report contains 418 sections. The question asks about Red Hat’s revenue growth in 2025 and its contribution to the overall software segment. Here the agent must assess whether a selected section is enough and fetch more information when it is not. The walkthrough illustrates repeated relevance checks without supplying a final financial answer. Moving from one report to large collections then raises a separate question: how should conversion itself be deployed?

13:5714:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:57 · section reference included

Expose conversion as a service

Docling Serve exposes document processing through a REST API. The deployment can be a microservice, a container or a Kubernetes workload. In the walkthrough, installation starts with pip install docling-serve, followed by launching the server from its CLI on a selected port.

Clients submit documents with options such as whether to use OCR, which processing backend to select, and whether to annotate images. This separates application code from the conversion process and gives multiple consumers a shared processing endpoint. Current Docling Serve documentation uses a stable v1 API; its request syntax should be taken from that documentation rather than inferred from the recorded setup.

16:4316:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:43 · section reference included

Give an agent document-processing tools

The REST service addresses centralized processing; Docling MCP addresses how an agent discovers and invokes document operations. Clyburn presents the service as a route to larger collections, without a throughput measurement. Through the Model Context Protocol, a developer assistant can access document-processing capabilities without the user manually assembling every conversion command.

The local example connects a Qwen model running on a Mac to VS Code. The MCP tools cover conversion, generation and manipulation, including operations on a selected part of a PDF. Clyburn checks the local MLX model server, identifies the running model as Qwen 3.6, and checks the Docling MCP server using uvx. The architecture shown connects Continue in VS Code, the model server and Docling’s document tools.

Notebook architecture diagram linking Continue in VS Code to a docling-mcp server, Qwen via MLX, and document processing.Open full source frame
Continue connects a local model to Docling document processing through MCP.

In the demonstrated client setup, adding the MCP server means editing config.yaml; that filename is specific to the client configuration shown. Current Docling MCP defaults to remote conversion, so reproducing a fully local arrangement requires the local installation extra and explicit local configuration. Starting it with uvx alone does not establish that documents remain local.

Once connected, the agent can turn a user request into document operations: convert a document and summarize it, or create a document with action items, pull a list from another PDF, and export the assembled result as Markdown. Continue, Claude Code, Codex and Cursor are among the client options mentioned. The interface changes from a notebook cell or REST request to an agent tool call, while the underlying parsing capabilities remain available.

17:1817:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:18 · section reference included

One structured document, multiple downstream paths

The architecture ends where the first conversion began: OCR and layout analysis assemble a Pydantic Docling document, and downstream consumers choose how to use it. PDF-to-Markdown or JSON conversion can run entirely locally without a GPU. The same structured result can support format exports, dataset creation, or retrieval through an outline.

Chunkless retrieval is one option, not a requirement imposed by the parser. Docling also offers a hybrid chunker for workflows that need chunks, alongside integrations with RAG frameworks, agentic systems and harnesses. Preserving structure first keeps those choices open: the application can select a retrieval or agent architecture without starting over from a flattened, incomplete copy of the source.

19:2619:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:26 · section reference included

Resources

From the talk

  • Local document conversion library and CLI with layout analysis, OCR and structured exports.

  • PDF-derived training corpus with documentation of CPU extraction, GPU OCR, filtering and deduplication.

  • Run Docling as a REST service using Python or published container images.

  • Expose document conversion and generation tools to MCP clients.

Transcript download
Full transcript

Loading transcript…