From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto

[music]

Hi. Can everyone hear me? Sweet. We have

a short 20 minutes here and so I wanted

to jump in and get right to it. Uh my

name is De. I'm the co-founder and CEO

of Redducto. Uh and today we wanted to

talk about one of the I think really

practical but maybe less sexy parts of

building agents that actually work in

the real world uh which is data. Um I'm

sure you've seen plenty of talks about

data today. We primarily have focused on

building infrastructure for anybody

working with some of the hardest sources

of data which is unstructured images,

PDFs, spreadsheets, everything that

humans are used to using day-to-day. Uh,

have any of you used Reduct already or

trial it? Cool. Um, I guess helpful

context maybe to start is we're an

agentic document processing platform.

Um, so we help a lot of the world's

leading AI teams build uh both AI

applications and also workflows

depending on what they're trying to do.

Uh, that includes a lot of the AI

natives that you've probably seen today.

It includes the Harveys, the Lores, the

Rogos of the world. Uh, but also some of

the largest enterprises in the world.

And I think that's important context for

what we're going to talk about. Um, we

work with the largest tech companies,

global financial institutions, insurance

orgs, people that have decades worth of

historical data that historically has

been really, really hard to actually use

outside of a demo context. And across

all these companies at this point, we've

processed many billions of documents for

our customers. I've gotten kind of lazy

with that plus symbol at the end, uh,

but the number keeps changing and so

we'll let it sit there. Uh but the main

thing that we've learned across those

and what I wanted to focus on today is

not actually a reduct product itself.

It's the intricacies of what we've

learned that hopefully you can actually

take home and implement in the work that

you're doing as well. Um and I think a

lot of the work that we've done and a

lot of the learnings we've had are in

this sort of broader thematic context of

the scope of AI applications has changed

a lot recently. Uh from just like

information synthesis products to actual

agents doing work. And with that we

found that there are a few things that

are worth talking through. One is

framing the problem itself. Uh the

bottleneck that people face. Uh why PDFs

in particular are hard even though

you've probably seen two dozen different

PDF processing launches on Twitter. Uh

where we find strengths and weaknesses

with different tools. Uh we think

there's a right place in time for

traditional CV versus BLMs. Um and then

more interestingly I think the latter

half of this talk will actually be

focused on the next frontier. uh what

we've seen be possible as a result of

having agents in the loop uh things that

we found as a result of being able to

use harnesses for different types of

tasks uh and our learnings around things

like evaluations uh as you go from rag

to building agent products but I'll

start with the first thing which is

something that I assume has been harped

on a lot if you went to AI engineer a

few years ago the word that you would

have heard in every single talk would

have been rag um and everybody was

building some form of rag application

for a while all applications ations were

really some form of information

synthesis, right? You would pull in

information from some context, whether

that's a perfect prompt or a file that a

user uploaded and you would have

something like a search product. Um,

you'd have enterprise search, you'd have

a chatbot that would do simple question

answer on top of the content and that

was it. But today, the buzzword that

you've probably heard a million times is

agents. Uh, and for those of you that

are engineers, you've probably had cloud

code or similar tool do a lot of your

endto-end work for many, many tasks. And

that same sort of shift is starting to

happen for all sorts of white collar

work. Um, whether you're in finance or

insurance or healthcare, people are

starting to make autonomous decisions.

They're trying to create end-to-end work

products, not just answer questions from

a PDF, but generate and modify PDFs as

well. And that's a very different

framing. the tools that you need, the

problems that you face change a lot

versus just trying to build a retrieval

uh platform.

And the common core for all of this is

it actually ends up being an even more

importance problem to solve for when you

start having these multi-step pipelines.

If you're just doing question answer,

there is obviously a risk that comes

with just answering the question. But

when agents are making multiple

decisions that compound over the course

of a source of files, when they're

pulling in multiple sources of data, the

risk of bad inputs becomes really,

really pervasive in your pipeline. And

that's what we focus on because at the

end of the day, a lot of the value of

language model tools in the real world

only applies in the context to which you

apply that intelligence. And for a lot

of enterprises, data is unstructured, it

is scattered, and it is multimodal. uh

it's not this like cleanly organized

repository. You have random teams and

different organizations that go through

and put things in Google Drive and Box

and wherever else you might have things.

Um you're going to have data for formats

that are unstructured by default. You

don't necessarily know what is in your

corpus of information. And you're going

to have all sorts of downstream problems

that come with that. Um some of the

problems are going to be parsing

extraction accuracy and that definitely

matters. But it's also going to be about

problems like are you retrieving the

right context? It's going to be about

how do you actually interact with that

context and apply modifications to it.

And the thing that you've probably heard

harp on and others in the space is PDFs

are surprisingly still a very hard

problem. Um I don't know if any of you

follow Serge which is a data lab that

works with a lot of the foundation model

companies. uh they have this really

great benchmark called GDP PDF where

even Fable uh like current frontier of

model intelligence is at about 30% on

their benchmark um and it's entirely

predicated on this idea of can we have

models go through and actually make

determinations off of contents that

would be in documents like PDFs uh and

the reason why they're hard I'll come

back to that benchmark in a second is

fundamentally PDFs as a file format are

both very old but were designed in a

very different context I've genuinely

met people that have worked on PDF

processing longer than I've been alive.

Uh I've met the people that worked on

printer drivers for printers to print

PDFs in the early 1990s. And that's what

it was like. You wanted to be able to

represent what was faithfully on the

documents when you originally created it

and have it be printable at the end of

it. A lot of the considerations today

are not around that. Um ultimately what

we want is something like a markdown

representation, something that agents

will effectively reason on. And in the

real world, humans encode so much

context visually. Like the average

financial analyst that is a new grad in

their IB role uh is not going through

and thinking in terms of will agents

consume this deck. They're creating

these really creative slides. I'm sure

you've seen the softbank slides with uh

goose laying eggs. Those details matter,

right? Like a lot of the data that

you're going to reason on is going to be

tabular structures that maybe don't have

clean grid lines and separate out the

merge cells. You're going to have things

like line charts and graphs. You're

going to have messy handwriting that

even I as a human would often struggle

to read. And that's the sort of problem

that you need to solve for if you're

dealing with that long tail.

And we started the company in 2023

because we felt that there was a sort of

step change in what was possible here.

uh for a while as people would think

about any sort of PDF processing

problem, it used to be some modified

version of an NLP pipeline. You do a

simple OCR pass and then you try to

post-process that text. And that worked

when you would have really consistent

layouts, right? Like if you knew that a

W2 is always going to look like a W2,

that's a constrained problem space that

you can template your way around. But

VLMs are interesting because they are

fundamentally horizontal in nature. um

for the first time you can have this

premise of read the document the way

that a human would have. You can have

this premise of we want to address the

longtail. Uh and so we found that they

were incredible for all sorts of things

like handwritten text in a way that

traditional OCR just never was. But the

flip side is that we don't think that

they're a one-sizefits-all solution. And

if you're solving this problem at scale,

if you're a company dealing with

hundreds of millions of documents, there

are all sorts of secondary

considerations like determinism. you

care a lot about the efficiency of your

processing. Uh, and so we find that

there are some things where traditional

CV is actually still really really

strong. And this has been really

underappreciated as autonomous vehicle

research has gotten better. Techniques

like object detection are more

sophisticated than they were a decade

ago. And so we find that subund million

parameter models are really effective

for things like detecting the layout of

a document. Um, you can go really really

far without even needing a large

foundation model. And these are models

that can actually run on CPU. You can

run them at scale. You can make sure

that you understand on a region level

what are the hard things for you to

process. And VLMs introduce this notion

of semantics. You can go through and

actually identify and make correct the

sorts of mistakes that you were likely

to have in your pipeline.

And off of that idea of semantics, uh,

when you're deconstructing this problem

and you have a clear sense of where the

more nuanced things are, if you've

segmented the text on your page and you

understand where the handwriting is, you

can introduce this notion of a sort of

agent in the loop. Whereas historically,

you would have had a human review team

go through and annotate and correct

mistakes. Uh, VLMs can now present this

sort of idea of what we call agentic

OCR. Really for us what that looks like

is if you've ever used you know a tool

like cursor that's applying fast edits

in your IDE there's this notion of

speculative decoding where you're

applying token level edits to your

output. You can apply a similar sort of

principle here uh that is not just you

know sending OCR to Gemini writing a

really pretty prompt asking it nicely to

not deviate too much from the original

because we find that when you're doing

that sort of next token prediction you

introduce net new loss cases where

models that are really intelligent will

start actually correcting things not

faithfully to what was in the document.

They'll see the word total and if the

human made a mistake in that table

models will actually sometimes go

through and add up the values in the

table themselves. What you want is to

correct the token level edits that you

want. Maybe you messed up a period

versus a comma, a zero versus an O.

Those sorts of details really, really

matter. And that's a question of how do

we actually represent what a human would

have seen if they had read that

document.

So the way that we see this is agentic

OCR is almost like that human loop

analogy where you have the first inputs

uh go through with a CD plus VLM parse

but then you have a verification

correction layer that ends up leading to

a high confidence output.

But I mentioned earlier that we don't

see the range of problems as purely just

parsing and extraction. And a lot of

what this looks like is I think it's

important to think through the details

of your pipeline. Even if you have a

great documents to markdown pipeline and

a great example of this is if you've

built any sort of rag platform um you've

probably had some consideration around

things like tables.

There are a lot of things that you can

encode well in something like markdown.

uh but things like this table where the

merge cells actually encode a lot of

meaning. It matters that you're

preserving that sort of structure and

it's not a model limitation. LMS are

incredible at reasoning through like an

HTML structure of the same table, but

you're also wasting a lot of tokens and

that gets expensive quickly. And

obviously on the other end of the

spectrum, you probably don't want to go

through and encode simple tables in HTML

because then you have a lot of HTML tags

that are erroneous. And so what we ended

up doing was looking at this as sort of

like a dynamic problem of when you have

a simple table, great, we can

approximate that data in markdown. When

you have a more complex table, you may

want to use something like HTML. But

it's not only language models that

should be a consideration in your

pipeline. If you're doing anything

related to embedding, you're also going

to have the secondary problem of

retrieval of that context. Right? That

same table that I showed you earlier, if

you look at the HTML representation is

really really messy. The vast majority

of that snippets is just HTML tags. It's

just classifying the structure of the

document. And the unfortunate thing is

whereas in some blanket evals, you might

have contents that is really trying to

do the work for the model and say

exactly what you're looking for. A real

world person does not enumerate the

values in the table. They just say how

did revenue change over time and they

assume that you're going to retrieve the

right table when it's relevant. And

whereas language models can reason

through that text effectively if you're

pulling from a large corpus, we find

that embedding models really struggle to

correlate that natural language human

prompt with this messy blob of HTML tags

and numbers. And so a big thing that you

can do that actually takes very little

effort is creating a representation

that's more so designed for the

embedding model itself. taking the same

table, we're creating a natural language

representation of that table. So you

have the best of both worlds. When

you're actually passing this into the

model for reasoning, you're using the

HTML table representation. And when

you're trying to make sure that you

retrieve the right snippets, you're

using that natural language block.

Off of that, uh there's also this idea

of there's a lot to do that is not just

parsing and extraction. Um, and I think

a lot of the industry's focus has been

on parsing and extraction historically

because we do think that there's a

massive uplift there. And I talked

earlier about the GDF GDP PDF benchmark

which I think is a great illustrative

example of what you can see uh as a

result of improving your data pipeline.

So what we found is that if you take the

same exact benchmark that I mentioned

earlier, uh, unfortunately we couldn't

test it on Fable because our access was

cut. uh but if you test on other models

and you give it both the original PDF

but also a structured representation of

the PDF like the parse results here

across models whether it's Gemini

whether it's anthropic or openi um you

find that you actually improve end LLM

performance just from better inputs and

it's to an extreme where models like GPT

5.5 and opus actually outperform

something like Fable out of the box not

just on an accuracy basis but as a

result of giving better inputs, the

models end up needing to use fewer

reasoning tokens as well. Um, they're

focused less on representing the data

and more on the actual outputs and as a

result they end up driving down latency

and also getting to the correct answer

more quickly. But even once you have

that sort of pipeline and you've gone

through and you've actually inspected

everything in your parsing layer, a lot

of human work is going to require

actually understanding the range of what

you have in your corpus, routing it to

the appropriate pipeline and sort of

decomposing that problem or even at the

end editing and modifying your document.

And so what we tried to do is look at

this as this problem of how do you make

sure that every interaction that a

language model has with a document is as

effective as if a human would have done

it. Um if you're filling out a form, how

do you make sure that you have precision

in where you fill out fields? And a

really good example of this uh on the

orchestration side is I think

classification splitting are a very

underappreciated way to have an LM do

its best work. Obviously, you can just

go through and dump as much context as

you want. And if you're doing a sort of

needle in the haststack test, that might

be fine. But in practice, there is

erosion that you find in quality outside

of just the token economics as a result

of passing in too much. And instead,

what we find is you can get a lot of

headroom by thinking through things like

how do you classify the right documents

to the right sort of pipeline? And even

for large documents, how do you make

sure that you're passing in the snippets

uh that are actually relevant? We see

use cases where people will have things

like paper mail. Uh and these paper mail

packets can be hundreds of pages long.

You don't necessarily know what is going

to be contained within it. Um you might

have issues like a person interle the

content and having the model do that

sort of work is almost like a

distraction from the work that you're

actually trying to achieve which might

be extracting the data from the paper

mail reasoning on it or making a

decision.

Again, uh I mentioned earlier that the

second half I think is the more

interesting piece. Um which is once you

have the sort of initial classification

and splitting layer, you've figured out

sorting. Uh I think the thing that we

are really excited about as a team is

agent harnesses have been this really

really interesting frontier to push past

what canonically used to be hard

unsolved problems. Um one good example

of this that I'll talk about in a second

is things like line charts. uh we work

with many of the largest hedge funds in

the world and things like line charts

historically have been really really

difficult because one they're an imaged

format but two there's a lot of pixel

level granularity that if you're doing

anything with a traditional vision

encoder you're probably going to lose

you're going to get a rough plot of how

revenue trended but you're not going to

get the individual data points and so

we've been thinking through how do we

give agents the ability to have the

right tools to solve for the specific

type of problem that you're looking at

in the case of chart extraction the

chart on the left encodes codes a

massive table of data. If you actually

went through and tried to plot every

single pixel, it would be really really

difficult. U but it would also be hard

for a model to even approximate the

intricacies of the lines in between. And

there's no model that out of the box can

do this as a singleshot problem. What

you're seeing on the right is a

reconstruction of the markdown table

that we're able to generate off of the

initial line chart. And the only way

that we were able to get there was to

have an agent with all sorts of tools.

It has its own code interpreter. It has

the ability to visualize the chart that

it's generating. And it's iteratively

going through. It's finding mistakes in

the line chart again and again and again

until it's able to get to the final

output. That applies for problems like

structured extraction as well where for

a while we've had this documents to

structured output feature. Uh but you

can really take it a step further by

having an agent harness around that same

sort of task. You can have a parent

agent go through set validation criteria

for sub agents to follow. Uh and this

means that if you have something like a

CBP form with tens of thousands of

fields, that's the sort of problem where

you end up finding a lot of issues that

are silent in nature like you drop

content, you drop rows and micro one

actually released a really good

benchmark in the space this morning

where there's this like bifurcation in

the market. Uh Frontier models with Max

Reasoning are really really precise.

Like provided that they extracted a row,

odds are it's not a hallucinated row

like they actually got it correct. but

they silently drop a lot of the contents

across the benchmark. Uh recall really

really struggles. On the flip side, a

lot of dedicated document processing

services are actually behind frontier

models from a precision perspective, but

close that gap on a recall perspective.

And so there's always been this sort of

trade-off and it was only with an agent

harness that we were able to find that

sort of local maximum of both precision

and also recall for this sort of task.

The last thing and maybe the most

important thing from this talk uh is

that I think at the end of the day eval

should underpin all of your decisions

and it's been a big part of how we think

about our product. Um that applies both

to off-the-shelf data sets that you eval

against but also to things like

real-time production monitoring because

your production data is going to differ

from whatever else you have in your

contrived set. And I really think it's

important to think of eval not as just

this like macrolevel view, but also the

best teams that we work with look at

eval on a granular level for each step

of their pipeline. Uh the first thing

might be that you want to make sure that

the inputs to your pipeline are great.

And of course, you should eval things

like your parsing pipeline. But even

perfect parsing with a horrible

retrieval pipeline is not going to help

if you're not passing the right context.

And so it's important that you're

thinking through details like your

retrieval pipeline, your formatting at

the end of the pipeline, and also

ultimately the most important thing is

are you able to improve end agent

performance.

I'll close off just with a a sense of

where we are headed and where we've seen

the industry head. Uh the most important

thing I think is as agents get better

and better, you can deviate from the

sort of deterministic pipeline that you

would have had a few years ago. Um, a

lot of our customers will actually

create effectively a file system for

their agent to go through and navigate

and let the agent decide what sorts of

tools it wants to use. So, we create a

CLI where instead of people creating a

endto-end pipeline where documents

always follow one specific flow, um, the

agent will decide if it needs to read a

certain type of document and they'll

split that into two sets. Uh, one is a

content field which the agent can read

as it needs to. Uh, but the other is all

the metadata that you would have wanted.

If you're doing things like citations,

you may want bounding boxes, so on and

so forth.

Um, I'll skip this part on editing. I

think there's a lot of interesting work

being done here. We've already released

some of it, but in the next few months,

you will see us look more and more

towards things like document generation.

But the recap for today, and I really

appreciate your time, is one, I highly

recommend that you decompose the parsing

problem. To the extent possible, you

should think of it as the right tool for

the right task so that you can hit that

perimeter frontier of accuracy, cost,

and latency. Two, I think agentic

verification is the biggest step change

that the industry has had for a while

and it's a really good opportunity for

you to make sure that you're building

pipelines that work in production.

Three, I think it takes very little

effort, but there's a lot of headroom

from details like formatting the data

for its consumer. or similar vein, I

think it's really really important to

think about not just the data processing

but the the data orchestration and so

you should always think about tools like

classify and split as a way to augment

your pipeline. Five uh make sure that

you eval at every stage and six think

through what that next frontier looks

like for you because I think most

successful companies in today's era have

deviated a lot from what we used to do

two three years ago. But if you have any

questions uh please feel free to reach

out at any point. My email is just first

namered reductto.ai. Uh and you can also

reach out on our website if we can be

helpful for your use case. Thank you.