AI Engineer World's Fair 2026

Bypassing the Multimodal Tax: Framework-Free Hybrid RAG, Raw SQL RRF, and Live UI Telemetry

Read the talk

Bypassing the Multimodal Tax: Framework-Free Hybrid RAG, Raw SQL RRF, and Live UI Telemetry

Prepare documents before employees ask questions, inspect the chunks that drive each answer, and keep retrieval, application rules, and telemetry visible in a local Python stack.

From a talk by Abed Matini

Before you start: Familiarity with Python, SQL, and the basic idea of retrieving document excerpts before generating an answer will help you follow the implementation choices.

Why process the handbook before the first question?

Drag a PDF into a chatbot and ask it to get ready for questions. Before the first question arrives, the system has already done document-processing work that may consume paid tokens. Add a vector database, keyword search, semantic search, and more tools, and a second problem appears: the application becomes harder to operate. These are the two costs Abed Matini starts with—document ingestion and fragmented infrastructure.

Slide with two boxes titled “Paying to read documents twice” and “Search split across too many tools,” beside an FAQ assistant.Open full source frame
Two document-chatbot problems: ingestion costs and fragmented search tools.

His working example is a local HR FAQ assistant. An administrator uploads an employee handbook; employees ask about sick leave, parental leave, and other recurring policies. The inputs can be PDFs, Word documents, presentations, or images. Rather than leaving document preparation hidden inside the chat interaction, the admin dashboard exposes the chunking strategy and the material that will become searchable.

The walkthrough follows those documents into embeddings, keyword and semantic retrieval, rank fusion, and finally an answer with telemetry and safety checks. The intended packaging is a repository that can start in GitHub Codespaces with one command. That goal shapes the architecture: keep the components small enough to run together and make their behavior inspectable.

1:041:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:04 · section reference included

A local stack with two paths

The application uses Python and FastAPI for the backend, React for the frontend, PostgreSQL for storage, and Docker for reproducible services. Ollama serves the local language and embedding models. Matini presents CPU-only operation as sufficient for his small deployment, including staging servers or Codespaces; this is a deployment claim rather than a hardware sizing result. Langfuse records conversations and latency. Here, framework-free means avoiding a specialized RAG orchestration framework, not avoiding web frameworks or libraries.

There are two paths through the system:

  1. Prepare the knowledge base. Convert source documents to Markdown with Docling, split the result into chunks, and store the searchable material in Postgres. The blueprint also includes an ingestion scan.
  2. Answer a question. Retrieve relevant chunks for the employee’s query, pass the highest-ranked material to the language model, return its answer in the chat interface, and observe the interaction in Langfuse.

Separating these paths lets document preparation happen before an employee needs an answer.

Flowchart linking raw documents, Markdown, chunking and embedding, Postgres, RRF retrieval, an LLM, React UI, and Langfuse, with an injection-scan branch.Open full source frame
The end-to-end blueprint connects document processing, retrieval, and answer delivery.
5:526:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:52 · section reference included

Make document structure visible

Uploading a document to an opaque ingestion service leaves an important question unanswered: what did the system actually read? A table may have lost its structure, or a chunk boundary may have separated a statement from its context. Saving an inspectable intermediate document makes these failures easier to find. The structure-first path converts locally, saves Markdown, applies the chosen chunking strategy, embeds the chunks, and inserts them into vector storage. This walkthrough uses Markdown post-processing; that should not be confused with Docling’s current native structural and tokenizer-aware chunkers.

Matini motivates local preparation by contrasting a short document with hundreds of pages and repeated employee use. The intended saving is to avoid unnecessary cloud document-processing work and send smaller, relevant prompts later. Local operation can avoid per-call provider fees, but it still consumes compute and storage; no cost comparison is measured here.

The admin demonstration makes the quality problem concrete. Existing indexed documents can be removed before updated HR policies are uploaded. In the whole-handbook preview, acknowledgements and fragments such as signature dates have become chunks even though they are poor answers to employee questions. Matini recommends splitting the handbook into relevant files and cleaning them before indexing.

For an FAQ assistant, one useful preparation step is to rewrite policy material as explicit question-and-answer pairs. Working-hours information, for example, can become a question followed by its policy answer. The resulting unit is useful both for retrieval and for citation: the reference points to the specific policy explanation instead of an arbitrary stretch of the handbook.

8:118:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:11 · section reference included

A question, its answer, and a traceable chunk

Matini uploads the prepared FAQ PDF with heading-based chunking selected. Docling identifies the question headings, and the application keeps each heading with its answer as a chunk. The admin preview then shows those question-and-answer units individually. This makes the retrieval unit match the way an employee is likely to ask for information.

The insurance question provides the first useful end-to-end check. The sample policy says the company covers 90% of individual employee premiums and 75% of dependent premiums. The chatbot returns that information, and the opened reference shows the matching FAQ excerpt. The visible source matters as much as the fluent answer: it lets the reader check which policy text supplied the numbers.

Document-splitting slide beside an FAQ answer about insurance premiums, with a source popup showing the matching FAQ question and excerpt.Open full source frame
A completed insurance answer with its source excerpt open.

That reference also gives the developer a debugging path. If an answer is wrong or missing, inspect whether the intended chunk was retrieved before changing the prompt or model. The demo retrieves two chunks. Alongside the clean FAQ unit, it retrieves material from the larger employee handbook; that larger chunk is harder to follow and assess. A retrieval result can contain related information without being a good, readily checkable unit of evidence.

13:5214:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:52 · section reference included

When headings are not the right boundary

The next upload is a general-policy document using paragraph-based chunking. Each paragraph becomes a separate chunk whether or not a heading exists. Matini compares the admin preview with the prepared source: the document’s paragraph boundaries now determine what retrieval can return. This works naturally when each paragraph already contains a coherent policy statement.

For a less organized global-travel document, the demo uses 512-character chunks with a 64-character overlap. These are character counts, not token counts. Overlap repeats text near adjacent boundaries so that a cut does not remove all local continuity. A Python implementation of that fixed-window strategy is straightforward:

python

def character_chunks(
    text: str, size: int = 512, overlap: int = 64
) -> list[str]:
    if not 0 <= overlap < size:
        raise ValueError("Require 0 <= overlap < size")

    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append(text[start:end])
        if end == len(text):
            break
        start = end - overlap
    return chunks

The code keeps the final chunk and avoids creating another chunk consisting entirely of the previous chunk’s overlap.

In the preview, Matini describes the shared boundary region as two halves adding up to the overlap. But the same example exposes the limitation: a boundary cuts through the word assistance. Overlap preserves nearby characters; it does not make a boundary semantically sound. Fixed windows are a practical option when cleaning the source is difficult, rather than a universal chunk-size optimum.

Sentence grouping offers another boundary: collect a chosen number of sentences, with five used as the example. The options differ primarily in what structure they trust:

StrategyBoundarySuitable source
HeadingHeading and associated textPrepared FAQ
ParagraphParagraph breakGeneral policy prose
Fixed windowCharacter count plus overlapUnorganized text
Sentence groupSentence countShort messages

The next demonstration uses sentence groups for a temporary update that is too small to justify extensive document preparation.

17:5018:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:50 · section reference included

Turn a maintenance screenshot into an answer

Suppose HR receives an email announcing weekend maintenance, but the available input is a screenshot. The useful outcome is simple: an employee should be able to ask whether maintenance is planned and receive the current notice. Matini uploads the screenshot, converts the image to text and then Markdown with an unnamed conversion model, and selects sentence-group chunking.

The admin preview exposes the extracted chunks before the screenshot becomes part of the knowledge base. An email may have neither meaningful headings nor reliably separated paragraphs. For a short, temporary notice, sentence grouping makes it possible to ingest the update quickly without first turning it into a polished policy document.

Matini asks whether there is a maintenance plan for the weekend. The answer cites the newly uploaded screenshot and reports the sample maintenance window as tomorrow from 6 p.m. to midnight. A descriptive filename would make that reference easier for an employee to recognize. After the notice expires, the administrator can archive or delete it so it is no longer visible.

The demonstration ties ingestion quality to answer quality without changing the language model. A prepared FAQ, a coherent policy paragraph, and a quickly extracted maintenance notice need different boundaries. The common requirement is to inspect what was indexed instead of assuming that uploading the entire handbook produced useful evidence.

22:1122:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:11 · section reference included

Keep predictable work in Python

With the knowledge base prepared, the next question is how much work should require a model call. Several capabilities Matini calls agents are ordinary Python functions. His concern is local response time: he estimates that several agents repeatedly looping could leave a user waiting 20–30 seconds in his Ollama scenario. This is an illustrative estimate, without a measured workload or hardware comparison. If a function can do the work, it avoids another generation step and can be covered by a test suite.

The settings expose switches for these capabilities. Current-date lookup and arithmetic do not require a language model to invent an answer. Planned customer-service extensions include product information, prices, and calculations. A deeper search can add information, but Matini flags a cost beyond latency: the extra path may lose the references established by the original retrieval.

The selected chat model is Qwen2.5 0.5B Instruct. Matini tentatively puts its download at about 400 MB; that is an artifact-size estimate, not a runtime-memory requirement. Optional agent mode adds another model invocation and takes longer. The intended division of labor is to let code gather and constrain the information, then ask the language model to express the result.

26:1726:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:17 · section reference included

Combine similarity with exact terms

The database associates a chunk with an ID, a section, and its vector embedding. Dense retrieval finds nearby material by meaning, while keyword retrieval provides another route to relevant text. The distinction matters when the query names a product number or medication: a semantically similar item may be the wrong item. Semantic proximity cannot substitute for an exact identifier.

The dense-retrieval slide shows a cosine-distance SQL query with a 20-candidate fetch before fusion; the FAQ’s final retrieval limit is two chunks. Those limits serve different purposes: a candidate pool gives ranking more material to consider, while the final limit controls the context passed onward. A product catalog may need a wider window than this FAQ so that other products from the same brand are not excluded before they can be considered.

“Raw SQL: dense retrieval” slide showing a SQL query and notes about cosine distance and fetching 20 candidates before fusion.Open full source frame
Dense retrieval SQL with cosine distance and a candidate-fetch limit.

Matini describes the lexical side as BM25. On the dense side, the query is embedded and compared with stored embeddings using cosine distance. On the sparse side, terms and constraints such as language, IDs, product names, SKUs, and brands help locate the intended records. BM25 here is the speaker’s description of lexical retrieval; it should not be read as the name of PostgreSQL’s built-in full-text ranking function.

The two result lists must then be combined into a useful ordering—the role assigned to Reciprocal Rank Fusion in this architecture. The important application choice is how much of the combined result to keep. Matini contrasts a small answer set with a long list that would confuse the user, and recommends wider retrieval for catalogs and tighter retrieval for focused medical questions. Retrieval breadth should follow the task; narrowing a medical result set does not itself establish clinical accuracy.

After discussing ranking and the displayed result limit, Matini contrasts two execution paths:

ModePathTradeoff
Direct RAGEmbed query → hybrid retrieval → answerPredictable sequence and easier tracing
Agent modeAdd tools such as search or product comparisonMore information, extra calls, less control

The fixed path is attractive when compliance and traceability matter. Optional tools broaden what the application can do, but each additional call introduces more work and another place where the answer’s provenance must be preserved.

29:2829:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

29:28 · section reference included

Inspect the conversation behind the answer

Langfuse is integrated into the application’s functions and runs locally in the demonstration. Matini opens the anonymous widget conversation under the FAQ agent project. A conversation ID connects the maintenance question to its answer, so the interaction remains traceable even though the user was not logged in.

The trace shows the selected model, agent mode off, the chat operation, returned chunks, and latency in milliseconds. In this maintenance conversation, retrieving two chunks produced two references to the same screenshot. That is a useful distinction: the number of returned chunks is not necessarily the number of independent source documents.

For external models, telemetry can also support cost estimates. Where user identities and login information are available, session and user IDs help connect interactions across visits. These fields make the trace useful for investigating both an individual answer and how people use the assistant over time.

35:3135:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

35:31 · section reference included

Stop an out-of-scope request before generation

Matini tests the FAQ assistant with a flu-treatment question. Instead of asking the answer model to respond medically, the application returns a configured escalation message directing the user toward a healthcare practitioner. The branch happens before answer generation. For a request the product should not answer, the application can choose a response without delegating that decision to its normal chat prompt.

This helps explain the short system prompt shown in the settings. Some do-and-don’t behavior lives in code rather than in a longer instruction string. The medical rule is an example to expand with relevant keywords, sentence variants, and attempts to bypass it. Matini proposes the same pre-generation pattern for unrelated questions and prompt-injection attempts.

Telemetry and application guardrails slide beside the question “how do i treat flu” and a yellow medical escalation response directing the user to a healthcare practitioner.Open full source frame
The FAQ assistant responds to a flu-treatment question with a medical escalation message.

After noting that Langfuse requires a generated key configured in the system, Matini returns to injection screening. He names intent regexes, term dictionaries, and an LLM classifier as possible checks. The boundary to preserve is before the answer-generating model: using a classifier for screening is distinct from allowing the normal answer path to run, although its exact placement is not specified here.

Code makes a rule’s execution predictable and its expected behavior testable. Tests can specify which requests should be blocked and why. They still need to challenge the detector, not merely confirm its intended examples: a deterministic regex can consistently miss an injection. The medical response demonstrates an application branch, while comprehensive injection resistance requires broader coverage than that one successful case.

37:2737:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

37:27 · section reference included

Fit the model and deployment to the prepared context

Matini initially used Qwen2.5 with 7 billion parameters, then switched to the 0.5-billion-parameter model because the larger model took longer on his small server. The admin settings retain alternative model choices. He also found larger responses could be too wordy, while the smaller model could answer effectively when given vetted context. His observation that it hallucinated less and declined to answer when information was missing is not a comparative safety evaluation; the practical design lesson is to assess the model together with the evidence supplied to it.

BGE-M3 provides the document embeddings. Matini describes the core setup as one chat model plus one embedding model. That describes the retrieval-and-answer path; the earlier screenshot conversion is an additional ingestion capability whose model was not named.

Settings page with qwen2.5:0.5b-instruct selected as the chat model, Top-K chunks set to 2, and Agent mode switched off.Open full source frame
Admin settings show the selected chat model and retrieval count.

Docker packages the services, with Python requirements installed and React providing the frontend. At the end, Matini catches an omission in the deployment presentation: Langfuse also needs to run, on a separate port. It belongs in the deployment inventory alongside the application, database, and model service, rather than being treated as something that appears automatically when tracing calls are added.

The resulting architecture keeps document preparation, storage, generation, and observation separate: Docling produces structured Markdown, Postgres holds the searchable material, Ollama serves the chat model, and Langfuse exposes the interactions. It does not require a specialized orchestration framework or a paid framework service. The same arrangement can support document collections, product catalogs, or database-backed assistants, provided their source preparation, retrieval breadth, and application rules are adapted to the questions they need to answer.

42:0842:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

42:08 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…