AI Engineer World's Fair 2024
Training Albatross: An Expert Finance LLM
Loading slides…
Read the talk
Training Albatross: An Expert Finance LLM
Building a finance expert requires both deeper domain knowledge and access to the right evidence at inference time. Gradient’s approach combines curated training data with extended context.
From a talk by Leo Pekelis
Before you start: Familiarity with language-model prompts, tokens, and the distinction between training and inference will help.
What does a finance application need from a model?
With foundation models appearing from AI companies, data companies, and general technology companies, why should a finance application simply choose a generalist model off the shelf? Each model brings different features and capabilities. Context windows were changing just as quickly: speaking at the 2024 AI Engineer World’s Fair, Gradient chief scientist Leo Pekelis described maximum context lengths growing from about 100,000 tokens to roughly forty times that over the preceding year. Those were the historical conditions behind Gradient’s premise: a model’s fit matters more as the application becomes more demanding.
Gradient’s answer was an AI foundry: a collection of custom language models and workflow primitives assembled into customer-specific systems. For finance, two components had proved particularly useful—a model trained on financial information and an extension to the amount of context a model could accept. The team had identified six requirements for finance applications; this walkthrough develops the two addressed by those components, rather than presenting a complete financial system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Broad coverage does not guarantee deep domain knowledge
General-purpose models train on a broad range of material, but technical financial knowledge can occupy the sparsely represented tail of that material. A large model may therefore encounter plenty of language about finance without seeing enough relevant documents to answer a particular technical question reliably.
The displayed chart, citing Kandpal and colleagues’ Large Language Models Struggle to Learn Long-Tail Knowledge, makes the exposure problem concrete across BLOOM model sizes. Pekelis reads the chart as showing that even the 176-billion-parameter model needed thousands of relevant pretraining documents to exceed roughly 50% accuracy on related questions in the illustrated evaluation. This is a result about the displayed question-answering evaluation, not a universal document threshold for learning any subject. The practical response is to give the base model more exposure to the finance information it lacks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce the corpus before asking people to review it
The immediate obstacle to additional finance training is volume. There is far more financial data available than people can inspect manually, so human review cannot be the first operation applied to every document. Gradient built an automated pipeline whose distinctive step borrows from membership inference: estimating whether a candidate document was already part of the base model’s training data.
The pipeline proceeds in this order:
- Collect a large financial corpus. Assemble the candidate material for domain training.
- Estimate prior exposure. For each document, use membership-inference-inspired techniques to estimate whether a base model such as Llama has already seen it during training.
- Filter likely previously seen material. This reduces redundant exposure and shrinks the candidate set. The decision is probabilistic; it does not establish the contents of the original training corpus with certainty.
- Review the remainder. The smaller set is manageable enough for human inspection.
- Augment the reviewed data. Synthetic augmentation both upsamples material and introduces variations in representation and formatting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn the information, then learn how to use it
The curated dataset feeds a training pipeline that starts from an existing foundation model. Continued pretraining uses next-token prediction on the domain data. Gradient then applies alignment through both supervised fine-tuning and preference optimization. These stages have different jobs: acquiring financial information and shaping how the model uses it.
| Stage | Training operation | Teaching analogy |
|---|---|---|
| Continued pretraining | Next-token prediction on domain material | Read the textbooks |
| Alignment | Supervised fine-tuning and preference optimization | Take an exam or work on a project |
The textbook analogy separates retaining information from applying it well. Alignment supplies instruction about useful behavior and best practices once the model has been exposed to the underlying material.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Correct the evidence available during inference
The second component—extended context—addresses hallucinations through the information available when the model answers. Here, hallucination includes content that is irrelevant, invented, or inconsistent with the input. One possible cause is outdated training data: a model may answer from information that was once current but has since changed. Another is source-reference divergence caused by bugs or inconsistencies in automated data collection, leaving the model trained on material that does not make sense.
Continued training and alignment can alleviate these problems. In Gradient’s experience, however, in-context learning was the most direct and sample-efficient mitigation: put corrective information into the prompt during the execution pipeline. A relatively small amount of relevant evidence can compensate for faulty or stale information learned during training. That is a way to reduce hallucinations, not a guarantee that every answer will follow the evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When useful evidence exceeds the prompt
Once supplying context helps, the natural next step is to supply more of it. The bottleneck becomes the context window: useful examples and evidence can outlast the available prompt space. Working around that limit adds its own costs.
- Example selection and summarization. When the examples do not fit, someone must choose which ones to retain or compress them through a potentially lossy summary.
- Preprocessing. More complex problems can require brittle pipelines that transform the evidence before the model sees it. Each stage creates another opportunity for error.
- Retrieval dependencies. In retrieval-augmented generation, one chunk may contain the information needed to discover that another chunk should be retrieved. Pekelis identifies this interdependence as a weakness of typical RAG systems: retrieving individually relevant chunks does not necessarily recover the connected evidence needed for the answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move domain adaptation into the context window
Extending the context window changes what can be supplied directly. With capacity in the million-token range, a prompt can contain thousands of examples rather than a small selected set. This returns to the domain-learning idea from earlier, but now the adaptation happens at inference time and can change with the problem, without a training update.
Pekelis says that some tasks need examples at this scale to reach production-grade—or potentially dangerous—accuracy, without specifying a task-specific threshold. The other opportunity is to let the transformer attend across the supplied material. Relationships that would otherwise require external orchestration can sometimes be resolved through the model’s attention layers. The benefit depends on the model actually using that context; capacity alone does not establish reliable reasoning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Public models and the evidence for each component
Gradient released models for both components on Hugging Face, tying the releases to transparent, verifiable benchmarks and to giving back to the open model ecosystem on which its work depended. The finance release, v-alpha-tross, applies the domain-training approach to a Llama 2 base. Pekelis reports competitive performance on general open-LLM benchmarks and better performance on finance-specific benchmarks relative to peers in the same model class.
The separate Llama 3 context-extension release targets a window of approximately one million tokens; the benchmark slide labels it Llama-3-70B-Instruct-Gradient-1048k. Pekelis reports 100% needle-in-a-haystack scores at tested context lengths above one million tokens. This measures recovery of a planted item from a large context, rather than establishing success on every task involving that much text.
He also reports a substantial improvement over the base model on NVIDIA’s RULER benchmark. In the talk’s historical comparison, he places the extended model in the realm of Gemini 1.5 Pro, GPT-4, and Command R+, without giving exact comparison scores in the narration. The finance benchmark, needle retrieval, and RULER results address different capabilities and should remain separate when assessing the releases.
Domain training and extended context are two important parts of a financial expert, not a complete account of how to build one. Pekelis closes by pointing interested builders toward Gradient for the rest of that system. The scope remains clear: teach the model more relevant financial information, then give it enough evidence at inference time to use that knowledge in the task at hand.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The public finance model's training recipe, reported benchmarks, limitations and loading instructions.
Configurable synthetic evaluations for retrieval, multi-hop tracing, aggregation and question answering.
Further reading
Research connecting factual question-answering accuracy with relevant document frequency in pretraining data.
Model weights, context-extension training details and needle-in-a-haystack evaluation settings.
Full transcript
Loading transcript…