A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo

Loading slides…

Read the talk

A Bank Trained Its Own Foundation Model to Catch Fraud

Wells Fargo’s transaction foundation-model presentation connects raw banking transactions, a shared backbone and specialized applications.Open full source frame
Wells Fargo’s transaction foundation-model presentation connects raw banking transactions, a shared backbone and specialized applications.

Freddy Lecue explains how Wells Fargo turned transaction tables into a transformer vocabulary, pre-trained a shared backbone from scratch, and used transaction history to improve fraud classification against its production XGBoost baseline.

From a talk by Freddy Lecue

At a glance

Ideas worth remembering

  • A transaction foundation model can learn from structured financial activity through next-token or next-transaction prediction, then be fine-tuned for separate business tasks.

  • Tokenization is a central design problem: numerical values, small categories and location information need different representations, and vocabulary size constrains the choices.

  • The selected model had 1.1 billion parameters. Experiments required substantially more training tokens per parameter than the team’s initial twenty-to-one allowance.

  • For fraud classification, the model receives a transaction sequence so the new event can be evaluated in historical context. The team credits these relationships with helping detection.

  • Lecue reports that, on holdout testing, the model with a 12-transaction history caught a 33% higher share of fraud accounts and produced 29% fewer non-fraud alerts per fraud alert than the production XGBoost baseline. Credit-line increases, anti-money laundering and marketing remained under investigation.

One backbone for billions of transactions

Wells Fargo already had tabular data and an XGBoost fraud model in production. The opportunity Freddy Lecue introduces is to learn more from the bank’s transaction history—and make that learning reusable. Debit cards, credit cards, digital payments and checks generate different records across different channels. Separate teams can build separate models around those records, but each then has to solve its own representation problem.

A shared foundation model can be adapted to fraud detection, credit decisions and other financial-services tasks.Open full source frame
A shared foundation model can be adapted to fraud detection, credit decisions and other financial-services tasks.

A transaction foundation model would learn a common representation of customer transactions at scale. Teams could then fine-tune the same backbone for credit decisions, fraud detection, anti-money laundering or marketing. Foundation models need not begin with language: here, the raw material is structured financial activity, with billions of transactions arriving each month.

The obvious objection is practical: why replace XGBoost, an established choice for tabular data? Lecue’s answer begins with the existing production system: “We did that.” The transformer approach had to earn its place against that baseline. He previews higher fraud detection, fewer false positives and a latency advantage, then works backward through the data representation and training decisions that made the comparison possible.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Predict the next transaction to learn a reusable representation

The training pattern resembles a language model’s, with a different unit of meaning. A language model learns from language tokens by predicting what comes next. The transaction model instead predicts the next transaction, or the next token within a transaction, using activity from the available channels. That prediction task gives the model a way to learn patterns before it is adapted to a specific business decision.

The slide compares predicting words in language with predicting transactions or transaction fields.Open full source frame
The slide compares predicting words in language with predicting transactions or transaction fields.

Next-transaction prediction is an intermediate objective. The useful output is a representation that captures relationships between transactions over time and, as Lecue describes the intent, across customers. Post-training then adapts that representation to questions such as whether to increase a credit line, flag a fraudulent transaction or identify potentially suspicious activity. Predicting a customer’s next action and deciding whether an action is fraudulent are different tasks; the first supplies a foundation for the second.

“From scratch” matters here. Wells Fargo reused transformer architecture rather than an existing model’s learned weights, then pre-trained on its own transaction data using its GPU cluster. The bank was therefore learning the financial representation itself, rather than continuing the training of a language model that already had a different vocabulary and training history.

3:123:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:12 · section reference included

Recover the raw information beneath engineered features

The input design starts with five questions about a transaction. Together, they describe the event before any particular downstream classifier decides what to do with it:

  • Who: The customer or merchant involved in requesting, processing or providing the service.
  • What: What is being bought or transacted.
  • How: The channel, such as credit, debit or digital.
  • When: When the transaction happens.
  • Where: Where it happens.
A transaction is described through who, what, how, when and where, with an explicitly illustrative record.Open full source frame
A transaction is described through who, what, how, when and where, with an explicitly illustrative record.

Those dimensions expand into hundreds of features. Existing production models also carried substantial handcrafted feature engineering, which took considerable time to build. The team stepped back from that inventory to identify roughly a hundred core raw features among thousands of available features. The aim was to let transformer blocks learn relationships among the underlying inputs, rather than begin with an extensive set of features already shaped around individual downstream tasks.

This still leaves a representation problem. Even after combining digital, checking, savings and other transaction types, the input has a relatively limited set of characteristics—but their meanings are specific to financial services. Lecue gives that specialized vocabulary as a reason for training the model’s weights from scratch. The next challenge is to express those characteristics in a vocabulary a transformer can process.

5:416:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:41 · section reference included

Tokenization carries the difficult design decisions

A transaction table mixes numerical and categorical values. A transaction amount, an elapsed time and a ZIP code cannot all be treated as interchangeable pieces of text. Each needs a decision about what information a token should preserve. Giving every distinct ZIP code its own token is one possibility, but a vocabulary that grows too large makes training harder. Vocabulary design therefore balances the distinctions the model needs against the size of the representation it must learn.

Once that vocabulary exists, an incoming transaction follows a concrete mapping: its characteristics are matched to tokens, those tokens have IDs, and an embedding lookup supplies their model representations. The difficult work precedes the lookup. The lookup applies decisions already made about which values should receive distinct representations and which should share one.

How much vocabulary does each field need? The examples show why the answer varies:

  • Small categories: Some categorical fields have only eight types; binary characteristics need fewer distinctions still.
  • Numerical and temporal values: Transaction amount and the time difference between transactions carry different kinds of information that need encoding.
  • Location and network information: Two postal-code fields account for 2,858 tokens. One IP-address field maps to 23 tokens. Lecue gives grouping ZIP codes by their distributions as an example, without specifying the grouping algorithm.
The vocabulary-design slide maps 97 raw features to 5,164 tokens across eight feature types.Open full source frame
The vocabulary-design slide maps 97 raw features to 5,164 tokens across eight feature types.

These examples make tokenization part of the model design itself. Grouping location values changes which geographic distinctions remain available to learn; representing time differences makes spacing between events available alongside their other characteristics. Lecue later calls tokenization and post-training the “secret sauce.” The transformer machinery can be familiar while the financial vocabulary requires substantial domain work.

8:119:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:11 · section reference included

A 1.1-billion-parameter model, with more data than first expected

What architecture was sufficient for the transaction task? The team used Chinchilla scaling mathematics as a starting point, then adapted through experimentation in the tabular setting. The selected model had 1.1 billion parameters: Lecue describes it as the smallest model that delivered the highest boost in their experiments. It used a decoder-only architecture for next-token or next-transaction prediction. An 11-billion-parameter version was work in progress at the time of the talk.

The backbone slide describes a decoder-only transaction model and its scaling and training configuration.Open full source frame
The backbone slide describes a decoder-only transaction model and its scaling and training configuration.

The initial data budget used twenty training tokens per parameter. Experiments required substantially more—described as roughly ten times that initial allowance. The useful correction is that a sizing rule borrowed from language-model training did not settle the transaction model’s data needs. The team had to test how much transaction data its chosen model required.

History length was another training decision: how many transactions should fit into the model’s context? That choice determines how much past activity is available when learning a sequence. Evaluation included perplexity-based checks of predictive learning and accuracy dashboards for downstream tasks, keeping the pre-training objective and the eventual business task in view.

The training setup otherwise drew on established large-model techniques. Lecue lists mixed precision, distributed data parallelism and maximizing microbatch size, alongside RoPE, FlashAttention and activation checkpointing. These supported training and memory efficiency; he presents no new training infrastructure algorithm. The distinctive work remained the transaction vocabulary, adapting pre-training to the data and deciding how to post-train for useful tasks.

10:0710:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:07 · section reference included

Judge the new debit-card transaction against its history

Fraud classification supplies the concrete downstream test. The production XGBoost model learned from transactions labeled fraudulent or non-fraudulent. The transaction foundation model brought a sequence representation to the same decision: someone has just used a debit card; given the preceding transactions, should this new transaction be classified as fraud?

Pre-training a transaction representation is followed by fine-tuning a fraud-classification head.Open full source frame
Pre-training a transaction representation is followed by fine-tuning a fraud-classification head.

The proposed advantage is access to relationships between the current transaction and earlier ones. In Lecue’s account, the XGBoost system needed those relationships to be encoded into features explicitly, a difficult engineering task. Transformer attention lets the current transaction draw on past transactions in the supplied sequence. Lecue identifies this as the team’s explanation for the higher detection rate, rather than presenting an experiment that isolates attention as its cause. The comparison concerns their production baseline; it does not mean tree models cannot use historical features.

At inference time, the input is therefore the transaction sequence, including the new event and its historical context. This makes the history-length decision operationally relevant: the classifier can use the history supplied to it. The model must also fit a card-decision processing window of less than 250 milliseconds. Lecue presents the engineering outcome as satisfactory and earlier claims a latency advantage over XGBoost, but the closing latency wording is internally inconsistent, so it does not establish a reliable measured inference time.

13:3714:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:37 · section reference included

Catch more fraud while declining fewer legitimate payments

Wells Fargo compared its production XGBoost model, which used about 100 engineered features, with a fine-tuned transaction model that used none. On a holdout set of about 10 million samples, the discovery rate measured the share of fraud accounts caught. The false-positive ratio measured non-fraud alerts per fraud alert at a 0.5% cutoff. With last-token pooling and a five-transaction history, the model raised the discovery rate by 28% and lowered the false-positive ratio by 26% relative to XGBoost. With the same pooling and a 12-transaction history, the discovery rate was 33% higher and the false-positive ratio was 29% lower than XGBoost’s. The slide gives no absolute rates or holdout dates.

How do those two metrics turn into business value? More detected fraud addresses fraudulent activity directly. Fewer false positives avoid another chain of costs: a legitimate customer’s payment is declined, the transaction is lost, and the customer may call the bank, creating operational work. Lecue attributes millions of dollars in savings to the improvement. That is a reported business outcome, without a savings calculation developed in the talk.

Wells Fargo reports higher fraud discovery and fewer false positives against its engineered-feature XGBoost baseline.Open full source frame
Wells Fargo reports higher fraud discovery and fewer false positives against its engineered-feature XGBoost baseline.
15:3716:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:37 · section reference included

Fraud is the demonstrated use; other teams reuse the foundation

The closing distinction is between building the foundation and adapting it. Wells Fargo pre-trained on billions of its customers’ transactions, using an existing architecture without importing existing learned weights. Lecue describes initial pre-training runs taking a couple of weeks and later runs taking a couple of days. Once the model had learned to predict transaction patterns, that foundation became available for downstream training.

The closing slide summarizes transaction-model lessons and proposed extensions beyond fraud detection.Open full source frame
The closing slide summarizes transaction-model lessons and proposed extensions beyond fraud detection.

Fraud detection was the completed application presented here. Credit-line increases, anti-money laundering and marketing were under investigation. The same foundation was being used across organizations within Wells Fargo, fulfilling the original goal of sharing learned representations. The fraud result gives that approach a concrete first use; the other applications still need their own task-specific evidence.

16:3717:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:37 · section reference included

Transcript download
Full transcript

Loading transcript…