[
  {
    "startMs": 12620,
    "text": "I'm going to show you the work we've been doing lastly on building, uh, our first foundation model, uh, a Wells Fargo foundation model. So I explain why we did that, how we did it, and what we accomplished in doing so. So, so first found-foundational model, it doesn't necessarily need to be language-based. I mean, we're all playing with, uh, Claude and, uh, ChatGPT and others. Uh, they're all language-based foundational models. In financial services, uh, we also have language, of"
  },
  {
    "startMs": 42600,
    "text": "course, uh, from a, uh, structure-- unstructured representation data. But we have also have a lot of structured representation. A lot of this data is tabular data. We have a lot of information from our customers. Uh, we have information in investment banking, consumer banking, and, uh, all of this data is data that we have in, in Wells Fargo and all fun-big financial organizations have too. And, uh, we want to take benefit of all of this huge amount of data that we have. And it's not like you"
  },
  {
    "startMs": 72320,
    "text": "just take it and put it in a, in a language model and get the best of it. Of course, you can get a lot of benefit out of it, but it's not necessarily the best way to do it. So, so what we decided to do is to look at the data we have, all the transactions, all the customer transactions that they're doing with their debit card, credit card, digital, check, all of those coming from many different channels, and see if we can build a foundation model that we can use and fine-tune for"
  },
  {
    "startMs": 102200,
    "text": "downstream tasks like credit scoring, or that you can use it for fraud detection, anti-money laundering, do marketing. So how we can build one backbone that you can use and streamline across downstream task. So now you're moving from many models very siloed across different organizations into really one backbone that every organization can just reuse because you learn the representations of customers' transactions at, at scale. So that's the intent. And just to"
  },
  {
    "startMs": 132040,
    "text": "follow on it, you may say, \"Hey, that's tabular data. Why do you just don't use XGBoost and the usual suspect?\" We did that. This is the model we had in production and-- But the model I'm going to show you is something that get-- that beats XGBoost, uh, even from a latency perspective, but also from a, uh, from a fraud detections. We get much better false, uh, um, reduction of false positive and better detection rate. So it was a very good lesson learned in term of, uh,"
  },
  {
    "startMs": 161840,
    "text": "having better model, uh, using foundational model. And to foundational model, I mean transformer blocks, uh, and that, that's, that's what we built. So, uh, so we have, as I mentioned, billion of transactions, uh, on a monthly basis, um, our customers interacting with their debit, credit card, and they do check, and they do different sort of transactions. And the, the question is really how we can use this informations at, at scale and not for one and every single downstream task. So I'm going to cover"
  },
  {
    "startMs": 192320,
    "text": "a bit of the data, the representations, the architecture that we use, uh, how-- that what the result we get, and the lesson we learned in building this. All right. So, I mean, left-hand side, language foundation model. They're processing language tokens, and then the, the goal is to learn a representation that going to predict the next token. And then you have, uh, you have supervised fine-tuning, and then you do uh, LHF, LRV, uh, and all of the great, uh,"
  },
  {
    "startMs": 222420,
    "text": "post-training process to get, uh, better at the task that you want to, right? And that's for languages. That's great. Uh, but what if your data is, uh, very much more specialized in term of representation? So that's where we build this transaction foundation model. So instead of having languages as the base, the base is now transactions, transaction from our customers. And, uh, you still try to predict the next thing, and here the next thing is the next transaction or it's the next"
  },
  {
    "startMs": 252180,
    "text": "token in the transaction, uh, using all of the channel that you have. Uh, but that's not the end goal. I mean, similarly to language model where you try to predict the next token, that's not the end of the story, right? You learn a representation, and then you use that representation, and then you, you, you fine-tune it, and you post-train it for any other task, right? So it could be classification task for languages, but in our context for transaction foundation model, it could be used for, uh, credit line increase, could be used for doing fraud detections and"
  },
  {
    "startMs": 281560,
    "text": "flagging bad actors from an anti-money laundering perspective. And the goal is to learn this, uh, relationship between transactions over time and transactions across customers. That's really the intent. So the intent number one is you do pre-train from scratch. I'm saying it again because people sometimes confuse. We pre-train transformer blocks, uh, at scale from scratch using our, uh, transaction data. So the weight, the zero weight that we use from open"
  },
  {
    "startMs": 311560,
    "text": "weights, we just use the architecture and then we, we pre-train it from scratch using our GPU cluster. And then we train it for couple of days and, and weeks. And then, then you do the, the downstream task of, uh, of post-training. But the goal here... And you pre-train it for predicting the next token or the next transactions. And you, you try to, to do this, right? You try to predict the next, the next pa-pattern of the customer. But that's not the intent. The intent is to do the downstream task later on. And in terms"
  },
  {
    "startMs": 341504,
    "text": "of scale, uh, uh, I mean, we, we operate at a large scale, so we have, uh, billions of transactions, a million of accounts, uh, hundreds of transactions per clients on a monthly basis interacting with their debit card, credit card, or doing check. And, uh, the five main dimensions we look at is really who is doing-- I mean, who is processing and who is asking and who is providing the service. So really the customer or the merchant."
  },
  {
    "startMs": 371704,
    "text": "Uh, the, the what, so, I mean, what, what do we, what do we buy? What do we transact? And how, so which channel? Is it credit? Is it debit? Is it digital? When is it happening, and where is it happening? And all of those, they're basically breaking down in hundreds of features. And, uh, if you look at the existing model we have in production, there's a lot of, um, and then people that build, build those model in similar organizations may, uh, may understand that they build a lot of very"
  },
  {
    "startMs": 401264,
    "text": "handcrafted, handcrafted engineering, uh, feature engineering effort on that. And that's required a lot and a lot of time. So we step back and look at really the raw features we need, not the one that's been probably pre-built and post-built for doing the downstream tasks, but try to understand which raw features. And we end up with, uh, basically hundred features. Even with, at the end of the day, we had thousands of them, but we tried to find which ones are really the, the raw information you need. And then we"
  },
  {
    "startMs": 431324,
    "text": "let the architecture or the transformer blocks to figure out the relationship between those core features. And, uh, as you can see, uh, we do have a, a, yeah, a bunch of features on f- those five dimensions. And the vocabulary is very unique to the financial services setup. So, and that's the reason why, I mean, we didn't really use, uh, existing pre-trained model, uh, because those-- the vocabulary that we use is very, very specific to"
  },
  {
    "startMs": 461304,
    "text": "the work that you want to do. And even if you merge, uh, well, all digital and checking and, uh, saving and, um, all of those sort of transactions, even if you merge all of those, you still end up with a very limited number of features and characteristics. So at that scale, then the question is, well, how, how you work with that? So, so we started, everything is tabular data. I mean, there's no secret, it's tabular data. And then the goal is to understand how you're going to"
  },
  {
    "startMs": 491524,
    "text": "capture the vocabulary, because you do have numerical values, you have categorical values, you have, uh, cat-- I mean, you have information like ZIP code. Do you encode each and every ZIP code in a token? Does it make sense? So there's a lot of efforts you need to push in understanding the vocabulary that you need and the vocabulary that you want also to scale. Because at the training, if you have a too huge vocabulary, it would be kind of messy, uh, to train such, such a model with a vocabulary which is far, far too big. So,"
  },
  {
    "startMs": 521484,
    "text": "so there's a lot of effort in understanding this, uh, value to token. So which information do you encode in a token? And then you have the token trading, the tokenizations, and then the embedding lookup. So then when we have a transaction coming up, you do have the vocabulary. So it's a-- basically, it's a lookup and aligning, uh, transactions and all the characteristics of that transactions to, to a token and an ID and then an embedding."
  },
  {
    "startMs": 548144,
    "text": "So just to give you a number-- well, some numbers. So we have, yeah, quantitative, qualitative, categorical, uh, of different kind, right? So categories with only eight types, some with much bigger types. Uh, you have transaction amount, which is a big deal. You have the delta time between transaction. So you have anything related to IP address, postal code. So just to give you example of, uh, uh, variety and heterogeneity of each and every, uh, characteristic could be encoded in a"
  },
  {
    "startMs": 577824,
    "text": "binary token or could be enco-encoded in a much larger number of tokens. And even if you look at the b-- the, the ZIP code, I mean, you have a much larger number of ZIP code in the, in, in the US than those, uh, twenty-eight hundred. But you, you have fancier-- well, you have much better way to structure how you're going to, uh, regroup ZIP code based on distributions, for instance. So, uh, architecture. So, so here just, um, what we-- So we use the, the Chinchilla,"
  },
  {
    "startMs": 607904,
    "text": "um, well, mathematics behind, uh, to define how we build it, right? I mean, initially, 'cause we, we had no clue how it's going to work in a tabular space, even though there are existing, uh, well, well, open weights and open solutions for doing, um, tabular data foundation models. Uh, so we, we, we started from what they learn and adapted to what we needed to do. And that gives you an example of embedding dimensions, number of decoder blocks we used, and, um, all of the,"
  },
  {
    "startMs": 638204,
    "text": "the recipe from a, from an architecture perspective. Um, so we have-- And a-as you can see, we're not trying to build a trillion parameters model. We don't try to compete with any of the, uh, the, the leaders in the language models because we're working with the-- and focusing on transactions, right? Transactions foundation models at billion level. And 1.1 billion is because that-- the model that give us, uh, that was the, well, kind of the smallest and give us the"
  },
  {
    "startMs": 667964,
    "text": "highest boost. And now we're working on a, on the next versions at eleven billion. But that's good enough to, to give a lot of, uh, ROI, uh, from a false positive predictions for, uh, downstream tasks like fraud detections and, uh, detection, u-unique detection rate. So, and, uh, and the, the type of architecture, so the decoder-only architecture, uh, that's basically for predicting the next, the next token or the next transactions. And that's really the, the pre-training. So, uh, so"
  },
  {
    "startMs": 698024,
    "text": "here we initially used twenty times the number of parameters to define the number of tokens we might need. We ended up actually ten times factor, factor than that. So by doing a, a few experimentations, we needed much more, uh, much more tokens per parameters. I mean, this is-- this kind of the initial experimentation we did. Um, uh- So I'll pass on this because this is the, the usual suspect from a, a objective function. So causal language, uh, modeling and math language modeling type of, uh,"
  },
  {
    "startMs": 727816,
    "text": "um, uh, optimization work. And, and, yeah, from an architecture perspective, nothing really, I mean, fancy here. The, the, the-- again, the most complex part was really the tokenization, and then, and then adapted and doing the pre-training, and then how you do the fine-tuning. So if... I would say the, the recipe or the secret sauce is really how you do the tokenization and then how you're going to do the post, the post-training. But anything from a training perspective, it's, uh, it, it, it-- the question is just how many,"
  },
  {
    "startMs": 757756,
    "text": "how many transaction you're going to put in your, in your context or in your history for, for doing the training. Uh, and evaluation perplexity-based accuracy, accuracy dashboard on downstream task. That's, that's the usual suspect. And, uh, so yeah. So use mixed precision, distributed data parallel, um, maximize microbatch size, uh, get faster convergence, um, uh, yeah, using Muon, uh, RoPE, uh,"
  },
  {
    "startMs": 787536,
    "text": "usual and memory efficiency, FlashAttention, activation checkpointing. I mean, here there's nothing really new. It just, uh, we use the, the, the, the, the advices from the, from the best and from the, the one that built even much bigger models. So, uh, I'll pass on this one. Time is short. But, uh, I wanted to spend some time, uh, yeah, on the results. So, um, yeah, so the... W-when you build this foundation, that foundation is"
  },
  {
    "startMs": 817136,
    "text": "really is, is of, is really, is, is on your backbone. It's something you can use for downstream task. And, uh, the downstream task was to do fraud classification. So you have a transactions, and then you try to predict if that transaction is going to be fraudulent. So in the model we have in production, it's an XGBoost model, and it's trained on large amount of transactions, which is, uh, fraudulent or non-fraudulent. And the missing pieces, that was the, uh, relationship"
  },
  {
    "startMs": 847096,
    "text": "between whatever transaction you try to predict, try to classify, and the relationship with this transaction with all of the past transactions. And that information is kind of gone when you use an XGBoost model, uh, unless you encode it in a very weird way, which is, which is not so easy. But if you use transformer architectures, you do have these nice, uh, attentions from a, from transactions to past transactions. And that's basically what help, uh, I mean, this is what we think. This is what really help,"
  },
  {
    "startMs": 877316,
    "text": "uh, pushing the fraud detection rate much higher. But at the end of the day, the, the downstream task was to say, \"Here's the sequence of transactions. I have someone just using the debit card. Uh, is that transaction, uh, fraudulent with respect of all historical information?\" And, uh, and that's, that's exactly what you put in, in, uh, at inference time when you try to do the prediction. So, uh, so that's the, yeah, architecture. So nothing really fancier. You just need to have the good, uh,"
  },
  {
    "startMs": 907716,
    "text": "engineering in place to do, um, less than two hundred fifty milliseconds for doing the processes bec- the processing, because the, the, the decline need to happen within that timeframe. But it's, it's, it's... We were, uh, well, ten times slower than that from a latency perspective, so it was all good. And, uh, and just in term of, uh, final result we get. So the main metrics, and those metrics relate right away to, uh, to dollar, to cost, well, to, to savings for"
  },
  {
    "startMs": 937656,
    "text": "the firm. So it's detection rate and false positive rate. Of course, you can detect much higher, but, uh, in ma-many cases, sometimes you can get much more false positive. And in this case, we managed to get m-- we flag more fraudulent act- activities, and we even reduce the false positive. So that, that was a big win. That was a big win because at the end of the day, if you decline a, a, a transaction which is not fraudulent, uh, then that person is going to call. There is then operation cost associated to it. You lose"
  },
  {
    "startMs": 967656,
    "text": "the transaction related to the, that non-fraudulent action. So it's a, it's a very, it's a very... Well, something not-- you, you don't really want to happen. So, so again, that's the, that's compared to the state-of-the-art of GV-- XGBoost model, uh, using fraud detection. So it's, it's basically huge. Uh, it's millions of, uh, dollar savings. Uh, so it's a, it's a very big boost. So just to finish up, uh, on that slide. So,"
  },
  {
    "startMs": 997516,
    "text": "um, so that foundation model has been built on billions of transactions from, uh, Wells Fargo customers. Uh, that's number one. It's, uh, it's pre-trained from scratch, so we don't pre-train from any open-weight existing model. We use existing architecture, but the weight is zero-weight. We pre-training on couple of days. I mean, we did it on a couple of weeks initially, but then that's pre-trained. And then you f- And then when it's pre-trained, it's pretty good in"
  },
  {
    "startMs": 1027496,
    "text": "predicting the next transaction. Then when you're done with that, then you can use it for any downstream task. We did it for fraud detection. Now we investigating for credit line increase, investigating for anti-money laundering, investigating for marketing. So now the same foundation is used across different organizations, uh, w-within Wells Fargo. And, um, yeah. Thank you for having me, and thank"
  },
  {
    "startMs": 1050616,
    "text": "you."
  }
]
