# Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo

Francesco Galletta, Monzo · Felippe Felisola Caso, Monzo

[Talk page](https://ai.engineer/talks/ATMcd9tkhhc-stop-chatting-spreadsheets) · [Video](https://www.youtube.com/watch?v=ATMcd9tkhhc)

## Stop Chatting With Spreadsheets

What will the wedding cost, and will the money be there when each payment is due? Francesco Galletta and Felippe Felisola Caso use the second question to build a financial-analyst agent whose calculations can be rerun and whose assumptions can be challenged.

![Five colored bands name the parts of the harness and give a one-line purpose for each.](https://ai.engineer/api/talk-essay-media/2743020862903845cc0372b1c277f2881f33a1f9010d38c4d67114c5f42774f7)

The five layers give different jobs to the user, the model, the calculation code and a reviewer.

### What would it take to delegate the analysis?

[0:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=0s)

A financial analysis can finish in the background while its owner does something else. If the owner then has to reconstruct the work before acting on it, the fast answer has saved very little. Francesco Galletta and Felippe Felisola Caso bring that problem from their work in data at Monzo to a personal finance workshop, speaking in their own capacity. At work, they judge an AI project by the business benefit it produces. At home, Galletta has a similarly practical test. Can he hand off the tedious calculations and trust the result with his own money?

The workshop’s *harness* is the software around the model that guides the interview, runs calculations and gives the user access to the decisions behind an answer. A trace can show where a number came from. A unit test can catch a broken calculation. The builder still has to decide which assumption a person needs to see before committing money. The speakers organize those choices around five principles. Shared domain establishes what the user means. Consistency makes the arithmetic repeatable. Evidence records what went into it. Accountability gives the person who bears the consequences a say in the decision. Verification brings in a perspective that can challenge what both the user and the agent have accepted.

### A dinner price is only one part of a wedding budget

[11:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=717s)

Galletta has an ulterior motive for convening the room. He needs help working out whether he can make each wedding payment on time. He and his fiancée have already found that a perfectly reasonable number can answer the wrong question. She has the price of dinner. He is counting two additional variable costs on top of dinner. Multiplying the dinner price by the guest count would faithfully reproduce just one part of the bill.

Participants can open an empty workspace for a financial question of their own or load a worked wedding example. The repository includes a setup guide, the harness, the example and a place to save their own work. A successful local setup opens the UI. The empty workspace is where Galletta demonstrates the interview. The prepared plan is where participants can inspect decisions, run tests and try corrections without first having to agree on every wedding expense. [Repository QR code and setup instructions at 14:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=861s)

### Shared domain begins with three payments and an unknown head count

[20:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1238s)

In the empty workspace, Galletta dictates the problem into the chat on the right. He owes three payments. The first is $6,000, the second is an unknown amount, and the third covers the remaining balance. The DJ is a fixed cost. Dinner is priced per seat, and attendance is uncertain. Even a correct total for the wedding wouldn't tell him whether he can pay on time. For that he needs the amount due at each deadline and the cash available by then.

What can the planner establish from that first request? The harness instructs the agent to interview the user and specifies what it needs to learn before proposing a plan. Galletta supplies the next details in deliberately untidy order. For the exercise, he has no savings, his mother *may* contribute 5,000, the price per person is *around* 150, flowers cost 500, and he still needs a suit. The qualifiers matter as much as the numbers. Counting a possible gift as available cash could make a payment look affordable before a single arithmetic error had occurred.

The follow-up questions put some work back on the one person who knows this wedding. A model can recognize a term and still misunderstand what the user includes in it, just as two people can mean different things by the cost of dinner. The interview gives the user a chance to catch that difference before it becomes a formula. Galletta asks for a proposed plan to keep the exercise moving, then switches to the prepared example to show what comes out of a fuller exchange.

### An accepted plan gives each decision a place

[30:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1843s)

The prepared example picks up after roughly ten minutes of earlier discussion. Its accepted plan breaks the problem into decision points. The graph on the left shows how those points connect, and the plan appears at the upper right. This is an implementation choice in the interview instructions. Specifying what the agent must produce means the user finishes with individual decisions they can reopen.

Who put a particular assumption into the plan? Selecting step three brings up the source of its information and the reason the agent considers it a decision. In the worked example, a word Galletta used prompted an inference about what he meant. The annotations distinguish what he supplied, what the agent proposed and what still needs a decision. Accepting the plan leaves him free to question that inference.

The next exercise is to select step two and challenge an assumption there. In a long chat, a correction joins every earlier proposal, detour and correction in one stream. After ten or more exchanges, it can feel easier to start over than to recover what was agreed. Attaching the discussion to a node gives the user a way to revisit one question with the rest of the plan still in view.

![The actual workshop app shows linked calculations and the accepted plan with seven steps and eight inputs.](https://ai.engineer/api/talk-essay-media/72f4c0d631ef57f68a456f8ab61b1a68d71499eac65e002b3ee64e81cea365bc)

The wedding example records the accepted plan as a graph. A user can trace a question about monthly savings back to the calculations it depends on.

### Consistency means running the same calculation again

[36:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2201s)

Some nodes in the accepted plan are ordinary calculations. Given the same head count and price per seat, the dinner line should have the same value on every run. Galletta recalls a chat response that typeset a formula beautifully and followed it with a result so far off he could spot the error in his head. The typography had done a better job of convincing him than the arithmetic.

An audience member presses the objection. If the model also writes the code, what has improved? Saving an implementation makes it possible to inspect and test the method independently of the conversation. A forecast can call an established library. A custom calculation can be broken into functions with unit tests, then checked end to end. Asking the agent to generate fresh code for each question gives up much of that benefit. The hundredth run could quietly use a different implementation under the same description.

Galletta ran into this while closing his own monthly budget. He let an agent categorize statement entries, aggregate them and handle refunds and expenses. It got the result he wanted three times and missed on the fourth. Reconciling the slight differences cost time and money. His correction was to put the repeatable work in a script. The model can still choose a tool for the problem. Once the method is settled, next month’s close can run the same calculation.

### Build a calculation node, run its tests, then try to break it

[45:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2723s)

Which implementation produced a figure? The wedding graph identifies nodes that need code, and the UI offers to build that code once the user accepts the plan. Galletta also describes a practice from other implementations that matters on repeat runs. Record the version of the code and flag a change, even if the function has kept its name. A result earned with an earlier formula tells you little about what the revised one will do.

To inspect the first wedding calculation, select step one, open **Code**, open **Tests**, and run the tests. The tests execute in the participant’s environment. There is no model deliberation to wait for. The prepared tests are expected to pass, and the same action is available after an edit. Putting the tests beside the node gives the user a way to check the implementation at the point where they depend on it.

The repository’s build-your-own guide works a little like an IKEA guide for the harness. It gives each component a job, such as gathering enough information to propose a plan, and leaves the implementation choices to the builder. An audience question about spec-driven development brings out the result of that choice. The saved workspace acts as a specification. As the wedding changes, the analyst can revisit a decision or a calculation through its own workflow. A single growing document would ask the agent to interpret the whole specification afresh.

Changing an input and changing a definition are different operations. A new guest estimate can go through the existing calculation. A change to what a metric counts may require new code. The suggested stress test is to replace a multiplication with an aggregation and save the file. The edit can save successfully even though it breaks the analysis. On rerun, a failing test is designed to stop the harness before it draws a conclusion from that node.

### Trace the number and the method that produced it

[56:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3364s)

Suppose Galletta supplies the number of people invited and his fiancée has just invited three more. The code can calculate dinner costs correctly for a head count that is already out of date. Evidence lets someone work backward from a total to the input the calculation used, then ask who supplied it and what that person knew.

Three kinds of information can end up looking equally certain in a cell. The harness needs to preserve the difference.

- **Interpretation.** Turning an unstructured remark in chat into a budget item requires a reading of what the person meant.
- **Assumption.** A value the agent infers or recalls needs to remain identifiable as its contribution.
- **Recorded fact.** A bank statement documents the amount of a recorded transaction. That amount has a different standing from an estimate of a future expense.

In Galletta’s master-file story, someone entered familiar numbers into the spreadsheet everyone used on the shared drive. Everything looked fine. The file had changed, including its assumptions about the range of reasonable values, and the person using it had no reason to know. The inputs stayed the same and the results were entirely wrong. Tracing the origin of every input would have missed the problem. The trail also needs to identify the process that transformed those inputs and what changed in it.

Which part of the wedding plan came from the assistant? The evidence view highlights where information entered the plan, including the assistant’s proposals. A financial analyst might need finer detail still, down to the supplier or connector that reported an Apple share price at a particular time of day. The follow-along exercise tests the same idea with a chat message. Enter a correction, inspect the affected inputs, then select an underlined figure that changed. Its trail should lead back to the user’s correction and explain why it matters to the rest of the calculation.

A slide-building experiment shows how easily an example can acquire the status of evidence. Galletta’s agent gathered relevant documents and helped pick points for a team presentation. During revision, he supplied an example to explain a change. The model repeated it throughout the deck. When senior leaders asked where a number and an assumption had come from, he couldn't account for them. An instruction about how to edit had become a claim about the business.

![The exercise opens the evidence behind an underlined number in the wedding example.](https://ai.engineer/api/talk-essay-media/53cf75c2bd12a60e9dbac606178446fda54fcaa41c1f283003d83e55a67b4a99)

Opening the underlined 31,000 reveals which inputs and test run produced it.

### Put the human decision where the money depends on it

[1:06:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4001s)

The person who acts on the analysis bears the consequences. Asking that person to press Enter for every edit spends their attention quickly. One participant describes reviewing a task, letting the agent carry it out and reviewing the result. The financial equivalent is to let the system evaluate three scenarios, then put them side by side for the person making the choice. A test can settle whether a function still performs its calculation. Choosing which scenario to budget for calls for judgment.

What timing assumption could change the savings decision? The prepared wedding plan has one final human checkpoint. Galletta reviews the scenarios and chooses how much to save each month. One assumption worth opening is an incoming 20,000 expected on the same day as payment three. Sharing a calendar date still leaves a question about whether the cash will be available when the payment is due. Accepting that premise records a decision that later work can use. The exercise stops at choosing a savings amount, short of transferring money or reporting a final figure.

A user can also reopen a decision after learning something new. Hover over the guest-count chip, add that item to the chat and tell the planner to expect **no fewer than 150 guests**. That supplies a lower bound. It changes which attendance scenarios make sense and, through the per-seat costs, what those scenarios cost. The planner can suggest a change to the plan or revise the input if the existing calculation already fits. The history distinguishes an assumption the user allowed to stand from one they corrected.

Step five offers a different exercise. Open a branch conversation and explore a question about that decision. Galletta had often opened an entirely separate chat to ask a speculative question because a stray sentence in a long conversation could redirect the analysis. A branch gives the aside a home beside the decision it concerns. Whether its conclusion should enter the main plan is a separate choice.

![The app’s decision graph sits next to a note about a payment and the balance forecast.](https://ai.engineer/api/talk-essay-media/ce1d92c05d776b1199aadc8ff4161317a773c30de232145eb6af81c9f45c2404)

A note about a 20k payment changes the cash-flow forecast because the deposit and the last wedding payment fall on the same day.

### Give the reviewer an outside reference and a way to change the plan

[1:20:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4834s)

A second pair of eyes can catch something the author has stopped seeing. It can also become a way of handing a colleague a pile of plausible work and making them responsible for every mistake. Galletta’s model for verification comes from finance teams. An experienced reviewer opens a spreadsheet, checks a few cells that reveal the rough total, and asks how marketing could possibly have spent that much. The discrepancy tells the analyst where to investigate. Responsibility for the analysis stays with its owner.

Golden examples give a test known cases with expected answers. They also reflect the cases their author chose. In an earlier business intelligence project, a query looked sound, its result seemed reasonable and the analyst’s checks found nothing wrong. A stakeholder from another team saw the dashboard and immediately questioned it. They knew roughly how many sales the company had made that year. That outside estimate exposed a discrepancy without requiring the stakeholder to read SQL. The tolerance for a discrepancy also comes from the people using the result. A figure within 5% might look plausible while missing by enough to change the team’s decision.

When does a review actually change the wedding plan? A separate agent reviews the plan and its assumptions. The user can open a tagged challenge in a branch, and a decision can carry a note with an outside point of reference. These contributions have different jobs. The second agent can raise a question. An independent source or a person’s domain knowledge can supply the evidence to resolve it.

The fork has two outcomes. In the demonstration, a branch rejoins the main conversation and changes the plan. The join records where that revision came from. The companion exercise is to open another challenge and dismiss it when it has no bearing on the decision. Giving the review a route back into the analysis matters more than collecting verdicts. A judge could issue an opinion after every step and leave Galletta with precisely the savings plan he started with.

![The live review pane questions savings and guest counts while tracking a suggestion used on step three.](https://ai.engineer/api/talk-essay-media/af917be04c4e57c649c59f553b8f9fe342f46c4e02df7479823cdf9b2e48f481)

The reviewer raises questions about other wedding costs and the guest count. Its suggestion on monthly surplus has been used in the plan.

Which review findings reach the working plan?
The plan and its assumptions supply the reviewer's inputs. At the fork, the user decides whether a finding warrants a revision.
- Working plan — Contains the decisions and assumptions in use.
- Separate reviewer — Examines the plan for assumptions worth questioning.
- Tagged challenge — The user opens a branch to consider the question.
- Revised plan — The branch rejoins the main conversation with a change.
- Dismissed challenge — The user closes a finding that is immaterial to the decision.
- Working plan → Separate reviewer (Supplies plan and assumptions)
- Separate reviewer → Tagged challenge (Raises a question)
- Tagged challenge → Revised plan (Incorporate finding)
- Tagged challenge → Dismissed challenge (Dismiss finding)

### When does a task earn its own harness?

[1:33:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5603s)

The Q&A opens with a question about how many harnesses a team should build. Felisola Caso ties the decision to repeated failures, the number of users exposed to them and the cost of a wrong answer. A person doing an occasional, low-stakes task may be happy to steer an existing agent for a few more turns. A team of 50 credit analysts using the same tool to support key company decisions has a stronger reason to control variation. The five principles reflect the stakes in finance. Producing a stream of social videos, Galletta suggests, would put a different value on gathering evidence for each output.

Galletta’s warning about spending too long sharpening the axe comes with a practical test. Try a skill or a command and watch what the user still has to do by hand. If the same task repeatedly involves opening a fresh session, copying text from another chat and explaining the context again, that sequence is a candidate for software. His wedding provided both outcomes. Web search produced a usable shortlist of five Madrid photographers to compare. Financial planning by voice had him repeating the same conversation three times on a drive. He could point to the work a dedicated harness would save.

Another participant asks how to talk to leadership when mentioning AI seems to raise expectations of accuracy. Felisola Caso starts with a measurable outcome. If a new dashboard takes too many exchanges to validate, count the exchanges. If users keep opening internal tickets because a number looks wrong, track those tickets. These are proposed ways to evaluate the change. They give a team and its leaders something specific to look for once people start using the tool.

### A skill can call Python. Who guarantees it gets called?

[1:43:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6235s)

One participant already has a skill that asks questions and runs the numbers with a Python script. Why add a UI and a database? The script can perform repeatable arithmetic while the model still decides whether to call it and how to interpret the surrounding instructions. Distribute that skill to other users and their own agent settings may conflict with it. For a critical calculation, even an occasional decision to skip the script can defeat the purpose of writing it.

Felisola Caso had seen the cost in reviews of internally shared extensions. Teams would add prose for each unsupported case, down to what the agent should tell a user who tried it. Each new clause became another instruction the model had to interpret. Backend code can decide which operations are available and enforce those conditions directly. The reason to move a rule into the harness is that users need it to hold across runs.

There is still room for interpretation. A cash-flow figure can look odd because someone attributed revenue a month later. Recognizing that possibility is a different job from adding the entries once their dates are settled. The checks build confidence together, and that confidence has a limit. Galletta’s illustration is financial: even a 1% error can represent a lot of money. The percentage describes what a small error could cost, rather than a measured failure rate for the harness.

The final audience question asks where, exactly, a user can see trust in the UI. For someone puzzled by a total, the answer spans several places. They can follow the input to its source, inspect the code that ran and see how it was tested. Someone worried about paying on time may care more about opening the challenge to that incoming 20,000. Both are looking at the same plan. The interface earns its keep when the person about to make the decision can follow the particular doubt they have.

## Key takeaways

- Preserve qualifiers when turning conversation into inputs. A possible contribution, an approximate per-seat price and a lower bound on attendance each mean something different for the payment plan.
- Save and test the code for recurring arithmetic. Trace the implementation version alongside the input so a changed result can be explained.
- Give a review finding somewhere to go. A challenge that affects a payment assumption can revise the plan, and the user can dismiss one that has no bearing on the decision.
- Look for the recurring manual workaround before building a dedicated harness. Repeated transfers between chats and conflicting instructions across users identify specific work that software can take over.

## Slides

### Building a financial analyst harness

[0:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=12s) [The opening slide credits Francesco Galletta and Felippe Caso.](https://assets.ai.engineer/sites/aiecode2025/sha256/472d5c21d6deaa5b572b50e4d97659334c7bca0d53f84cdf0f8fc41aa0e1b64d.webp)

Stop Talking to Your Spreadsheets; Building a Financial Analyst Harness You Can Trust. Francesco Galletta · Felippe Caso.

### The speakers’ financial-services background

[3:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=206s) [The presenters distinguish their professional experience from their personal experiments.](https://assets.ai.engineer/sites/aiecode2025/sha256/baa581b1b82147a573f31f799334c79c21dc2e3a0edac455916757e44015d2c1.webp)

Data teams in financial services. Monzo Bank, Nubank, and a few more. Now pushing what AI can do there. Applied AI to make our lives easier.

### A personal workshop

[3:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=231s) [The presenters explain who they represent.](https://assets.ai.engineer/sites/aiecode2025/sha256/4df360e9e61431dfb2408da1d3458432dccbeef737eec4748086cbd44b7a6868.webp)

We are not representing Monzo today.

### Why speed is a poor proxy for trust

[5:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=356s) [The slide connects reliable collaboration to decisions involving money.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c0d546e896b17bf952df88974c66c965d7280b9e6a17dfbb117223081839a1a.webp)

The gap is no longer speed. It is how the work meets us. So we can trust agents with the highest stakes we have: money.

### Familiar building blocks

[7:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=432s) [Handwritten notes accompany five familiar engineering techniques.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ff2ef1f780316bda2a04280a43e3d14692cf5e255010dab2ee1b4a5a7bd0781.webp)

None of this is new. Unit tests. Planning with your agent. Evals. Traces and observability. Loops and graphs.

### Five principles for a financial harness

[10:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=604s) [Five colored bands name the parts of the harness and give a one-line purpose for each.](https://assets.ai.engineer/sites/aiecode2025/sha256/2743020862903845cc0372b1c277f2881f33a1f9010d38c4d67114c5f42774f7.webp)

Trust. 5 Verification: A reviewer challenges the plan. 4 Accountability: Ask only where a person must decide. 3 Evidence: Every number can be followed back. 2 Consistency: Numbers come only from tested code. 1 Shared Domain: Agree the plan before any number.

### The example is a wedding budget

[12:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=732s) [A ring illustrates the real planning question behind the workshop.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0e53a350c313718d9113b3be8fe401918766440b08f2bfda1001aed88fc5e09.webp)

My wedding. Figuring out the finances around it, to be exact.

### Inside the workshop repository

[15:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=900s) [Four folders separate the exercise, wedding example, reusable harness and a participant’s plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa84c09016abf5120207d8dd97b807b22d7563ab39c7d45ef858d1594e748cf3.webp)

workshop/README.md — today’s guide: start here. examples/wedding/ — today’s example. harness/ — the harness: a core and five layers. my/ — your own plan, for later.

### Define the task together

[20:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1248s) [The first layer works backward from the user’s goal.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d684c4ef5c63277efece521780e98ae70882a96f0c743e338530b9cef19b239.webp)

What is obvious to you is not what the model learned. Start from the goal. Work backwards, together.

### The wedding plan as a decision graph

[30:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1820s) [The actual workshop app shows linked calculations and the accepted plan with seven steps and eight inputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/72f4c0d631ef57f68a456f8ab61b1a68d71499eac65e002b3ee64e81cea365bc.webp)

Financial Advisor Harness. All 6 built. Whole plan. Review. Plan accepted. 7 steps, 8 inputs. Estimate total wedding cost. Work out the three payment… Work out monthly surplus. Project wedding fund balance at…

### Where each input came from

[33:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1988s) [The workshop labels information supplied by the user, a proposal and an open decision.](https://assets.ai.engineer/sites/aiecode2025/sha256/d604abc5b88d01e08d088794c5215d81bcc162a3b31e70435ef0c6ff8e36121b.webp)

Click step 3. Each item says where it came from: your words, proposed. ● is still open. You can tell how the plan is being built, and where decisions are coming from.

### What changes after the first payment?

[35:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2111s) [The slide distinguishes a change to the plan from recomputing its numbers.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd6c880f663edcc42ed542d103b68060b3d7b9bc7fa7cd09a5a19497721be428.webp)

I already made the 1st payment, do we need to change the plan? A new trunk node. Step 2: Stale · rebuild. Only your words change the plan. Its numbers wait for a rebuild.

### Put the arithmetic in code

[43:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2633s) [The presenter assigns tool selection to the model and the calculation to code.](https://assets.ai.engineer/sites/aiecode2025/sha256/069fe5be74ee618c6250903a5a6b29ed74001fd6d509311248f185141c80bab6.webp)

Write the calculation once. Run it the same way every time. The model picks the tool. It never does the arithmetic.

### Six tested calculations

[44:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2693s) [Six boxes list the calculations used in the wedding plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/80d711f242704b2a1209f11e0ea58021a6959525a19472183dd1824d50877c7e.webp)

Total cost. Payment schedule. Monthly surplus. Monthly saving needed. Fund balance forecast. Cost per guest. Each one is code, with tests.

### Inspect and run a node’s tests

[47:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2847s) [The exercise opens the first node’s implementation and its test results.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a40509ff83dc0cf31b231cc4e7d2621a11c1b3e1cb900ede69507c3dd4df9d5.webp)

Click step 1, then Code and tests, then Run tests now. The code, its tests: 4 tests, all passing. Every number comes from code you can read, tested now.

### Keep evidence for each number

[56:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3362s) [The third layer adds a trail back to the calculation.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a47181cdf288677ea9d11f0e8b43de0a8d8ebaa9013ac70ecf6b02afa676641.webp)

3 Evidence. Every number can be followed back. Tested code (and everything else) leaves a trail we can follow.

### When a changed template deserves a closer look

[1:01:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3660s) [The presenters distinguish where receipts matter most.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf5c37d88f9a9aaf1ef0211c07cd9d756131b17e1be6ab272ad0bb643ea589fb.webp)

“I ran the numbers, but the template changed recently.” You would ask for a closer look. Show the receipts for what matters, not everything.

### Trace the 31,000 result

[1:02:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3773s) [The exercise opens the evidence behind an underlined number in the wedding example.](https://assets.ai.engineer/sites/aiecode2025/sha256/53cf75c2bd12a60e9dbac606178446fda54fcaa41c1f283003d83e55a67b4a99.webp)

Click the underlined 31,000. Step 1 opens: your inputs, Came out 31,000, after test run N. Any number leads back to its step, its inputs and its tests.

### Who decides?

[1:06:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4001s) [The fourth layer puts a person in charge of consequential decisions.](https://assets.ai.engineer/sites/aiecode2025/sha256/2da7ea84ca919d0b1766cd447a2012896db29f96959b4dbb99da226a41569f52.webp)

4 Accountability. Ask only where a person must decide. The evidence is what you decide with.

### When the harness should ask

[1:13:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4428s) [Three rules separate conflicting interpretations, labeled assumptions and plain calculations.](https://assets.ai.engineer/sites/aiecode2025/sha256/a606b7286b412e21f0e931624a22d34b9d90bc004594bd0364c77ba7bb329e6d.webp)

Few gates. High signal. Two plausible readings that conflict: stop and ask. An assumption: run it, and mark it. Arithmetic: never ask.

### The app records changes to a payment date

[1:14:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4490s) [The app’s decision graph sits next to a note about a payment and the balance forecast.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce1d92c05d776b1199aadc8ff4161317a773c30de232145eb6af81c9f45c2404.webp)

Financial Advisor Harness. Plan accepted. 7 steps, 8 inputs. 20k lands on the same day as payment 3. Project wedding fund balance at each payment date (running balance, cash flow forecast style).

### A side question need not change the plan

[1:18:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4728s) [A branch answers what a sinking fund is without changing the agreed plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab37843ee43e1d8024f4a75140ed756001c4e978ac73fa6e8175f44ae978b65e.webp)

Click step 5, then New branch. Type: What is a sinking fund? An answer. Close branch: it dims, no change. Nothing to decide, so nothing is asked and nothing changes.

### Have a reviewer challenge the plan

[1:20:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4831s) [The fifth layer asks a reviewer to raise challenges for the user to accept or dismiss.](https://assets.ai.engineer/sites/aiecode2025/sha256/e217e5882b0f5aa7206f61831d4dd66b79dcf47a76dd43edcfd3e0f649d6474d.webp)

5 Verification. A reviewer challenges the plan. Each challenge is yours to use or dismiss.

### What expert review contributes

[1:23:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4980s) [The workshop describes how an experienced reviewer prioritizes checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/83c745d236622c63b311bc12e2a599206df2c3516f90461df6497e31f11841d6.webp)

Experts do not check every cell. They know where it usually goes wrong. And roughly what the answer should look like.

### Reviewer questions alongside the working plan

[1:30:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5431s) [The live review pane questions savings and guest counts while tracking a suggestion used on step three.](https://assets.ai.engineer/sites/aiecode2025/sha256/af917be04c4e57c649c59f553b8f9fe342f46c4e02df7479823cdf9b2e48f481.webp)

Other wedding costs from the same savings? Does the 5k count toward the total? Surplus rests on a 5k figure from memory. Reviewer’s suggestion used on step 3.

### Dismiss a reviewer’s suggestion

[1:32:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5541s) [Dismissing a challenge preserves the agreed plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/39ae9ea208b4ab76c56197962df47d57fb645f05256ba731aeb44a540dc67810.webp)

Open another. Click Dismiss. It dims: dismissed. The plan stays as it was. Each challenge is yours to use or dismiss.

### Show how the numbers ran

[1:48:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6508s) [The closing slide links the workshop’s design to the detailed assumptions of finance.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3a30ae80895646be07fd810101e3a80c68b8fddc861786bdf07088b7d53f81b.webp)

Show how the numbers ran. Nothing generic. Finance is an ocean of nuance. ask any accountant. Design to increase trust. Never assume it.

### The final point

[1:48:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6538s) [The presenters close by returning to the original question of trust.](https://assets.ai.engineer/sites/aiecode2025/sha256/cce571db4659c03226f6d35e08fbb7b6a1de78123b2a3c7f14e74c23f691eb32.webp)

Better models still have to earn trust.

## Full transcript

[0:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=0s) Morning. [music]

[0:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=14s) Thanks everyone for being for being here

[0:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=16s) and welcome to AIE. I guess first first

[0:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=19s) session for everyone and for us as well.

[0:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=22s) So h we wanted to introduce ourselves

[0:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=25s) and tell you about what we're going to

[0:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=27s) discuss. We think we need to stop

[0:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=29s) talking with spreadsheets. So, this is

[0:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=32s) all going to be about building a

[0:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=34s) financial analyst harness that you can

[0:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=36s) actually trust.

[0:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=37s) We are Franchesco Filipe. A little bit

[0:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=39s) of a quick introduction about me. I'm

[0:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=42s) originally from Venezuela. I lived in

[0:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=44s) the UK for about seven years and

[0:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=46s) recently moved to Madrid. H I work at

[0:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=49s) data and I work at a bank as well with

[0:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=52s) Felipe. Erh. I like just finding the

[0:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=55s) best eeries in any city that I visit. So

[0:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=57s) if you have any recommendations for New

[0:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=59s) York, I'm more than happy to take them

[1:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=61s) and just travel around. And the reason

[1:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=63s) my hobbies sound so vanilla is because I

[1:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=66s) just spend most of my time building

[1:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=68s) stuff and nerding out at the expense of

[1:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=70s) my personal time. So that's how you end

[1:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=73s) up with a list like that. Uh and then

[1:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=76s) nice. All right. Um happy to meet

[1:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=78s) everyone. Um also quick intro. I'm

[1:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=80s) Philipe. I'm originally Brazilian. Um, I

[1:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=83s) also moved to the UK. It's been a year

[1:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=85s) and a half now. So, it's been a fun

[1:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=88s) journey. Um, life turned pretty much

[1:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=89s) upside down. But, yeah, it's good to be

[1:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=92s) it's good to be there. Um, I like to

[1:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=94s) tell a story that before going to to

[1:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=97s) decide whether I wanted to do it in

[1:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=98s) university, I used to play piano for

[1:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=100s) like literally seven hours a day. Um,

[1:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=102s) and the natural decision there was to

[1:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=104s) then study electrical engineering

[1:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=106s) because who wouldn't, right? Um, I ended

[1:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=109s) up at some point landing in in data

[1:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=112s) because someone showed me something that

[1:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=113s) at the time called uh Google Data

[1:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=115s) Studio. Now it's Looker Studio, but now

[1:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=116s) they went back. I don't I don't even

[1:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=118s) know what's going on. Um, and then my

[2:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=120s) first dashboard looked like a papyrus of

[2:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=122s) how vertical it was. You had to scroll

[2:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=124s) scroll scroll scroll scroll down to see

[2:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=125s) something. Um, but yeah, the the rest is

[2:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=128s) history. So I worked at a lot of um

[2:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=130s) startups and small teams and scaleups.

[2:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=133s) Um and then eventually went to a big

[2:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=135s) company at New Bank. Maybe some of you

[2:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=138s) do know um and then before going back to

[2:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=140s) to Monzo which is I would say a middle

[2:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=143s) turn between a scale up and a big

[2:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=145s) enterprise company but it's growing uh

[2:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=146s) so fast for now. Um and yeah most of my

[2:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=149s) time currently has been working at Monzo

[2:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=152s) with building internal AIdriven

[2:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=154s) solutions for internal productivity

[2:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=156s) primarily for the data team. So, we

[2:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=159s) basically want to get our data team

[2:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=160s) better by um building with AI. Um and

[2:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=163s) yeah, when I'm not talking to robots and

[2:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=165s) agents, uh you can probably find me

[2:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=167s) climbing somewhere um either in nature

[2:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=169s) or in a gym. Um and some starting like

[2:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=175s) points of why are we bringing this for

[2:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=177s) you today? I think um both Franchesco

[3:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=180s) and I worked in data teams in financial

[3:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=182s) services. So again, Monzo, New Bank and

[3:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=184s) a few other companies in the in the

[3:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=185s) sector. Um, we're now pushing what AI

[3:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=188s) can do there. Our team at Monzo has been

[3:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=190s) um, working a lot with with adopting AI

[3:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=193s) from an impact perspective. We don't

[3:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=196s) want to be like token maxing. We don't

[3:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=197s) want to be spending a lot of money if

[3:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=199s) that doesn't bring us like concrete

[3:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=201s) business value. Um, so the adoption is

[3:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=203s) very very deliberate and we've been

[3:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=205s) seeing very good results on that. And on

[3:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=208s) the flip side, this is something also

[3:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=210s) something we've been we've been doing,

[3:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=211s) Franchesco particularly is applying AI

[3:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=213s) to make our lives easier. Um and again

[3:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=216s) not through the approach of I'm going to

[3:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=218s) set up an open client instance nothing

[3:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=220s) against it um and just give it

[3:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=222s) connections to everything but actually

[3:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=224s) what are the things that I care about in

[3:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=227s) my life that I can um get the help of AI

[3:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=230s) to to u make easier. So this is why we

[3:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=234s) say we're not representing Monzo um in

[3:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=237s) this example because we're going to use

[3:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=238s) the knowledge we bring from Monzo but uh

[4:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=241s) build with you an example where we can

[4:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=244s) um actually see that in practice without

[4:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=245s) having to discuss a lot of domain

[4:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=247s) knowledge. Um but if you are curious

[4:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=250s) about the things we are building at

[4:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=251s) Monzo we have a breakout session

[4:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=253s) tomorrow um where we're going to talk

[4:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=255s) about agent chip which is our inhouse

[4:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=257s) remote agent capability. Um so with that

[4:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=261s) Franchesco back to you.

[4:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=262s) Cool. So, as a reminder, today the goal

[4:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=265s) is trust

[4:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=269s) and it may seem a little bit

[4:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=270s) counterintuitive because it feels very

[4:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=272s) wishy-washy and this is a workshop and

[4:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=273s) you're supposed to be coding really hard

[4:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=275s) for two hours. H but you're going to see

[4:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=277s) we're going to give it a little bit of a

[4:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=279s) spin. Uh because I think we're past the

[4:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=282s) point of having to spend hours like

[4:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=284s) grinding through your keyboard uh to get

[4:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=286s) something that you can trust. So, let's

[4:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=289s) start with the domain. Finance is

[4:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=291s) tailored.

[4:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=293s) You have your numbers. You have your

[4:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=295s) strategy.

[4:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=297s) Sometimes you don't really have a

[4:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=299s) strategy. And there is the knowledge

[5:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=301s) that you have on finance, on your

[5:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=303s) personal situation and so on or

[5:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=305s) generally misconceptions. H when we talk

[5:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=308s) about a financial analyst, there is a

[5:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=310s) very specific job description. We are

[5:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=313s) being a little bit generous with that

[5:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=314s) job description. So it's going all the

[5:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=316s) way from doing research for investments,

[5:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=319s) investment firms, banks and so on all

[5:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=321s) the way to like personal use cases as

[5:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=324s) well. But finance is finance, right? So

[5:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=327s) right now the gap on producing

[5:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=331s) something, getting hold of your

[5:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=332s) finances, doing an analysis is not

[5:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=334s) really speed. H I'm pretty sure some of

[5:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=337s) you may have some background tasks

[5:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=339s) running with like claude or chpt getting

[5:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=342s) some stuff done. So that's not the

[5:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=344s) bottleneck. H we believe that

[5:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=348s) based on the experience we had over the

[5:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=350s) last few months, the bottleneck now is

[5:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=353s) how it connects with us. Um

[5:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=357s) so trust is the thing we cannot take for

[6:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=361s) granted and transparently because we're

[6:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=363s) talking about money specifically. Erh

[6:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=366s) this is one of the highest stakes

[6:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=368s) topics. uh we need to invest in making

[6:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=371s) sure that there is trust built into the

[6:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=372s) system and transparently if money is not

[6:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=377s) high stakes enough for you and you can

[6:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=379s) like oneshot your money problems your

[6:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=381s) budgeting your financing everything else

[6:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=383s) h find me after the workshop I will buy

[6:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=386s) you in into some investment ideas so

[6:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=389s) yeah let's chat later please

[6:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=393s) um so where everything starts is from

[6:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=396s) the goal which is earning the trust So

[6:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=399s) then you can truly delegate the boring

[6:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=401s) work to multiple agents, your swarms,

[6:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=404s) your factories, whatever. But how you do

[6:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=406s) trust the results that you're getting.

[6:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=409s) Erh, I'm going to namerop a few things

[6:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=413s) because honestly none of them are new.

[6:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=416s) You're probably already grinding

[6:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=417s) thinking, well, you buy trust by h

[7:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=421s) setting up traces or doing loops or any

[7:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=424s) of the other keywords we have. So unit

[7:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=426s) testing that gets old planning with your

[7:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=429s) agent we know that that's good. Uh doing

[7:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=431s) evals for sure traces and observability

[7:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=435s) loops and graphs like I can keep naming

[7:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=437s) components of how you think you can

[7:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=440s) start building trust but they are

[7:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=443s) basically components. H what we want to

[7:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=446s) discuss today is actually the design

[7:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=448s) principles on how to achieve that trust.

[7:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=451s) So building a hardness that actually

[7:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=453s) works for money matters

[7:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=456s) requires an in immense amount of trust.

[7:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=460s) So we're not going to be discussing

[7:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=462s) specific technologies. We're not going

[7:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=463s) to read you the readme of a particular

[7:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=466s) product. H and we will assume you know

[7:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=468s) some of the concepts that have been

[7:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=470s) listed so far. Um and mainly this is

[7:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=474s) going to be a slightly different

[7:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=476s) workshop because we want you to feel how

[7:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=479s) the principles work in reality.

[8:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=482s) So we have built a hardness today

[8:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=486s) around a very simple example and we want

[8:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=488s) you to try it out but don't focus on oh

[8:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=492s) if I type this then I get the response

[8:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=494s) so much quicker and the UI is so slick.

[8:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=496s) it spoiler it won't be h but more

[8:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=499s) importantly we want you to feel how it

[8:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=502s) may feel like having trust in an agent

[8:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=505s) that has been built with a very specific

[8:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=507s) hardness uh so again this is just one

[8:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=510s) very simple implementation an example h

[8:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=513s) you can roast me later if you think this

[8:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=515s) is too top level but we can discuss that

[8:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=518s) later so trust and I'm probably going to

[8:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=522s) run out of times I can say trust without

[8:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=524s) going insane but we think you need to

[8:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=526s) build it from the ground up, right? So

[8:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=528s) you have a shared domain, the whole

[8:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=530s) thing about a plan, making sure that you

[8:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=533s) and the model are talking in the same

[8:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=536s) terms. And it sounds more trivial than

[8:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=538s) it actually is. Then consistency, and

[9:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=542s) pretty sure everyone in the room will be

[9:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=543s) on the same page, but it's making sure

[9:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=545s) that the numbers come from a correct

[9:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=548s) place. So rather than a language model

[9:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=551s) trying to process a lot of arithmetic,

[9:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=553s) it will do some tool calling. It will

[9:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=556s) write us some scripts. It will try to

[9:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=557s) come up with a way that the results are

[9:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=560s) consistent. We also want evidence where

[9:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=564s) basically show the receipts, right? You

[9:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=567s) can follow back and trace back the

[9:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=569s) decisions and what led to them.

[9:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=572s) Accountability is probably a very spicy

[9:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=575s) one and maybe the mo most abstract of

[9:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=579s) them, but ultimately even if you're

[9:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=582s) running a 100 agents, the results of

[9:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=584s) what they do, you're still accountable

[9:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=587s) for them. So that's something we take

[9:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=589s) very seriously at work. And we think

[9:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=591s) it's a mindset shift that when people

[9:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=595s) make it, they start building systems

[9:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=597s) that evolve with them and not black

[10:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=600s) boxes that they just throw problems at

[10:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=602s) hoping that something sticks.

[10:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=605s) And finally, verification. So we want to

[10:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=608s) make sure that there is a separate point

[10:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=610s) of view, another perspective. Erh, I

[10:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=614s) would say I'm guilty of thinking I have

[10:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=616s) the best idea in the world probably like

[10:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=618s) three times a year and then only to have

[10:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=620s) Filipe tell me please don't implement

[10:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=623s) that please please please and I I Oops

[10:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=626s) one second and I would just compliment

[10:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=629s) on on this bit that I think two things

[10:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=631s) the first one is we have a very generous

[10:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=633s) window here to to work together so um it

[10:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=636s) will be very nice to learn from your

[10:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=638s) folks's experience as well as we build

[10:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=640s) through this principles so if this

[10:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=643s) resonates with any work you've done or

[10:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=645s) any particularly any challenges you've

[10:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=647s) faced um when working with agents feel

[10:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=649s) absolutely free to to raise that and

[10:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=651s) then we can we can build it uh together

[10:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=654s) um and just ding uh doubling down on

[10:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=657s) something that Franchesco said this

[10:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=658s) we're not saying this is the model you

[11:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=661s) won't say us going to to the big stage

[11:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=663s) and saying forget everything you know

[11:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=666s) this is how to do it this is a model a

[11:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=669s) way to approach it uh that has been

[11:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=671s) working for us in a number of use cases.

[11:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=673s) I'll share a few examples at Monzo later

[11:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=675s) and Francisco has a few other examples

[11:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=677s) as well. Um but yeah, again there's not

[11:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=680s) a single silver bullet to to solve every

[11:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=683s) single problem we face. And also fe feel

[11:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=686s) free to challenge us if you think

[11:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=688s) actually trust overrated. Just throw a

[11:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=692s) 100 agents at a problem eventually will

[11:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=694s) get solved. happy to hear the success

[11:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=696s) stories but from repetition from

[11:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=699s) experience and a lot of different use

[11:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=700s) cases at work and outside we do think

[11:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=703s) you need to build a system that gives

[11:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=705s) you the confidence that things are going

[11:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=707s) in the right direction and not just that

[11:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=710s) things look correct

[11:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=713s) so yeah trust happy days um

[11:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=717s) speaking of trust I'm here to confess

[12:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=721s) that I have an ulterior motive with

[12:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=723s) today's workshop Uh so breaking your

[12:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=725s) trust immediately. What is it? Less than

[12:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=727s) five minutes.

[12:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=729s) You are here to help me Francesco

[12:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=733s) to plan my wedding.

[12:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=736s) So nobody's super excited but you will

[12:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=738s) get excited. Erh specifically just to

[12:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=742s) figure out some some stuff around the

[12:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=744s) finances of paying for a wedding. Um and

[12:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=747s) because it's a conference I want to make

[12:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=749s) it as screeny as possible. So show of

[12:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=751s) hands. Uh, who has had a wedding? Who's

[12:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=755s) gotten married?

[12:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=757s) Okay, awesome. Number of guests under

[12:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=760s) 50.

[12:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=762s) Yeah. Okay. No, still show of hands.

[12:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=764s) It's still like cringey mode. Okay,

[12:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=766s) cool.

[12:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=767s) About 100.

[12:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=769s) Yeah. Nice.

[12:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=771s) About 150.

[12:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=773s) Okay.

[12:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=774s) Nice.

[12:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=776s) 200.

[12:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=777s) More. All right.

[12:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=778s) Yeah. Yeah. Big wedding. Nice.

[13:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=781s) This is probably going to resonate, but

[13:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=783s) it sounds like it's going to be a very

[13:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=786s) simple calculation, very simple process,

[13:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=788s) and the wedding is going to go by nicely

[13:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=790s) and smoothly. And for the people who got

[13:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=792s) married, probably you know that is not

[13:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=794s) the case. Um, and even running the

[13:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=797s) calculations and figuring out how things

[13:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=800s) got so expensive so quickly. It's not

[13:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=803s) trivial. This is a little bit of

[13:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=805s) storytelling and hopefully my girlfriend

[13:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=807s) doesn't fiance doesn't see this.

[13:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=810s) But we were doing some calculations and

[13:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=812s) she was like, "No, yeah, but the cost

[13:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=814s) per like dinner is this much." And I'm

[13:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=817s) like, "Yeah, and you have another two

[13:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=819s) variable costs on top of that. So that's

[13:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=822s) not the number." And anyway, budgeting

[13:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=824s) for this has been a challenge to

[13:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=826s) underelling

[13:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=828s) it a little bit. I wouldn't recommend

[13:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=830s) doing it, but you know, I'm already on

[13:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=832s) the train. Erh, so today you're going to

[13:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=836s) give me a hand with the wedding unpaid

[13:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=839s) labor or you can do your own plan. Uh,

[14:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=843s) so we put a few things in a repo that

[14:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=845s) you're going to get access to. So you if

[14:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=847s) you want to pick your own adventure, do

[14:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=849s) that. Wouldn't recommend because I do

[14:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=851s) need help, but whatever you choose.

[14:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=855s) So we'll give you a few minutes uh for

[14:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=857s) you to set this up.

[14:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=861s) just either scan the QR code or go to my

[14:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=865s) repo and there will be some instructions

[14:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=868s) on how to set it up.

[14:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=873s) We did not plan a background music like

[14:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=875s) an elevator music for this moment.

[14:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=878s) Yeah, let us know if you have any any

[14:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=880s) trouble setting up.

[14:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=884s) Also, if anyone sings, it would be a

[14:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=886s) good time to

[14:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=887s) karaoke live.

[14:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=891s) By the way, you don't need to execute

[14:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=893s) anything just yet. Just make sure you

[14:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=894s) have the repo in your preferred um setup

[14:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=897s) and then we'll guide you through the the

[15:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=900s) execution.

[15:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=901s) Cool. Uh so you just cloned a virus. No,

[15:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=906s) just kidding. But a lot of coded stuff

[15:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=909s) in there. The structure does make sense.

[15:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=912s) So we have a guide on how we start for

[15:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=915s) today's workshop. H we have the example

[15:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=918s) most important thing of the talk today.

[15:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=920s) So helping me figure out my wedding. H

[15:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=923s) also the harness itself the thing we

[15:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=925s) build to show you how the principles are

[15:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=927s) applied. And then finally there is a

[15:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=930s) folder where everything you build that

[15:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=932s) drifts away from what we're doing is

[15:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=934s) going to get stored in. So again, it's a

[15:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=937s) little bit of a pick your own adventure

[15:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=939s) uh with the added risk that we may not

[15:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=942s) be able to help you if you get stuck,

[15:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=944s) but you can um and everything you do

[15:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=947s) today, you can take it home and you can

[15:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=949s) check what you did and what you like,

[15:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=951s) what you didn't

[15:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=953s) and particularly I was going to say

[15:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=954s) particularly the harness itself talking

[15:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=956s) to to the jokes we were just making

[15:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=958s) about like this is as far as this was

[15:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=959s) built. Um again the the principles we're

[16:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=962s) sharing here the principles that are

[16:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=964s) built here. We were discussing just

[16:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=966s) yesterday is like uh this can feel like

[16:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=969s) we are showing you a table to teach you

[16:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=972s) how to build a table if that makes

[16:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=974s) sense. Uh but we really do we really do

[16:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=976s) believe that the again building is not

[16:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=980s) the hard thing. The hard thing is

[16:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=981s) knowing how we want to approach the

[16:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=983s) problem. So that's why we're bringing it

[16:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=985s) like this. But we're going to open the

[16:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=987s) open the look under the hood a bit. um

[16:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=990s) at times just to show you what's going

[16:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=991s) on um behind the the principles we're

[16:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=994s) applying

[16:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=996s) and pretty much like Filipe was saying

[16:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=999s) it can feel akin to saying like hey

[16:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1003s) Claude please build me a $1 million

[16:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1006s) company

[16:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1008s) if you don't know exactly what are the

[16:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1010s) principles and the decisions you're

[16:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1012s) making in order to make something happen

[16:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1014s) doesn't matter how many times do you try

[16:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1016s) you're likely going to be just leaving

[16:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1018s) yourself up for the randomness of

[17:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1022s) something good happening by accident.

[17:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1025s) So we want to anchor the principles

[17:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1028s) again because they build on top of each

[17:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1030s) other and we think there is a nice

[17:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1032s) sequence that kind of makes logical

[17:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1035s) sense. So building that shared domain

[17:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1038s) plan mode however you want to call it

[17:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1040s) then setting up consistency making sure

[17:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1043s) that whatever can run or should run

[17:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1045s) reliably does. Then having the evidence

[17:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1048s) to back up where the decisions are

[17:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1050s) coming from, what information is coming

[17:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1052s) through, if things are being tested and

[17:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1054s) so on and putting that front and center.

[17:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1057s) Then the accountability, how do you keep

[17:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1060s) a good enough grasp of what the machine

[17:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1064s) is doing? You don't want to go full

[17:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1066s) blackbox mode because then your

[17:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1068s) accountability is gone. And that has

[17:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1069s) happened to many of us in this room, I'm

[17:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1071s) sure.

[17:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1072s) But the other side is also not very

[17:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1076s) realistic where oh for every change you

[17:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1078s) make send me the diff. I'm going to

[18:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1080s) review it line by line for sure. Nobody

[18:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1083s) has done that ever. Uh and then finally

[18:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1086s) verification. So in data and in finance

[18:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1090s) erh you often need to have a separate

[18:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1092s) point of reference a counterfactual that

[18:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1095s) you can check against. H and ignoring

[18:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1097s) that part of the equation just leads to

[18:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1100s) you running things on your own biases.

[18:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1102s) So your results are always going to be

[18:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1104s) skewed. There's some short-sightedness

[18:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1106s) that plays into that. So these

[18:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1108s) principles are just to try to

[18:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1111s) funnel our efforts into where is most

[18:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1114s) useful to get something that we can

[18:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1116s) actually trust.

[18:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1118s) Um

[18:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1120s) heads up again something will break

[18:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1122s) today surely. Just call us to you. we

[18:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1124s) will try to fix it with you. Uh some of

[18:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1127s) it is also overengineered. This is not a

[18:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1130s) session on good UI UX design. Okay. So

[18:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1134s) this is just to show you how it could

[18:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1136s) feel, show you the potential. Uh by no

[18:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1139s) means we're building a product

[19:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1143s) and yeah just raise your hand.

[19:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1144s) Something's not working out. You're a

[19:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1146s) little bit confused. Raise your hand and

[19:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1148s) we're going to float around the room.

[19:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1150s) Um,

[19:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1152s) so you may want to check if this works,

[19:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1155s) try and run that command.

[19:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1162s) Yeah. And I think at this point we can

[19:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1164s) just go around if things are not working

[19:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1166s) for anyone and give everyone five

[19:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1168s) minutes to get going with this bit. You

[19:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1171s) should see the UI opening if everything

[19:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1173s) goes successful with the setup works

[19:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1176s) success in the terminal as well.

[19:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1183s) Cool. And that second command there will

[19:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1185s) open

[19:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1187s) a very well-designed UI

[19:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1190s) for you to do some stuff. Here is the

[19:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1192s) example we're going to show you because

[19:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1194s) this is kind of like layered, you know,

[19:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1196s) like an onion or an ogre. H we're going

[19:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1199s) to show you an example hardness being

[20:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1202s) run on an example use case. Um, if you

[20:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1206s) want to run that command at the bottom,

[20:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1207s) you will see kind of like a more fleshed

[20:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1209s) out example. But if you stay on the just

[20:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1212s) running the second thing and just use

[20:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1214s) the UI, h it may feel weird that I'm

[20:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1217s) asking you to use an UI to see how a

[20:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1219s) hardness behaves. But that's the point.

[20:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1222s) The point is to show you how it feels in

[20:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1224s) should feel in the real world. Um,

[20:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1228s) should I leave it here for a moment?

[20:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1229s) Yeah, I think so.

[20:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1230s) Yeah. Okay. Okay. I hear less typing.

[20:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1233s) I'm going to assume you've typed

[20:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1235s) whichever command you wanted to type.

[20:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1238s) So let's go through the actual steps and

[20:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1241s) the principles of the hardness and the

[20:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1243s) share domain is the first one. Can I see

[20:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1246s) little show of hands of who opened the

[20:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1248s) UI and hasn't done anything else with

[20:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1250s) it.

[20:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1253s) Cool, cool, cool, cool, cool. All right.

[20:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1255s) So um long story short, you have a chat

[20:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1258s) window on your uh right right hand side

[21:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1261s) and it has nothing essentially. Uh if

[21:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1264s) you want to follow through an example of

[21:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1266s) a financial worry, some analysis you

[21:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1268s) want to do and that's the place to start

[21:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1271s) and basically we want to walk from the

[21:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1273s) goal we want to achieve and just try our

[21:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1276s) best to give an explanation. You're

[21:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1279s) probably doing this daily with your

[21:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1281s) agents anyway, but focus on

[21:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1284s) understanding that you're likely not

[21:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1286s) going to have the same context as the

[21:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1288s) agent. Um, wait, can I go back and

[21:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1291s) forward just in case? Okay. Yeah. Demo.

[21:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1294s) Um, so I wanted to explain something and

[21:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1297s) I think most people don't realize

[21:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1301s) the fact that what the model knows is

[21:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1306s) not necessarily what you think. So

[21:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1309s) unless you're very rich and you have a

[21:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1310s) lot of models trained for you, h you're

[21:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1314s) like likely your definition of certain

[21:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1316s) things are not going to be the exact as

[21:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1318s) what the majority of the population

[22:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1321s) defines. So if you're looking up for a

[22:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1323s) certain term, the agent may the model

[22:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1326s) may be smart enough to know well this

[22:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1328s) sits in this space but it's not

[22:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1331s) necessarily the thing that you meant. So

[22:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1333s) going back and forth with your uh model

[22:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1337s) really makes a difference. And the

[22:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1338s) hardness is supposed to encourage you if

[22:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1340s) you're trying it out yourself to dig

[22:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1342s) deeper to challenge some of your

[22:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1344s) assumptions to ask you questions you may

[22:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1347s) not knew you needed to to ask yourself.

[22:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1350s) So if you want to do Yeah.

[22:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1357s) So let me do the same thing that you

[22:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1361s) folks are doing, but I'm going to do

[22:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1365s) harness example.

[22:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1373s) Let me go back here for a second.

[22:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1377s) Just make sure I have everything

[23:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1395s) Is anyone completely lost or

[23:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1399s) are good? Okay, maybe could you give him

[23:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1402s) a hand

[23:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1403s) while I I I run everything. And for the

[23:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1406s) folks that arrived later, I will give

[23:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1410s) you the QR code as well

[23:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1414s) for the

[23:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1417s) so if anyone lost that you can you can

[23:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1420s) pick it up again.

[23:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1423s) Okay. Um let me show you two things

[23:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1426s) while I while we we let you go with it.

[23:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1429s) So this is what you should be seeing

[23:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1431s) when you run with a wedding example,

[23:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1434s) right? Um we're we're going to explain

[23:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1437s) in a bit um what's going on here. If you

[24:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1440s) are choosing just to show you as well,

[24:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1442s) if you are choosing to uh pick your own

[24:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1445s) pick your own direction, um you're going

[24:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1448s) to see something

[24:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1451s) like this where it's empty. And then

[24:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1454s) what you can do is

[24:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1456s) as Francesa is going to run through the

[24:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1459s) wedding example, you can say something

[24:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1460s) you want to to use a harness for as

[24:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1462s) well. So things like I don't know plan

[24:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1464s) your own wedding, plan a trip, um plan a

[24:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1467s) new investment in your company or in

[24:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1470s) your personal life, doesn't matter. Um

[24:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1472s) anything that relates to financial the

[24:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1475s) financial life that we can um implement

[24:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1477s) the same principles. If you ever need uh

[24:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1480s) again just opening the hood a bit. If

[24:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1482s) you ever need to switch between one and

[24:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1484s) another like the the wedding example or

[24:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1487s) your own um you can just delete the in

[24:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1490s) the my and var folder you're going to

[24:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1493s) have a database that's basically what's

[24:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1495s) storing this. So you can just delete it.

[24:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1499s) Oops.

[25:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1503s) And then once you open the the UI again

[25:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1506s) with the previous command, you're going

[25:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1509s) to get the example.

[25:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1510s) Do you want to open the clean one?

[25:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1512s) Yes, I can. Sure.

[25:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1513s) Yeah. So, appreciating that you're

[25:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1515s) literally you're literally being given

[25:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1518s) uh a blank canvas. We want to

[25:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1520s) disambiguate a little bit. So, if you

[25:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1523s) open the wedding example, that's a work

[25:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1525s) through example that has all the

[25:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1526s) elements. But let's say this is where

[25:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1529s) we're starting, right? And I have a

[25:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1531s) problem. Do you have a dictation thing

[25:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1533s) there?

[25:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1534s) Um, I do. Um,

[25:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1536s) okay.

[25:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1537s) Let's see if this is working. Wait a

[25:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1539s) second.

[25:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1540s) So, basically on that like chat window

[25:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1543s) on the uh right hand side, you are it's

[25:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1548s) a familiar interface, right? So, how

[25:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1550s) would you go about getting some help

[25:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1551s) from an AI agent for whatever specific

[25:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1554s) problem? Just start with a chat. You

[25:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1555s) start with a a problem and you start to

[25:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1557s) describe what you want to achieve. So,

[26:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1560s) if I were to do it, I'll say

[26:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1564s) I have a wedding and I need to make

[26:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1568s) three payments. One for $6,000, another

[26:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1571s) one for an unknown amount, uh, and

[26:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1574s) another one for the rest of the money.

[26:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1576s) And also, uh, some of my costs are

[26:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1580s) fixed. I know what I'm going to spend on

[26:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1583s) a DJ and

[26:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1586s) I don't know decorations, but I don't

[26:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1588s) know how many people are going to attend

[26:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1590s) the wedding. And that's the amount of

[26:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1592s) money I'm going to pay per h per dinner

[26:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1595s) per seat. So, I don't even know how to

[26:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1597s) go about making sure that I can pay in

[26:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1599s) time

[26:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1601s) in time. On time, on time on time. Um,

[26:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1604s) I'll say let me just cut the first bit.

[26:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1608s) Cool.

[26:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1610s) So if you want to run it for a problem

[26:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1612s) that you actually care a little bit more

[26:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1614s) about uh

[26:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1616s) you will start there and all I'm doing

[26:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1618s) is planning with the agent.

[27:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1622s) Most of the harnesses today come with

[27:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1624s) some notion that triages that you're

[27:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1626s) trying to plan for a problem and it will

[27:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1628s) try to come up with a reasonable idea

[27:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1630s) some set of questions to give some more

[27:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1632s) clarity. The harness itself is what

[27:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1635s) guides the sort of questions that you

[27:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1637s) will get asked and the sort of rigor

[27:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1639s) that the plan needs to have before

[27:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1641s) implementing anything. The idea is that

[27:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1644s) it doesn't go immediately into a yes man

[27:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1648s) mode and just implements whatever crazy

[27:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1651s) idea you came up with

[27:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1653s) that is very unclear but that helps you

[27:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1656s) drill down and dig into what it needs to

[27:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1658s) do. So that's the principle of getting a

[27:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1660s) shared context.

[27:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1664s) Quick one before you you continue. Is

[27:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1666s) the size okay for the back of the room?

[27:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1668s) The size of the text. Yes. Okay. Cool.

[27:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1670s) Thank you.

[27:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1674s) Yeah. Sorry, I'm reading because this is

[27:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1676s) like live.

[27:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1679s) Okay. From the one I have.

[28:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1682s) Okay. So maybe I can keep adding

[28:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1685s) clarifying information here. And I'm

[28:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1687s) going to skip ahead, but something like

[28:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1691s) I have no savings.

[28:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1693s) It will come from my pocket. My mom may

[28:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1697s) give me like 5k. H I know the price per

[28:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1702s) a person going to the wedding is around

[28:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1708s) 150

[28:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1710s) and

[28:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1711s) I'm paying

[28:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1714s) 500 for the for the flowers. I need to

[28:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1718s) buy my suit.

[28:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1721s) I'm just adding a bunch of like

[28:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1722s) complexity and being disorganized on

[28:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1724s) purpose

[28:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1726s) because the point of the agent is that

[28:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1729s) it doesn't let go the fact that you're

[28:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1731s) being vague um in not providing the

[28:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1735s) information ahead of time. The whole

[28:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1737s) point of this is that it's really

[28:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1739s) guardrail for you to for to forcing

[29:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1742s) yourself to actually go through um every

[29:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1744s) single data point that it needs to come

[29:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1746s) up with a plan. Erh, I think for the

[29:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1750s) next one we just build whatever h comes

[29:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1752s) up to try and move on.

[29:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1754s) Propose the plan. Now,

[29:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1756s) is anyone trying it out with a problem

[29:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1759s) you have personally?

[29:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1763s) Yes. Okay, cool. Do you want to tell us

[29:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1766s) a little bit about it?

[29:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1770s) Oh, fair, fair, fair. Nice.

[29:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1777s) Okay. It's asking a lot of questions and

[29:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1778s) I'm

[29:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1779s) No, I I accepted the plan. Yeah.

[29:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1781s) So, can you go back to the seated

[29:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1784s) example?

[29:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1786s) Yes, I can. Um,

[29:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1788s) so to save you a lot of the back and

[29:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1789s) forth back and forth here because it's

[29:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1791s) not very high signal for any of you.

[29:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1794s) There's a couple of takeaways. One is

[29:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1797s) having that self-awareness and forcing

[30:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1800s) the hardness to remember that your

[30:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1803s) domain knowledge and whatever the agent

[30:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1806s) knows is not going to be the same. You

[30:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1810s) want to find that kind of like vin

[30:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1811s) diagram superposition as much as you

[30:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1814s) can. H but it takes a lot of back and

[30:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1817s) forth and that's often where most of the

[30:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1818s) time is spent. If you design a really

[30:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1821s) good harness, it will guide you by the

[30:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1824s) hand on coming up with a really good

[30:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1827s) plan, implementing things well, uh, and

[30:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1829s) knowing what components are important

[30:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1831s) and which ones aren't. Uh, if your

[30:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1834s) harness is basically like, yeah, say yes

[30:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1836s) to everything I say and do your best to

[30:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1838s) implement it, then you're going to get

[30:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1841s) variable results. So, this is the seated

[30:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1843s) example. The plan accepted thing at the

[30:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1846s) top means essentially that I already

[30:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1849s) went through the back and forth for

[30:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1851s) around 10 minutes or whatever. Uh and

[30:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1853s) the agent feel it knows enough about my

[30:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1856s) problem to break it down in steps. So

[30:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1859s) what you have in the diagram over there

[31:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1862s) is the all the steps that it considers

[31:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1865s) as decision points in a decision tree if

[31:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1869s) you may erh to actually make a decision.

[31:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1874s) And if I if I may, um I think it's it's

[31:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1877s) as we said in the beginning, it's nice

[31:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1879s) to look under the hood a bit just to

[31:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1881s) give you an idea and you can of course

[31:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1883s) explore this by yourself or with your

[31:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1885s) preferred agents help. Um but when we

[31:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1888s) are when we're talking about everything

[31:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1891s) that Franchesco is describing um this

[31:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1894s) also comes from the instructions that we

[31:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1896s) have inside the harness, right? So this

[31:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1900s) is where we have the instructions of the

[31:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1902s) agent performing the interview and what

[31:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1904s) it must end up with. So this is the

[31:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1907s) materialization

[31:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1908s) of the principle built beyond the

[31:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1911s) harness, right? Like built inside the

[31:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1913s) harness. I mean

[31:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1915s) if you want to go back to the to the UI.

[31:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1918s) So this is hopefully going to start

[32:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1920s) making more sense as we as we progress

[32:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1923s) in the example. But we added a couple of

[32:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1925s) quality of life things where if you want

[32:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1929s) to know what elements of this whole UI

[32:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1932s) that is so beautifully designed uh make

[32:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1936s) sense for whatever uh principle we're

[32:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1938s) discussing. You just need to hover down

[32:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1940s) at the bottom and it will be broken

[32:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1942s) down. So you see when we did the

[32:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1945s) planning the planning the sharing the

[32:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1949s) domain context it came up with a plan is

[32:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1952s) highlighted at the top on the right and

[32:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1954s) then it came up with all the nodes and

[32:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1956s) the connections in between the other

[32:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1958s) decisions and what we need to start

[32:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1960s) scoping from here. So

[32:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1964s) should we move on? Yes.

[32:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1968s) Is anyone still doing any planning with

[32:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1971s) the chat?

[32:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1976s) Cool.

[33:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1985s) Okay. Yeah. So, we had a few things

[33:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1986s) here. If you wanted to uh play around

[33:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1990s) and peruse the uh harness yourself, you

[33:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1994s) could for instance go on the step three

[33:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=1997s) in the UI like in the little diagram on

[33:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2000s) the left hand side.

[33:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2002s) Oops, sorry. I was going to show

[33:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2004s) too many tabs.

[33:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2005s) You could click on like step no

[33:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2007s) step three in the Yeah. And it will give

[33:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2010s) you a little bit of a breakdown of where

[33:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2013s) the information came from where why the

[33:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2016s) model thinks

[33:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2018s) this is the node you need to make a

[33:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2020s) decision of uh including if it's making

[33:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2023s) any assumptions based on the language

[33:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2025s) you were using during your planning with

[33:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2028s) it. So for instance the agent did you

[33:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2032s) say this I could point with this

[33:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2034s) uh if you press and hold that yeah

[33:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2037s) awesome

[33:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2038s) not AI

[34:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2040s) so there's there's quite a few things in

[34:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2042s) the plan uh that the agent put together

[34:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2044s) that were proposed by it so you can

[34:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2047s) start dissecting a little bit and

[34:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2049s) understanding like actually this came

[34:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2051s) from me this is a fact that I think I

[34:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2054s) know or this is something the agent

[34:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2055s) inferred based on my language. If you

[34:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2057s) scroll up a little bit just on the same

[34:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2059s) Yeah. more more more more more

[34:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2064s) there. Yeah, sure. Uh here

[34:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2068s) there was some back and forth because I

[34:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2070s) was being vague with the language I was

[34:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2072s) using and it made an assumption that

[34:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2074s) because I was using this keyword here, I

[34:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2078s) meant this. Again, this is a pretty

[34:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2080s) obvious example, but you can play around

[34:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2082s) with clicking through, seeing things

[34:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2084s) around. If we go back to the

[34:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2086s) presentation, there's some annotation

[34:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2088s) because again, this is all about

[34:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2091s) grounding the principles into something

[34:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2093s) that feels like it was designed for

[34:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2096s) building trust. So you have some

[34:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2098s) notation. Again, not the best UI design

[35:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2101s) in the world, but knowing if things came

[35:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2104s) from you, from the agent, or is it still

[35:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2106s) an open point and you need to make a

[35:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2107s) decision on and you can tell how the

[35:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2111s) plan has been built.

[35:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2113s) You have another thing you can try. You

[35:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2115s) can go back into step two uh and

[35:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2119s) actually tell the agent if we click on

[35:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2121s) the node there. Yeah. And you can

[35:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2124s) actually tell the agent after you have

[35:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2126s) sorry uh you can just tell the agent

[35:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2130s) like well actually you're making some

[35:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2131s) assumptions about this step that are not

[35:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2135s) correct and then go a little bit back

[35:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2137s) and forth. Erh, I will still do the show

[35:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2141s) of hands thing, but because of

[35:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2143s) reinforcement and engagement, uh, who

[35:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2147s) has been in a chat session with over

[35:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2151s) like well over 10 turns, so back and

[35:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2153s) forth and back and forth at least 10

[35:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2155s) times where at some point you said, let

[35:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2159s) let me start from the beginning like

[36:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2160s) this is not making any sense. Some stuff

[36:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2162s) we already discussed like it's well

[36:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2164s) gone. Who has had that feeling on a

[36:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2168s) regular basis?

[36:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2170s) Yeah. So, that is a consequence of the

[36:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2173s) design of the harnesses we have today

[36:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2176s) where they're designed for a linear

[36:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2178s) discussion and to keep everything in the

[36:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2182s) context as one big conversation. Whereas

[36:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2186s) here the way we designed it is supposed

[36:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2188s) to break it down so you can discuss

[36:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2190s) individual elements without necessarily

[36:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2194s) making things you know more complicated

[36:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2196s) in the grander scheme of things.

[36:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2198s) Yeah.

[36:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2201s) Erh

[36:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2203s) cool. So I will let you play around with

[36:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2205s) it a little bit but I do want to maybe

[36:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2207s) go a little bit further. So I will

[36:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2211s) introduce the consistency principle. And

[36:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2215s) this build on the shared domain

[36:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2218s) because you already have a plan and the

[37:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2222s) agent is very capable to understand well

[37:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2225s) these pieces of your plan are just a

[37:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2228s) piece of code or just a calculation

[37:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2230s) spreadsheet like it will always give you

[37:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2232s) a consistent result is a deterministic

[37:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2235s) element of your whole plan. Erh and

[37:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2238s) based on that you need to implement

[37:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2241s) code. You need to implement something

[37:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2243s) that will give you the same result every

[37:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2244s) time. Erh similar show of hands thing.

[37:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2248s) Has anyone tried to do like budgeting or

[37:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2251s) any financial calculation in the chat

[37:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2257s) and how did that go? Like are you

[37:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2260s) confident on the results? You Yeah, a

[37:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2262s) little bit.

[37:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2262s) Temperature maybe like very well um not

[37:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2265s) too well. pretty bad.

[37:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2269s) Okay.

[37:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2271s) So, so all right. So, I would love to

[37:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2274s) know if anyone wants to volunteer their

[37:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2277s) story, but what they saw and why they

[38:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2280s) think it happened. But to me, a couple

[38:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2282s) of times I thought I was giving

[38:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2286s) all the necessary information. The the

[38:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2289s) agent gave me back a formula. It was

[38:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2291s) even formatted as a formula. You know,

[38:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2293s) the kind of like italics formula thing

[38:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2295s) that looks really nice. I was like knows

[38:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2298s) what he's talking about. And then the

[38:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2299s) result was like I'm not a calculator and

[38:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2302s) I know that result is wrong. So from the

[38:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2305s) get-go I was like nah start from

[38:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2306s) scratch. H anyone has any other like

[38:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2308s) examples

[38:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2309s) consistency right? So a lot of times the

[38:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2311s) results look good reasonable but you run

[38:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2314s) the same thing again

[38:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2316s) that's something.

[38:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2325s) Yeah. So that's basically the

[38:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2326s) inspiration on the principle. Sorry,

[38:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2328s) what do you want to share?

[38:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2329s) Yeah. So it goes to the point that

[38:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2336s) every time it might

[39:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2347s) Yeah. Uh and and that's a great a great

[39:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2351s) um example to say part of the benefit

[39:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2354s) why we are talking about building a

[39:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2356s) harness and not just plugging these

[39:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2359s) principles in a set of skills or

[39:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2361s) instructions to an existing harness such

[39:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2364s) as like codec, cloud code or anything

[39:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2365s) else. Like when we when we do it the not

[39:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2369s) the non-deterministic way, which is to

[39:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2371s) give the agent a bunch of instructions

[39:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2373s) that um say the right things, but still

[39:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2376s) rely on the agent uh make no mistakes,

[39:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2379s) please do the right thing. Um that's

[39:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2382s) that's the point. We're relying on a

[39:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2383s) bunch of things happening in the

[39:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2385s) background and basically getting a

[39:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2387s) nondeterministic result every time. So I

[39:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2390s) think the value here from building the

[39:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2392s) harness again this is looking a bit into

[39:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2394s) the under the hood and like from

[39:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2396s) principles we we applied at at Monzo as

[39:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2399s) well is mixing the non-determinism with

[40:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2402s) some deterministic behavior that we want

[40:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2405s) to control um when when relevant

[40:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2410s) I think you mentioned that for this demo

[40:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2412s) a lot of this harness was quoted right I

[40:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2415s) get this part right for production

[40:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2420s) like what's the difference between you

[40:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2422s) making calculations in in chat versus

[40:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2425s) you asking the model to code up

[40:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2428s) calculations for you in code.

[40:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2430s) So I have a couple of personal examples

[40:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2433s) on doing forecast where you can try and

[40:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2437s) do forecast yourself by hand or come up

[40:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2440s) with a spreadsheet with a lot of fancy

[40:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2442s) um calculations and you will likely get

[40:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2445s) a few things right and get a decent

[40:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2446s) forecast. You can also just get a

[40:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2448s) library from a trusted uh repo and that

[40:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2452s) has everything baked in. You can also

[40:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2454s) write some code in a more modular

[40:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2456s) fashion that is well unit tested, you're

[40:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2459s) doing end to end testing, you're doing

[41:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2461s) all the like good software um

[41:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2463s) engineering practices that we know how

[41:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2466s) to do to ensure that that piece of code

[41:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2469s) is going to give you a good result. Um,

[41:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2472s) so I think the different Oh, sorry. The

[41:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2475s) difference lies a little bit on the fact

[41:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2477s) that if you're just leaving it to the

[41:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2481s) agent to decide how to implement things

[41:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2484s) and when they need to do it in code

[41:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2487s) versus when it like you should just

[41:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2489s) build something on the side, you're

[41:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2491s) going to have a little bit of a big miss

[41:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2494s) ratio like hit or miss.

[41:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2496s) Yeah. Yeah. I I guess what I'm asking is

[41:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2498s) if this is why quoted no good

[41:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2500s) engineering practices were applied. I

[41:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2502s) get this part. Do you have like a

[41:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2503s) product that that was tested?

[41:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2506s) We we I'm going to do the the uh ad

[41:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2510s) again. Uh tomorrow we're going to have a

[41:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2512s) breakout session in the morning where

[41:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2513s) we're going to talk about uh what we

[41:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2515s) build at Monzo that applies some of

[41:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2518s) these principles. uh but yeah I think

[42:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2520s) the what is behind when we say like the

[42:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2522s) harness being voded I think it's more

[42:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2524s) about the implementation detail of like

[42:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2526s) abstractions methods the actual code uh

[42:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2530s) but the principles of mixing the

[42:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2532s) determinism and the non-determinism are

[42:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2535s) the same regardless if that makes sense

[42:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2538s) thank you

[42:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2541s) uh and another thing another thing that

[42:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2543s) signals that this is a a good principle

[42:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2547s) to consider

[42:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2549s) on over the last what year or so most of

[42:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2553s) the agents have started to do more

[42:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2556s) aggressive tool calling and building

[42:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2557s) their own tools to solve calculations in

[42:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2559s) a more deterministic way

[42:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2562s) and honestly agents are just getting

[42:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2564s) better right so we could just wait and

[42:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2566s) then all the problems are solved the one

[42:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2568s) challenge I have with that mindset is

[42:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2571s) that h it will create stuff on the on

[42:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2574s) the go like in flight so Yes, you may

[42:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2578s) get the right code for the right problem

[43:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2581s) at an at a point in time, but maybe the

[43:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2583s) time number 100, you run it, turns out

[43:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2586s) is inventing a new piece of code that

[43:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2588s) didn't need to be reinvented. uh and

[43:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2590s) other examples non-workrelated that I

[43:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2593s) can share err are around like doing

[43:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2596s) exactly that like I have a I don't want

[43:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2599s) to say strict but semi-strict close of

[43:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2602s) budget every month and I categorize

[43:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2605s) everything in my statements

[43:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2607s) I left it to the agent to decide how to

[43:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2610s) do the categorization and how to do the

[43:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2612s) aggregates and everything else and the

[43:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2614s) refunds and expenses and at some point I

[43:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2616s) was like no no no this needs to be a

[43:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2618s) script because I'm gettinglight slightly

[43:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2620s) different results every time and I'm

[43:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2621s) spending more money more money and time

[43:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2624s) uh wrangling with the agent to get it to

[43:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2627s) give me the results that I wanted that

[43:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2628s) it got right three times already but

[43:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2631s) didn't get it right in the fourth erh so

[43:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2636s) I don't want to repeat it again but make

[43:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2638s) sure that we run the calculations

[44:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2641s) you write the calculation once and then

[44:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2643s) you run it every time you need it um

[44:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2647s) and the model is able to pick the tools

[44:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2650s) is you just need to make sure that your

[44:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2652s) hardness makes it very strict that it's

[44:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2655s) not going to do any arithmetic or try to

[44:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2657s) solve these problems by just parsing

[44:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2660s) tokens. Um so what can happen if you

[44:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2665s) don't make sure that your calculations

[44:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2668s) and everything that is deterministic

[44:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2670s) runs in code they're in the thickest in

[44:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2672s) the model's head. So it's gone once you

[44:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2674s) start new session the new code for every

[44:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2677s) new question and or new session and it

[44:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2680s) becomes a mess and same question

[44:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2682s) different answer non-determination again

[44:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2685s) you're asking the stoastic machine to

[44:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2686s) produce a consistent output so you're

[44:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2688s) likely not going to get it. Um,

[44:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2692s) so here's some examples of things that

[44:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2696s) have a way to be calculated, right? Like

[45:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2700s) I'm sure if you ask 10 different

[45:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2703s) financial services companies, they will

[45:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2705s) have a slightly different way of doing

[45:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2707s) these things, but the general formula,

[45:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2709s) the general idea is the same. So you

[45:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2711s) could realistically export a library,

[45:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2714s) write it once, update it, and these

[45:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2717s) things are not new. they saw from like

[45:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2719s) software good practices. So demo again

[45:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2723s) if we just want to go to the seated

[45:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2725s) example

[45:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2729s) h

[45:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2731s) in the

[45:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2733s) can you highlight this? Okay.

[45:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2737s) So again just using this h you can see

[45:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2740s) how the nodes in your decision making

[45:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2743s) process um some of them needed to be

[45:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2746s) implemented as code and I think the one

[45:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2749s) of the easiest way to check is if it's a

[45:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2752s) calculation likely it has to be

[45:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2753s) implemented as code right uh as well as

[45:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2756s) some testing unit testing golden

[45:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2758s) examples etc. the you should build trust

[46:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2762s) with your agent when you're building a

[46:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2765s) harness like this by making sure that

[46:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2767s) the information you care about about its

[46:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2769s) implementation

[46:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2770s) it's in front of you. Another example of

[46:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2773s) how this has looked in the wild is

[46:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2776s) giving you versions of the code. Your

[46:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2778s) your code is more likely versioned. If

[46:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2781s) anything change with the version of the

[46:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2783s) code and you're trying to run the same

[46:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2784s) process, the agent should be able to

[46:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2786s) pick it up and surface that information

[46:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2788s) to you and say actually this code change

[46:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2791s) you probably even though the function is

[46:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2793s) the same you probably shouldn't trust uh

[46:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2796s) what's coming out of this because the

[46:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2798s) whatever calculation h is different now.

[46:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2801s) So as you're working with the with the

[46:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2805s) harness and you accepted the plan, you

[46:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2808s) have a nice graph like this being built,

[46:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2811s) you will get something at the top

[46:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2813s) saying, do you want me to build the

[46:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2815s) code? H and it will do it for you. And

[46:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2819s) again, it's all guided. That's the point

[47:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2820s) of a hardness that it guides things.

[47:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2824s) Erh, is anyone

[47:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2827s) building any of the steps on their plan?

[47:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2833s) Is that okay?

[47:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2834s) Yeah. Yeah. Cool.

[47:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2837s) I switch to focus.

[47:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2839s) Okay. Yeah. [laughter]

[47:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2841s) Fair enough.

[47:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2842s) Fair. Fair.

[47:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2848s) Cool. So if you're following on the

[47:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2851s) wedding example, you can try a few

[47:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2854s) things. Click on step one, then code,

[47:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2857s) and then the test and run the tests now.

[47:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2867s) Tests here.

[47:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2868s) Yeah.

[47:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2872s) 95.

[47:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2875s) Sorry, what time is it?

[47:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2876s) 9:54. [laughter] Okay, good. Um, but

[48:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2880s) yeah, the the the

[48:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2883s) principle here and just going back to

[48:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2885s) try and tie everything up together, you

[48:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2887s) shouldn't rely on you remembering to run

[48:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2890s) the test. You shouldn't rely on you

[48:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2892s) remembering that, hey, this this sounds

[48:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2894s) like I can h create some code to solve

[48:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2898s) it every time. The agent should guide

[48:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2900s) you through it. And that's one of the

[48:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2902s) principles we baked into building this.

[48:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2905s) Uh, as an aside, I think we were telling

[48:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2907s) you, uh, there's a little extra file in

[48:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2911s) the repo that says build yours or

[48:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2914s) something.

[48:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2915s) Um, it essentially has all the

[48:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2917s) principles baked in in order and it will

[48:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2919s) guide you on building a [snorts] similar

[48:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2923s) hardness.

[48:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2925s) It's built

[48:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2927s) your own.

[48:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2928s) There you go.

[48:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2932s) Yeah. So yeah, it will guide you on

[48:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2935s) building something that feels similar to

[48:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2937s) the harness we're showing you that

[49:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2940s) prioritizes trust building mechanisms

[49:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2943s) over just giving you something that

[49:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2945s) looks correct. So you can try this. The

[49:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2947s) [snorts] test should be passing. You

[49:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2949s) will notice that there was no thinking,

[49:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2953s) perusing,

[49:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2956s) all the verbs that agents put when

[49:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2958s) they're loading something. Yeah. because

[49:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2961s) these are just tests that need to run in

[49:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2963s) your environment. Don't overengineer it.

[49:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2968s) Funny that I'm saying that, but it's

[49:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2970s) easier if all the components of your

[49:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2973s) harness just work together as they

[49:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2975s) should and not based on like hopes and

[49:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2978s) dreams and magic glue from an agent

[49:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2980s) orchestrating things together.

[49:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2982s) I think we had a question.

[49:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=2998s) Erh, so

[50:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3001s) if we Yeah,

[50:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3003s) I mean we can do that, but I'd rather

[50:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3005s) not do a reading session.

[50:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3007s) Basically, it it all goes about h

[50:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3010s) figuring out what you actually care

[50:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3011s) about and what are the functional

[50:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3013s) requirements for a functioning harness.

[50:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3016s) H I'm sprinkling some of that through

[50:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3018s) the conversation. Again, need to avoid

[50:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3020s) oversharing a little bit because of IP.

[50:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3023s) Uh but you want to make sure that the

[50:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3027s) objectives of each of the steps when

[50:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3029s) you're building the parts that will make

[50:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3031s) up the behavior of your hardness have an

[50:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3034s) intent. The intent of one of the first

[50:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3037s) components is gathering as much

[50:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3039s) information as possible to come up with

[50:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3040s) a plan. And that is written in here as

[50:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3043s) well. So this is basically instructions

[50:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3046s) for instructions. It's the IKEA guide.

[50:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3049s) Um it is supposed to guide you very

[50:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3052s) nicely on these are the things you

[50:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3054s) should care about but but then you can

[50:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3056s) pick uh your preference on how they get

[50:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3059s) implemented.

[51:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3063s) Question. So how is this different from

[51:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3066s) let's say the specdriven development

[51:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3069s) expected something like that which is

[51:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3071s) also linear from starting from all the

[51:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3074s) questions from you starting from idea

[51:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3076s) all the way to the end and how is it

[51:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3079s) different to this one is different to

[51:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3081s) that

[51:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3082s) do you mind if I come to that um I I

[51:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3085s) want to mention two things that I think

[51:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3087s) are are relevant here um I think the

[51:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3089s) first one is when we're building a

[51:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3091s) harness for a specific type of use case

[51:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3093s) it's a bit different from the coding

[51:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3096s) assistant world of agents because with

[51:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3097s) the coding assistance we usually the

[51:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3100s) code is a means to an end right like we

[51:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3102s) we're building something that performs

[51:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3104s) an action. It's a product, is a an app,

[51:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3107s) whatever that is. Um, in this case, the

[51:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3109s) agent is helping us like analyze, solve

[51:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3112s) a problem, and work us through that. So,

[51:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3116s) the the gigantic spec for this like

[52:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3120s) let's say for Franchesco's wedding,

[52:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3122s) things will change like it will evolve,

[52:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3124s) it will change. And then if we go back

[52:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3126s) to the everchanging spec, we are relying

[52:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3130s) again on the determin on the

[52:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3132s) non-determinism of the agent to

[52:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3134s) interpret that that can be very very

[52:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3136s) very um robust and perform this the the

[52:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3140s) correct thing. So the harness

[52:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3142s) essentially breaks down what would be

[52:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3145s) the spec in opinionated sections if that

[52:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3149s) makes sense. Um and I think the second

[52:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3151s) thing to complement with that and is

[52:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3153s) actually something I wanted to to like

[52:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3155s) ground on our experience from data. I

[52:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3157s) don't know how many of you work in data

[52:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3160s) engineering analytics or anything like

[52:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3162s) that. Yeah. I think something else that

[52:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3165s) we bring from this word in these

[52:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3166s) principles is the idea that um metric

[52:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3170s) definitions and dimension definitions

[52:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3172s) always change. like in our experience

[52:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3174s) we've like [snorts] had many many many

[52:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3176s) times and I'm sure many of you as well

[52:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3179s) um that conversation were is this number

[53:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3181s) correct? I don't know. It sounds like it

[53:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3184s) is, but we're never sure and then if

[53:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3186s) we're never sure, is the number correct?

[53:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3188s) Um, I think this is also coming from

[53:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3190s) that perspective is that to the point

[53:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3192s) where we trust the agent as well as we

[53:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3195s) trust a dashboard or a metric or a

[53:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3197s) visualization. Um, we've reached

[53:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3199s) alignment with it and the harness can

[53:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3202s) continue guiding us into that alignment

[53:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3204s) continuing to be true over time. But

[53:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3207s) premises will change, source data will

[53:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3209s) change, context will change. And then

[53:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3212s) having the harness able to adapt to that

[53:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3215s) is like recognizing that this change

[53:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3217s) will be a thing um rather than having to

[53:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3222s) every time pro prompt again and go back

[53:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3225s) to the process. Does that make sense?

[53:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3229s) Yep.

[53:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3234s) Follow question on that. any complnt

[53:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3236s) this perspective

[53:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3239s) because most most workable thing you are

[54:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3243s) developing something even though it's

[54:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3245s) subject to change through different

[54:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3247s) iterations you are developing something

[54:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3250s) with

[54:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3252s) an AI agent can be part of this flow as

[54:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3254s) well

[54:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3254s) absolutely use that this part this

[54:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3257s) harness to do that it

[54:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3260s) absolutely and this like under the hood

[54:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3264s) this is essentially what's happening as

[54:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3266s) well. So when we have when we're

[54:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3269s) creating

[54:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3271s) um the

[54:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3273s) if you if you look under the hood in the

[54:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3276s) my folder, you're going to have

[54:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3278s) everything that is saved. So you can

[54:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3282s) think of this as the spec that's behind

[54:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3285s) the harness. So absolutely like the same

[54:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3287s) principle applies here in the in the

[54:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3289s) specdriven development for again coding

[54:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3291s) agents is usually used. you have

[54:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3293s) something that is the source of um

[54:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3296s) definition for the thing being built. In

[54:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3298s) this case, it's the same thing. The

[55:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3300s) difference is that we surface that

[55:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3301s) through opinionated like uh workflows

[55:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3304s) rather than a single um looking document

[55:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3307s) if that makes sense.

[55:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3311s) All right, cool. So,

[55:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3316s) that's another thing you can try out. um

[55:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3319s) you can try and change how a calculation

[55:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3322s) works. Uh and you're likely going to get

[55:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3325s) some push back from the harness. Just to

[55:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3327s) reiterate, the point of the harness is

[55:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3329s) to guide you and force you to do the

[55:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3332s) right thing. And the principles are here

[55:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3334s) to help you make the right decision

[55:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3336s) every time. So again if you go try and

[55:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3339s) change uh multiplication for a an

[55:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3342s) aggregation you may just go make the

[55:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3345s) quick edit save the file success you

[55:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3348s) sabot attach the analysis but when you

[55:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3350s) run the test you won't be able to

[55:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3352s) actually reach any conclusion of the

[55:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3354s) analysis you wanted to do h because the

[55:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3357s) hardness is going to stop you by its own

[55:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3359s) like definitions and the way it was

[56:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3361s) built. Um,

[56:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3364s) should we move on

[56:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3367s) evidence?

[56:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3368s) Er, evidence is the whole thing I was

[56:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3371s) saying before about the receipts, having

[56:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3373s) the receipts and being able to show that

[56:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3376s) well, yeah, this is where this came

[56:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3378s) from. This is correct because of in

[56:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3380s) doing that causality relationship. So it

[56:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3384s) builds on the second principle as since

[56:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3388s) you have code that is tested as well as

[56:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3391s) everything else h you have a trail that

[56:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3394s) you can follow backwards. So you have a

[56:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3396s) way to substantiate the decisions you're

[56:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3398s) making and you have the evidence

[56:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3401s) and it's important in this principle

[56:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3404s) because a principle is something you

[56:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3406s) need to interpret right and I don't want

[56:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3408s) to leave you with a very vague

[56:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3410s) ambitional thing like oh yeah do

[56:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3413s) evidence it's awesome but what is what

[56:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3417s) about evidence is something that you

[57:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3420s) need to take care of going back to being

[57:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3422s) data professionals this is something

[57:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3424s) that bites us on a weekly basis. Erh,

[57:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3428s) not all information is the same. If you

[57:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3431s) really want to present a piece of

[57:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3433s) analysis or you went to do some

[57:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3435s) financial analysis on a potential

[57:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3437s) investment for your firm, you need to

[57:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3440s) show where your numbers came from

[57:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3443s) and be able to attribute your decision

[57:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3445s) to the body of evidence you had. Erh so

[57:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3450s) since we can show where everything came

[57:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3453s) from uh there's some mistakes we want to

[57:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3457s) avoid where everything is treated as a

[57:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3459s) fact. So oh yeah this number you gave me

[57:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3463s) for the number of um I don't know going

[57:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3466s) back to my wedding the number of people

[57:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3468s) invited is a fact. Turns out my fiance

[57:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3471s) invited three more people didn't tell

[57:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3472s) me. Now my whole model is wrong. uh and

[57:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3476s) I'm trying to reduce it to a very simple

[57:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3478s) example but I think you can imagine how

[58:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3481s) having a wrong assumption somewhere and

[58:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3483s) not knowing where it came from can be

[58:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3485s) fatal for any financial analysis.

[58:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3488s) Um

[58:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3490s) so let's say in the world of a gentic

[58:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3495s) building it's very easy to assume

[58:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3498s) something or treat something as a

[58:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3500s) gueststimate. erh the model can't tell

[58:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3504s) what is a hard truth and what is an

[58:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3506s) estimation. So you need to enforce that

[58:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3508s) through the design of your hardness.

[58:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3511s) Erh, and another example where this kind

[58:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3514s) of like falls fall into pieces really

[58:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3516s) quickly is imagine you have a data

[58:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3520s) person, an analyst, something coming to

[58:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3522s) you and being like, "No, no, no. Here's

[58:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3524s) the result. Trust me, I run the

[58:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3525s) numbers." Do you trust that as a Yeah, I

[58:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3528s) see some like no.

[58:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3532s) I don't think anyone here will want to

[58:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3534s) see all the logs and the math behind a

[58:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3537s) single number. But just having the trace

[59:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3540s) to figure out, okay, I can follow back

[59:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3542s) this decision increased increases

[59:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3545s) confidence for us humans quite a bit.

[59:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3548s) Erh, so we have some like different

[59:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3552s) examples here. Erh, unstructured

[59:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3554s) thoughts in a chat needs to be labeled

[59:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3557s) as an interpretation. the agent

[59:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3559s) reasoning or pulling information from

[59:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3561s) memory needs to be treated as an

[59:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3563s) assumption and then if I submit a bank

[59:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3566s) statement what is there to discuss like

[59:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3568s) that is a fact that's how much I spent

[59:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3571s) etc

[59:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3573s) if I take like go back again to being a

[59:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3576s) data person data labeling and making

[59:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3579s) sure that you have a curated

[59:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3581s) data set for either training something

[59:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3584s) or running any process is paramount and

[59:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3586s) this is basically the same thing you

[59:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3588s) just need to assign the right level of

[59:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3592s) importance to the evidence you're

[59:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3594s) gathering and make sure that you gather

[59:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3595s) as much as as much of it as you can.

[59:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3599s) Um, this is a really funny one, but

[1:00:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3604s) evidence also goes into the code. So,

[1:00:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3606s) it's not just where do you get this

[1:00:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3608s) number from, but how did you get this

[1:00:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3612s) number? And I had I have an anecdote

[1:00:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3615s) where someone was using the master file.

[1:00:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3618s) In every company there's a master file

[1:00:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3620s) and is favorited in the share drive and

[1:00:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3623s) whatever. And that person puts some

[1:00:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3625s) numbers in there and everything looked

[1:00:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3627s) fine. The master file had some

[1:00:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3629s) assumptions

[1:00:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3630s) of what range of values were reasonable.

[1:00:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3634s) That person didn't know because the file

[1:00:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3636s) changed. They didn't do anything wrong.

[1:00:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3637s) They just didn't know the file changed.

[1:00:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3639s) Uh and then the results were entirely

[1:00:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3641s) wrong. And the same will happen with

[1:00:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3644s) deterministic code you build with your

[1:00:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3646s) harness. If you don't have a way to

[1:00:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3650s) having a traceability even for changes

[1:00:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3652s) you make in your code, you're leaving

[1:00:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3654s) yourself uh vulnerable for blind spots

[1:00:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3657s) on the process change. The inputs were

[1:00:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3659s) the same.

[1:01:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3661s) Erh skipping ahead a bit.

[1:01:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3668s) So if we're following from the example,

[1:01:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3675s) can you open any of them?

[1:01:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3677s) Yes,

[1:01:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3679s) the evidence is highlighted. Erh,

[1:01:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3683s) and again,

[1:01:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3686s) you can see like the areas that you care

[1:01:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3688s) about through the example by just like

[1:01:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3690s) clicking on the bottom and then seeing

[1:01:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3691s) the bits that get highlighted.

[1:01:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3694s) But this is how the flow of information

[1:01:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3696s) goes through and what parts are proposed

[1:01:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3699s) by an assistant like you can show the

[1:01:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3701s) receipts of where where this came from.

[1:01:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3703s) Looking at other implementations of

[1:01:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3706s) financial analysts in the industry

[1:01:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3709s) you get traceability down to the data

[1:01:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3712s) set like you know which uh connector or

[1:01:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3716s) which supplier of data gave you the

[1:01:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3718s) information that whatever Apple had this

[1:02:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3721s) price at this point of the day. if

[1:02:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3723s) you're doing something about trading or

[1:02:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3724s) investing in stocks and shares. So, this

[1:02:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3727s) is extremely important. I don't think we

[1:02:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3730s) need to spend any more time here unless

[1:02:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3732s) anyone has any questions.

[1:02:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3736s) Oh, good.

[1:02:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3738s) Cool.

[1:02:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3743s) So, here's a few things you can try. Um,

[1:02:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3748s) so you can add a little bit of a a

[1:02:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3751s) curveball to the agent. Just typing in

[1:02:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3753s) the chat and you will see how some of

[1:02:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3757s) those evidence lines or some of the

[1:02:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3760s) inputs that go into the rest of your

[1:02:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3761s) process change and it it's able to trace

[1:02:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3765s) that it came from you correcting some

[1:02:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3768s) fact in the chat.

[1:02:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3774s) And another example is erh if you want

[1:02:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3778s) to go and make a change and click on the

[1:03:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3781s) underlined figure that changed. It

[1:03:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3784s) should show you all the way back where

[1:03:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3786s) this came from and why it matters for

[1:03:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3788s) the rest of your calculation.

[1:03:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3792s) I'm going to say something that might

[1:03:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3795s) feel repetitive. Um but again we are we

[1:03:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3799s) are this this harness is an example of

[1:03:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3801s) how to apply the principles. But I I'm

[1:03:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3803s) sure everyone is thinking about your use

[1:03:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3805s) cases, your companies, the agency you're

[1:03:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3807s) building, the agency you're using, and

[1:03:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3809s) how like to to to connect the dots and

[1:03:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3811s) to to I don't know put the wires

[1:03:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3813s) together. I think this is an an example

[1:03:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3815s) we can think about when our users, our

[1:03:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3819s) customers of the agents that we're

[1:03:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3821s) building are using it. And again the

[1:03:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3825s) trust that they have in your product in

[1:03:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3827s) your agent and in the output is what

[1:03:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3829s) will drive usage adoption improvement

[1:03:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3832s) and so on through the loops that we

[1:03:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3835s) mentioned through tracing evolves and so

[1:03:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3837s) on. So again, I think um what's

[1:03:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3839s) underlying here is when you're building

[1:04:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3842s) a a harness, make sure that you can um

[1:04:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3846s) make this happen to uh your customers

[1:04:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3849s) that are using your users that are using

[1:04:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3852s) so that they trust the results that your

[1:04:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3854s) agent is giving them. Again, maybe

[1:04:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3856s) repetitive, but like making sure we uh

[1:04:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3860s) draw get everything together as we go.

[1:04:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3864s) Before

[1:04:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3868s) we move on, anyone has any example of a

[1:04:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3871s) time where having the evidence trail

[1:04:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3874s) would have saved you a lot of pain?

[1:04:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3879s) I see nodding like painful

[1:04:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3882s) acknowledgement, but if anything you

[1:04:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3883s) want to share, more than happy to to

[1:04:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3887s) hear you out. Otherwise, we can carry

[1:04:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3888s) on.

[1:04:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3891s) basically every single time you try to

[1:04:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3893s) do something. And

[1:04:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3896s) so I'm playing with this. I really like

[1:04:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3898s) the idea of being able to show the

[1:05:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3900s) receipts and everything. That's

[1:05:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3902s) basically every single conversation I

[1:05:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3904s) have with an agent, whether I'm coding

[1:05:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3906s) or like working on a deck or something.

[1:05:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3908s) I'm like, "Where did this number come

[1:05:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3910s) from?" Like, "Well, we talked about this

[1:05:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3912s) blah blah blah blah blah." I'm like, "It

[1:05:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3914s) would be nice if I could just be able to

[1:05:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3917s) show me specifically like the paper

[1:05:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3922s) I like the that you use a presentation

[1:05:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3925s) as an example

[1:05:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3927s) because I had an example myself where I

[1:05:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3931s) was trying as hard as I could to not

[1:05:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3934s) have to resize text boxes in Google

[1:05:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3937s) Slides for a good like month and a half.

[1:05:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3940s) So I was forcing my way into having

[1:05:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3943s) something that could help me build

[1:05:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3944s) slides. Erh, and I had a really nice

[1:05:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3947s) harness that would like dig into the

[1:05:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3949s) problem, get all the relevant documents

[1:05:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3952s) and help me highlight what are the key

[1:05:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3954s) points that I wanted to present to the

[1:05:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3956s) team. And turns out in my conversations,

[1:06:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3961s) I never caught the fact that when I ask

[1:06:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3963s) for a modification and give an example,

[1:06:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3967s) the model will fixate on the example so

[1:06:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3970s) hard that it will put it everywhere in

[1:06:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3973s) my presentation. I didn't pick it up.

[1:06:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3976s) So, you can imagine I was showing some

[1:06:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3979s) colorful example to senior leadership

[1:06:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3982s) and they were like, "Where did you get

[1:06:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3983s) that number and that assumption?" And I

[1:06:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3985s) was like, "That's crazy. I don't know

[1:06:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3988s) where that came from. That's crazy." But

[1:06:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3991s) yeah, it tends to happen. It's really

[1:06:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3992s) annoying. And that trace of consequence

[1:06:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3995s) can really hurt the trust that you have

[1:06:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3997s) in your harness and the results it

[1:06:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=3998s) produces.

[1:06:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4001s) Let me carry on. So another principle is

[1:06:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4005s) on accountability. Uh

[1:06:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4008s) and this builds on the evidence because

[1:06:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4011s) you only make a decision when you have

[1:06:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4014s) some facts when you have some point of

[1:06:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4016s) reference. Erh well I do want to say

[1:06:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4019s) that but we all have that friend who

[1:07:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4021s) speaks very confidently about something

[1:07:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4023s) they don't know. So little bit of a

[1:07:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4025s) counter example,

[1:07:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4028s) but generally I think if there's one

[1:07:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4030s) takeaway I would love for everyone here

[1:07:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4033s) to feel aligned with is we are

[1:07:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4037s) ultimately the responsible people, the

[1:07:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4039s) accountable people for whatever AI

[1:07:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4042s) agents do, right? Um and there's a funny

[1:07:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4045s) thing where people discussing that AI is

[1:07:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4048s) going to um you know we're not going to

[1:07:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4052s) have any jobs because of AI. I always go

[1:07:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4055s) to back to the fact that I get higher, I

[1:07:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4058s) get paid partially because I am

[1:07:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4061s) accountable for the results that I

[1:07:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4062s) produce for the company, not for me

[1:07:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4064s) sitting in a computer and typing some

[1:07:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4066s) stuff. So there is an element of the

[1:07:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4069s) human experience where the whole trust

[1:07:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4072s) and accountability and working together

[1:07:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4074s) as a team matters a lot. And how do you

[1:07:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4076s) keep accountability high? You do need to

[1:07:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4079s) be able to see under the hood, inside

[1:08:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4081s) the black box, etc.

[1:08:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4084s) Erh,

[1:08:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4086s) I don't know who said this. I loved it.

[1:08:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4089s) I never able to quote it, but agents

[1:08:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4092s) face no real world consequences. You do.

[1:08:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4096s) So

[1:08:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4098s) I I love the show of hands thing, but it

[1:08:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4099s) probably is the only resource I have.

[1:08:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4101s) Maybe that's why.

[1:08:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4103s) Has anyone seen either written or been

[1:08:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4106s) told? Well, I don't know. That's what

[1:08:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4108s) Claude gave me. [laughter]

[1:08:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4112s) Yeah,

[1:08:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4113s) that feels great, right? Like, oh, that

[1:08:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4116s) makes sense. Yeah. Yeah. Everything is

[1:08:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4118s) forgiven. Don't worry. We face the

[1:08:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4121s) consequences of what agents do. So,

[1:08:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4124s) having accountability mechanisms in our

[1:08:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4127s) harnesses

[1:08:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4129s) reduces a lot the amount of stress you

[1:08:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4131s) need to go through. Just going in your

[1:08:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4134s) head like, "But do you think it made the

[1:08:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4136s) wrong assumption here?" Whatever. Erh,

[1:09:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4141s) so you want to reduce ambiguity and not

[1:09:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4145s) remove it. Uh, I'm pretty sure a lot of

[1:09:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4148s) people here will remember fondly the

[1:09:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4150s) days prior to auto mode. Uh, who wants

[1:09:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4154s) to go back to before when auto mode

[1:09:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4156s) existed and you had to approve every

[1:09:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4157s) single change by tapping enter?

[1:09:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4160s) Yeah. Yeah. Lovely times.

[1:09:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4164s) I still do it.

[1:09:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4167s) Fair enough.

[1:09:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4170s) [laughter]

[1:09:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4171s) Fair enough.

[1:09:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4176s) And from all the let's say 100 taps you

[1:09:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4180s) did, how many of them you failed where?

[1:09:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4182s) I mean, come on. This is a oneline

[1:09:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4184s) change versus this is important. Thank

[1:09:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4187s) god I was looking at this. Yeah, I just

[1:09:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4189s) pre

[1:09:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4198s) I used to do that a lot but now uh you

[1:10:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4201s) have the set of tasks you review the

[1:10:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4203s) task and then let it do it then I review

[1:10:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4206s) at the end

[1:10:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4208s) and and that's basic that's basically

[1:10:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4211s) part of the hardness like the design is

[1:10:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4213s) intentional so you review tasks instead

[1:10:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4216s) of individual changes is and we're just

[1:10:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4219s) providing you with a a reminder that

[1:10:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4221s) when you're designing a harness, you

[1:10:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4223s) want to make sure that you include on

[1:10:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4225s) purpose those gates to make sure that

[1:10:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4228s) you can still be accountable for what is

[1:10:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4230s) being produced. Um so make sure that any

[1:10:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4234s) judgment call, any disambiguation that

[1:10:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4237s) is important, not the trivial ones,

[1:10:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4239s) right? H you can get some hold of it

[1:10:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4242s) still. H what happens without it? I

[1:10:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4246s) don't know. The stoastic machine said it

[1:10:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4247s) was fine. Amazing answer. Approving

[1:10:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4251s) every calculation. The other side of the

[1:10:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4253s) spectrum doesn't feel great. Or

[1:10:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4256s) reviewing every PR line by line that

[1:10:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4258s) your agents ship. Yeah, that it's not

[1:11:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4260s) really well written, right?

[1:11:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4263s) But yeah, it leads to a lot of either a

[1:11:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4265s) lot of toil and unnecessary work if it's

[1:11:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4268s) not designed um in a way that you know

[1:11:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4272s) makes sense for the use cases.

[1:11:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4275s) Erh.

[1:11:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4278s) So what is the main takeaway on how to

[1:11:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4281s) design a good mechanisms for

[1:11:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4283s) accountability? Few gates with very high

[1:11:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4286s) signal. Your example of the tasks and

[1:11:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4289s) breaking it down in a way that you can

[1:11:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4292s) reason about the the way they connect.

[1:11:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4295s) It's a really good example h on where

[1:11:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4298s) people do h reviews only on functional

[1:11:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4301s) tests being passed. anything prior to

[1:11:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4304s) that. I think you can be a little bit

[1:11:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4306s) handwavy with yeah sure this looks about

[1:11:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4309s) right but you wouldn't push anything

[1:11:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4311s) into production that hasn't passed uh

[1:11:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4313s) all your tests right so you put some

[1:11:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4316s) gates in place but those gates need to

[1:11:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4319s) be very high signal um

[1:12:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4321s) and uh to to complement I think we're

[1:12:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4324s) using some examples in the coding

[1:12:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4327s) assistant world and and again this is

[1:12:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4329s) not how we for instance approach things

[1:12:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4331s) at at Monzo because of like a different

[1:12:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4334s) context text, but when we this is

[1:12:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4336s) bringing to to our hardness. I think we

[1:12:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4338s) use the AI coding examples because

[1:12:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4339s) they're close to us and they're the

[1:12:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4341s) tools many of us are using on a on a

[1:12:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4343s) daily basis. Uh but when we apply here

[1:12:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4345s) is the the same thing that claude code

[1:12:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4348s) does when they have a feature that

[1:12:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4351s) allows you to gauge how much you want to

[1:12:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4354s) be involved in the process. When we are

[1:12:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4355s) building our financial harnesses, we can

[1:12:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4358s) also guide our users in the in the same

[1:12:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4360s) way.

[1:12:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4363s) That is pretty spot on. Um, and again,

[1:12:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4367s) just going back to the domain with

[1:12:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4369s) money, there's some stuff you can leave

[1:12:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4372s) the agent to figure out. For instance,

[1:12:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4376s) if you have a few possible values for

[1:12:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4379s) something, you're like, well, do three

[1:13:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4381s) scenarios. It's probably going to be

[1:13:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4383s) fine. But you want to review them side

[1:13:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4385s) to side before making any decision. You

[1:13:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4388s) wouldn't hand over entirely the keys to

[1:13:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4390s) the kingdom to something that you don't

[1:13:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4392s) necessarily trust is going to have your

[1:13:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4394s) best interest in mind. Not that it will

[1:13:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4396s) have an interest at mine, but um

[1:13:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4401s) so this is a

[1:13:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4404s) it's very important that you know that

[1:13:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4408s) bad decisions carry forward into your

[1:13:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4411s) other decisions. So the gates you design

[1:13:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4414s) need to be fit for purpose. You need to

[1:13:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4416s) know the use case really well in order

[1:13:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4419s) to be able to put the gates where it

[1:13:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4421s) matters. Otherwise, you're just adding

[1:13:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4424s) before any decision, ask me and then it

[1:13:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4426s) becomes very low signal again. Um and

[1:13:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4430s) well for code you don't necessarily need

[1:13:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4432s) to ask because the tests will speak for

[1:13:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4434s) themselves. That's where the receipts

[1:13:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4436s) come in handy because you only need to

[1:13:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4438s) check that everything is passing and you

[1:14:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4440s) will have a lot of confidence because of

[1:14:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4442s) it.

[1:14:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4444s) So if you go back to

[1:14:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4446s) the demo,

[1:14:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4448s) funnily enough, this plan is not very

[1:14:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4451s) high on accountability.

[1:14:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4453s) The only step is I will check everything

[1:14:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4457s) that we calculated all the scenarios

[1:14:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4459s) whatever and I will make a decision on

[1:14:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4461s) how much I need to save every month.

[1:14:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4464s) I mean still nice because no money is

[1:14:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4466s) moving out of my bank account anywhere

[1:14:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4469s) but yeah there's a accountability

[1:14:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4471s) mechanism I feel confident that that

[1:14:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4473s) will gate any anything else

[1:14:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4476s) I think as a as a as an extra example

[1:14:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4478s) like in this this challenge and decision

[1:14:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4481s) here likely as we let's say we accept

[1:14:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4485s) this and this is now marked as a

[1:14:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4487s) decision we made if we ever use this

[1:14:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4490s) information like the 20k landing on the

[1:14:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4492s) same day as payment three as something

[1:14:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4495s) that influences the final decision. This

[1:14:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4498s) definitely comes with the with the

[1:14:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4499s) accountability.

[1:15:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4500s) Yeah.

[1:15:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4510s) Go up.

[1:15:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4511s) No, I actually like I'm improvising a

[1:15:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4513s) little bit, but if you scroll up a

[1:15:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4515s) little bit.

[1:15:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4515s) Mhm. Erh this is like again not not

[1:15:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4518s) priding myself of any good design at all

[1:15:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4521s) anything the contrary but h you see we

[1:15:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4525s) have a little bit of a weird structure

[1:15:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4527s) there looks like a tree a little bit

[1:15:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4530s) with something branching out joining

[1:15:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4532s) back in that's another accountability

[1:15:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4534s) mechanism as Philip is saying where

[1:15:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4536s) there's some decisions the model will

[1:15:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4539s) either assume and just try to verify

[1:15:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4542s) with you like hey is this right is this

[1:15:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4544s) the number you want to use here or just

[1:15:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4547s) try to figure out things how they fit

[1:15:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4549s) into the overall um system.

[1:15:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4553s) We thought a good way to showcase that

[1:15:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4556s) you need to design a harness that gives

[1:15:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4558s) you the accountability is making sure

[1:16:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4560s) that you can trace all the decisions

[1:16:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4562s) that have been made who made them uh and

[1:16:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4566s) have that sense of well this was

[1:16:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4568s) something that the agent assumed and I

[1:16:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4571s) let carry on in the rest of my flow or

[1:16:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4573s) it's something that

[1:16:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4576s) I provided a revised input to and the

[1:16:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4580s) reason again I'm a very visual person

[1:16:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4583s) that could be symptom of something. I

[1:16:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4584s) don't know. But the reason I think it's

[1:16:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4587s) kind of like a not a straight line

[1:16:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4589s) conversation, but rather branching out

[1:16:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4591s) and then branching back in is that when

[1:16:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4593s) you're chatting with an agent, again,

[1:16:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4595s) for financial purposes, you may reason

[1:16:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4598s) about something after a decision has

[1:16:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4601s) been made and you are well in your right

[1:16:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4604s) to correct information and add more

[1:16:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4606s) context. So, I think it's important that

[1:16:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4609s) when you're creating an a harness that

[1:16:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4612s) you not only build those gates for

[1:16:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4615s) validating things, making sure that

[1:16:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4617s) you're still accountable, but also for

[1:16:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4619s) correcting the course and making sure

[1:17:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4621s) that you can do so safely and it's not

[1:17:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4625s) like, oh, because I typed this sentence

[1:17:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4627s) in the chat, now all my analysis is

[1:17:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4630s) gone. H that has happened to me many

[1:17:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4633s) many times trying to solve a problem and

[1:17:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4636s) maybe I was like dispersed dictating to

[1:17:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4639s) my phone I said something in the middle

[1:17:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4641s) of it and then the whole analysis is

[1:17:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4643s) gone because I can't really take away

[1:17:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4646s) that one sentence and please keep

[1:17:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4649s) everything else as it was h so this

[1:17:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4651s) gives us another means to be accountable

[1:17:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4654s) for what goes into the system for

[1:17:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4657s) financial analysis and what

[1:17:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4662s) Do you want to move to the slides? Yes.

[1:17:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4668s) Okay, cool. So, uh, a few things we can

[1:17:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4670s) try. Erh, you can go and and hover into

[1:17:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4675s) the chip that guest that says how many

[1:17:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4678s) guests we have in the wedding example.

[1:18:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4680s) And you can add that that will add it to

[1:18:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4683s) the chat and you can say, "Oh, actually,

[1:18:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4684s) it's not this number. H we will get no

[1:18:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4687s) fewer than 150 guests for instance. Erh

[1:18:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4690s) that will get your harness to rethink

[1:18:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4693s) the plan see the consequences of that

[1:18:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4696s) input into other stuff and suggest a

[1:18:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4699s) either a modification of your plan or

[1:18:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4702s) just change the input whatever.

[1:18:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4705s) The important part here is you build the

[1:18:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4707s) feedback mechanism on purpose and you

[1:18:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4711s) have some judgment over making sure that

[1:18:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4714s) there is some consequential decision

[1:18:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4716s) making on the fact that this input may

[1:18:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4719s) change your whole analysis. H you need

[1:18:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4722s) to be on top of making sure that those

[1:18:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4724s) changes make sense.

[1:18:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4729s) The other thing you can try is on if you

[1:18:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4732s) click on step five erh you can open up a

[1:18:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4737s) new branch conversation like a branched

[1:19:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4740s) conversation I think is the button on

[1:19:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4741s) the chat at the bottom

[1:19:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4743s) here.

[1:19:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4744s) Yeah. Yeah. So you can open a new

[1:19:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4745s) branch.

[1:19:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4748s) Has anyone used the slash

[1:19:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4752s) btw by the way thing on cloud?

[1:19:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4757s) How has been your experience using it?

[1:19:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4760s) Sometimes it works, sometimes it

[1:19:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4762s) doesn't.

[1:19:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4762s) Sometimes it works, sometimes it

[1:19:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4764s) doesn't. And it feels like you're doing

[1:19:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4766s) an aside, but not really an aside. Aside

[1:19:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4768s) it becomes a part of the context, right?

[1:19:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4773s) But do you get a choice? Like maybe I

[1:19:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4776s) have a stupid question in the middle of

[1:19:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4777s) the conversation.

[1:19:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4778s) No, I don't have a choice. I don't

[1:19:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4779s) [laughter] if I ask a question then it

[1:19:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4781s) becomes the part of the session.

[1:19:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4782s) Yeah. Yeah.

[1:19:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4784s) So in here, what we're trying to do, and

[1:19:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4787s) going back to why we're doing it like

[1:19:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4789s) this, we want to show you how it could

[1:19:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4791s) feel if we design something with the

[1:19:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4794s) intent of, well, this really needs to be

[1:19:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4797s) an aside. I have too many stupid

[1:19:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4799s) questions when I'm chatting with AI. So

[1:20:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4801s) many. And it's shameful, but I end up

[1:20:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4804s) having to open a separate chat because I

[1:20:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4806s) don't trust it's not going to

[1:20:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4807s) contaminate what I'm doing. And then I

[1:20:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4809s) will take more time just being like,

[1:20:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4810s) "Hey, this is a stupid question. please

[1:20:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4812s) don't use for anything else is the

[1:20:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4815s) please don't make any mistakes thing

[1:20:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4816s) Philippe was saying

[1:20:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4817s) h doesn't work quite well. So whatever

[1:20:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4820s) you design if you have a feedback

[1:20:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4822s) mechanism make sure that you tailor it

[1:20:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4824s) to how you actually want to use it. Erh

[1:20:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4829s) oops

[1:20:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4830s) cool

[1:20:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4834s) and then as for h the last point the

[1:20:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4838s) last principle will be verification. So

[1:20:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4842s) you want a second pair of eyes and this

[1:20:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4845s) builds on the principle of

[1:20:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4848s) accountability

[1:20:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4850s) because you are accountable for that

[1:20:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4852s) review as well. Erh,

[1:20:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4856s) and maybe that's too abstract, but has

[1:21:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4860s) anyone been in a situation where someone

[1:21:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4864s) h passed the blame on a bad decision to

[1:21:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4867s) you or anyone else just on the basis of

[1:21:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4870s) No, but Franchesco, review it.

[1:21:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4874s) That happens.

[1:21:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4875s) That happens. So, um, we were having a

[1:21:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4878s) funny conversation like a month ago

[1:21:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4880s) where a few a few people are starting to

[1:21:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4884s) have thoughts that the burden of

[1:21:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4888s) something being good or correct with VIP

[1:21:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4890s) coding ends up being in the hands of the

[1:21:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4893s) reviewer in a team that feels really

[1:21:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4895s) demoralizing because you're just being

[1:21:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4898s) given a bunch of stuff that sure looks

[1:21:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4901s) about right, but then it's your

[1:21:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4903s) responsibility responsibility ultimately

[1:21:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4906s) to make sure that it is right.

[1:21:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4908s) This whole principle of verification is

[1:21:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4910s) just building more feedback loops so you

[1:21:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4913s) don't have blind spots but without

[1:21:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4916s) relegating any of the accountability.

[1:21:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4918s) Ultimately, this is just extra

[1:22:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4920s) information from a different source to

[1:22:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4922s) avoid biases and avoid

[1:22:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4925s) you getting stuck on your own thinking

[1:22:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4927s) and your own processes. But it doesn't

[1:22:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4930s) remove any of the responsibility or

[1:22:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4932s) accountability.

[1:22:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4935s) Um

[1:22:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4937s) yeah, funny enough, expert experts often

[1:22:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4940s) do not check every cell. And this is

[1:22:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4943s) more on like my experience working with

[1:22:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4946s) financial finance teams in data.

[1:22:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4951s) They could open a spreadsheet and just

[1:22:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4953s) look at the you know a few cells they

[1:22:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4956s) know will give you a ballpark total and

[1:22:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4958s) if it doesn't feel right they will let

[1:22:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4960s) you know. It's like, how did we spend

[1:22:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4963s) this much money on marketing? I don't

[1:22:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4964s) think that's right. Uh, and it's a

[1:22:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4966s) little bit of a gut reaction.

[1:22:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4969s) You don't need a lot of effort. You

[1:22:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4971s) don't need to like redo your plan

[1:22:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4973s) entirely or challenge every decision

[1:22:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4974s) you're made, but you do want to have a

[1:22:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4977s) separate point of reference. Erh,

[1:23:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4982s) I think I I had an example, but I don't

[1:23:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4986s) remember what was it was something about

[1:23:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4989s) a tweet that got really famous about

[1:23:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4992s) Elon Musk having I don't know how many

[1:23:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4993s) trillion dollars and someone saying

[1:23:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4996s) like, well, he could give a million

[1:23:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=4998s) dollars to every person in the world and

[1:23:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5001s) still have money. And he was like,

[1:23:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5004s) wait, what? That doesn't feel right.

[1:23:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5006s) Like you read it and you're like, wait,

[1:23:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5007s) what? Looks plausible. And then you're

[1:23:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5009s) like, "No, no, that's not right." And

[1:23:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5012s) that needed need that needed for you to

[1:23:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5015s) look into the calculations the person

[1:23:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5017s) made is like a little bit gut feeling.

[1:23:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5020s) Um, so just having something review and

[1:23:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5023s) give you a rough answer or some

[1:23:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5026s) challenge if something doesn't look

[1:23:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5027s) right is extremely helpful.

[1:23:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5029s) Sorry.

[1:23:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5037s) supply.

[1:24:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5046s) That is such a good question. Uh I was

[1:24:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5048s) trying to come up with a examples that

[1:24:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5050s) feel more down to earth, but I think

[1:24:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5054s) that's a very conversation to have. A

[1:24:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5057s) good argument is if you're getting

[1:24:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5059s) golden examples of how the data should

[1:24:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5062s) look like, why would your process give

[1:24:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5065s) you something different to that? Like

[1:24:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5066s) why don't you just check with your

[1:24:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5067s) golden examples? And this is something I

[1:24:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5071s) I wanted to abstract away from the world

[1:24:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5074s) of finance. You have a lot of sources of

[1:24:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5076s) information, a lot of them. You always

[1:24:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5079s) have another point of reference outside

[1:24:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5081s) of your data. So if some sources you're

[1:24:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5084s) using are claiming I don't know whatever

[1:24:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5087s) fact a number you have the internet you

[1:24:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5091s) should always try and

[1:24:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5094s) enhance the signal you're getting by

[1:24:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5097s) looking out look at all the resources

[1:24:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5099s) that are not the ones that I'm using and

[1:25:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5102s) you avoid that feedback loop where you

[1:25:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5105s) only have the information you have

[1:25:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5107s) available you only have your golden

[1:25:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5108s) examples and everything looks right to

[1:25:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5110s) me

[1:25:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5112s) Always try to get a little bit creative

[1:25:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5113s) when building a harness to try and guide

[1:25:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5117s) which other uh points of reference it

[1:25:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5120s) could use to help you make a better

[1:25:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5122s) decision or to know if something is not

[1:25:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5123s) quite right. Does that

[1:25:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5126s) Yeah.

[1:25:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5138s) And I guess we generally try to work

[1:25:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5141s) with

[1:25:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5142s) whoever it is to like understand what

[1:25:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5145s) would be the best set of things that

[1:25:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5147s) agent should use.

[1:25:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5150s) So trying to understand how that would

[1:25:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5156s) have been the optimal

[1:26:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5161s) to give you another example that could

[1:26:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5164s) be useful. I think the golden examples

[1:26:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5167s) give you your ground truth for making

[1:26:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5170s) sure that you are within the right space

[1:26:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5174s) but no domain is isolated. So what

[1:26:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5177s) happened to me a lot of times when I was

[1:26:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5179s) a BI analyst in some like company is

[1:26:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5183s) that I thought what I wrote was really

[1:26:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5187s) good. So my query was good. The number I

[1:26:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5189s) got seemed reasonable and I was checking

[1:26:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5192s) everything and nothing looked off. And

[1:26:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5195s) then when I presented the dashboard to

[1:26:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5197s) some stakeholder immediately went like

[1:26:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5200s) that's not correct and I was like what

[1:26:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5202s) do you mean? Do you know SQL more than I

[1:26:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5205s) do? Did you do do the analysis with me?

[1:26:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5208s) And turns out this person was from a

[1:26:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5211s) separate team and they just had an idea

[1:26:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5214s) of how many whatever sales were made in

[1:26:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5216s) that year. So they knew instinctively

[1:27:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5220s) that number can't be right because it

[1:27:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5222s) means my information in this other place

[1:27:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5224s) is incorrect. So it's not if we

[1:27:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5227s) translate that to the agentic world the

[1:27:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5230s) golden examples I will give my agent in

[1:27:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5232s) my hardness will be based on like biases

[1:27:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5236s) and selection bias and so on based on

[1:27:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5239s) the domain and I picked them for the

[1:27:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5241s) domain and I think they're good for the

[1:27:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5243s) domain but when you look at a slightly

[1:27:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5246s) tangential domain you may find out that

[1:27:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5250s) actually this is nonsensical like this

[1:27:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5252s) number is wrong just by looking at this

[1:27:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5254s) other thing next to me I I don't even

[1:27:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5256s) need to look again into my calculations.

[1:27:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5260s) Cool.

[1:27:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5262s) Erh. So there is a concept called thing

[1:27:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5267s) slicing

[1:27:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5269s) where you have thin slices of your

[1:27:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5273s) anecdotal experience and you know what

[1:27:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5276s) things feel right or feel wrong based on

[1:27:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5279s) your experience in your domain. Whatever

[1:28:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5282s) analysis you're doing. the model will

[1:28:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5284s) have an average of everyone else's

[1:28:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5287s) experiences. So when you're building

[1:28:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5289s) your hardness, you need to make sure

[1:28:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5291s) that you can uh have a feedback

[1:28:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5294s) mechanism so that you can verify from

[1:28:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5299s) what the model thinks it knows all the

[1:28:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5302s) way to what you think you know. And if

[1:28:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5306s) you flip that convoluted sentence I just

[1:28:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5309s) said it's the it's how we started the

[1:28:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5313s) whole framework and all the principles

[1:28:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5315s) where you're trying to get to that

[1:28:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5317s) shared domain shared context. You're

[1:28:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5320s) essentially translating your language

[1:28:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5323s) into the training data set for the

[1:28:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5325s) model. You're just working with the

[1:28:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5327s) model to be like okay this is how I

[1:28:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5329s) understand the problem or how my company

[1:28:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5332s) does this. and then the model is helping

[1:28:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5334s) like okay this is how would I fit it in

[1:28:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5337s) my training data set here we're doing

[1:28:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5339s) the opposite we're just saying okay I

[1:29:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5342s) think we got to a point where you can

[1:29:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5343s) make a reasonable judgment over this

[1:29:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5345s) domain but let's have a feedback loop so

[1:29:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5349s) I can tell you from my experience if

[1:29:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5350s) this looks entirely off or if it looks

[1:29:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5353s) reasonable so

[1:29:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5356s) um

[1:29:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5358s) yes it's probably fine may look nothing

[1:29:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5360s) like yours

[1:29:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5362s) I think is a way good way to describe it

[1:29:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5365s) like yeah it looks about right and then

[1:29:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5368s) turns out your company does care about

[1:29:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5370s) that 5% difference it makes a world of a

[1:29:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5373s) difference when you're building hardness

[1:29:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5375s) for your team that you're aware that

[1:29:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5377s) that 5% it will be a deal breaker for

[1:29:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5380s) the whole team.

[1:29:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5382s) Um there's some stuff that happens if

[1:29:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5385s) you don't have um a way to do this. So

[1:29:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5389s) checking everything line by line or

[1:29:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5391s) checking nothing or you know no sense of

[1:29:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5394s) what about right looks like and if you

[1:29:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5397s) don't have that point of reference your

[1:29:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5398s) harness won't have it either. So you

[1:30:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5400s) need to build those kind of like

[1:30:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5402s) external validation points.

[1:30:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5407s) I'm going to skip through some of this.

[1:30:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5413s) And for the verification, it's those

[1:30:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5415s) notes in the H decision as well where

[1:30:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5421s) we're adding uh an external point of

[1:30:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5424s) reference to some of the stuff the model

[1:30:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5427s) thought it knew.

[1:30:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5437s) It's kind of satisfac satisfactory when

[1:30:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5439s) the thing just like

[1:30:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5441s) the branch go comes back. Yeah, exactly.

[1:30:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5444s) And again, I think it creates a a trace

[1:30:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5447s) to to um be able to like the the thin

[1:30:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5452s) slice and the quick instinctive look

[1:30:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5454s) also comes from from this because you

[1:30:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5456s) understand the thought process that the

[1:30:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5458s) agent went through and not only the the

[1:31:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5460s) output only.

[1:31:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5466s) I go back. Yes.

[1:31:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5468s) So, as with the other examples, if

[1:31:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5472s) you're still following along, there's a

[1:31:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5474s) few things you can try. So, you can open

[1:31:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5476s) a branch uh on a tagged challenge.

[1:31:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5480s) That's basically a separate agent that

[1:31:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5483s) we tasked with reviewing the plan,

[1:31:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5485s) reviewing the assumptions, and then

[1:31:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5487s) doing some healthy challenge. Does this

[1:31:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5490s) read reasonable? Um so just click it

[1:31:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5494s) open it up and then you will see how

[1:31:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5498s) just Philippe did it will join back into

[1:31:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5500s) the trunk and actually change the plan.

[1:31:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5502s) So there is a consequence

[1:31:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5505s) and we think this is another important

[1:31:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5508s) part of being intentional with the

[1:31:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5511s) design of your harnesses.

[1:31:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5514s) We could have left it as LLM as a judge.

[1:31:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5517s) So for every decision, for every plan,

[1:32:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5520s) whatever, spin another uh agent and then

[1:32:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5523s) do a review and call it a day. The

[1:32:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5526s) problem is if there's no feedback loop,

[1:32:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5528s) our financial decision or decision

[1:32:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5530s) whatsoever doesn't improve at all. So

[1:32:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5532s) that's part of the design, the taste you

[1:32:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5535s) need to build h to create fitforpurpose

[1:32:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5539s) harnesses for whatever use cases you

[1:32:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5541s) have. Then the other thing you can try

[1:32:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5543s) is opening another and just clicking on

[1:32:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5546s) dismiss. And we did that on purpose

[1:32:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5549s) because sometimes as I was saying before

[1:32:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5551s) some conversations don't actually change

[1:32:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5553s) anything or not material to your process

[1:32:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5557s) and you will know better. So you need to

[1:32:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5560s) have that kind of like point of

[1:32:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5561s) reference for validating things. Um

[1:32:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5566s) so just to summarize and leave this on

[1:32:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5568s) the screen for a moment. Um, if you're

[1:32:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5570s) still playing around with the harness,

[1:32:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5572s) uh, the framework for building really

[1:32:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5576s) good harnesses that you can trust with

[1:32:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5578s) financial data with money, it's built on

[1:33:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5582s) these five principles. So having a

[1:33:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5584s) shared domain, having consistency,

[1:33:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5586s) showing the receipts, keeping your

[1:33:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5588s) accountability, and making sure that you

[1:33:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5591s) create intentional loops for

[1:33:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5593s) verification outside of what you think,

[1:33:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5596s) you know.

[1:33:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5598s) Erh,

[1:33:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5603s) what? No confetti. Disappointing.

[1:33:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5607s) Okay. So, there's a few cases where

[1:33:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5612s) we think it's important to define. Oh,

[1:33:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5615s) sorry. You had your hand up.

[1:33:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5617s) Yes.

[1:33:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5618s) So, I just confirm my understanding. You

[1:33:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5622s) brought us a framework principle. You

[1:33:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5625s) can build your harness upon it. And you

[1:33:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5628s) also brought one harness as an example.

[1:33:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5631s) So, I'm I'm thinking about when should I

[1:33:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5635s) build other harness, how generic or how

[1:33:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5639s) specific the harness should be because

[1:34:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5641s) you solve problems with the specific

[1:34:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5644s) harness and like for example how many

[1:34:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5648s) harnesses do you have in your work? It's

[1:34:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5651s) just one and you solve everything that

[1:34:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5653s) you have for this in this harness or you

[1:34:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5656s) have multiple more specific harness. How

[1:34:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5660s) generic and how often do you build one?

[1:34:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5662s) Like how much effort do you put in

[1:34:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5664s) building one? you make one every month

[1:34:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5666s) or you make one every six months.

[1:34:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5669s) That's a that's a great question and

[1:34:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5670s) there's no clear answer of course. Um I

[1:34:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5673s) think it's we could we could trace as

[1:34:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5676s) you said like the two ends of the

[1:34:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5678s) spectrum as I just use cloth code or or

[1:34:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5682s) codeex for everything and I just embed

[1:34:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5684s) everything into skills instructions so

[1:34:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5686s) on so forth and on the other end of the

[1:34:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5688s) spectrum is for every small new thing I

[1:34:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5692s) want to do I'll build it from the ground

[1:34:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5694s) up and like of course that will get

[1:34:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5695s) tiring at some point because like you're

[1:34:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5698s) spending more time creating your tools

[1:35:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5700s) than actually using the tools sometimes

[1:35:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5702s) Um again there's no clear answer. I

[1:35:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5704s) think it comes to

[1:35:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5707s) um when you feel the use case is worth

[1:35:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5709s) enough. So you want reproducibility,

[1:35:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5713s) you want it to feel few the results you

[1:35:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5717s) are getting using the pre-built

[1:35:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5719s) harnesses are very frequently getting it

[1:35:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5722s) wrong because the value you get from

[1:35:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5725s) building the harness is you get more

[1:35:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5727s) control again with the deterministic bit

[1:35:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5731s) together with the nondeterminism. Right?

[1:35:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5734s) Um the reason why we're talking about

[1:35:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5736s) the the principles in the case in the um

[1:35:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5739s) scenario of a financial analysis is

[1:35:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5741s) because if we do this type of use case

[1:35:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5746s) in a harness that doesn't have it we'll

[1:35:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5749s) probably have some of the f pitfalls

[1:35:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5751s) that Franchesco talks through each of

[1:35:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5753s) the steps. So then in this case it makes

[1:35:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5756s) sense for um to avoid these pitfalls we

[1:36:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5760s) are seeing over and over and over again

[1:36:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5762s) to build the thing. But again, I think

[1:36:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5764s) it comes down to how many users you

[1:36:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5767s) believe will go through that use case,

[1:36:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5770s) how critical is the problem that you

[1:36:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5772s) were seeing. So like if if there's a

[1:36:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5774s) non-b businessiness critical um task

[1:36:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5776s) that you were solving, you were probably

[1:36:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5778s) fine to have a human in the loop

[1:36:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5780s) reviewing the outputs that you get and

[1:36:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5782s) steering the agent even if that takes a

[1:36:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5784s) few extra turns. But if you're putting

[1:36:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5786s) this in the hands of a uh internal user,

[1:36:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5790s) for instance, let's say you were

[1:36:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5791s) building something for your team of

[1:36:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5793s) credit analysts, um you're going to have

[1:36:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5796s) 50 people using something to support key

[1:36:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5800s) company decisions. Yeah, you absolutely

[1:36:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5802s) don't want a lot of variability on that.

[1:36:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5806s) So again, there's no clearcut I would

[1:36:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5808s) say, but these are the things I would

[1:36:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5809s) consider in in making the decision.

[1:36:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5813s) Maybe I can help with uh the

[1:36:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5815s) counterfactual. So, we think this is a

[1:36:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5818s) good recipe specifically for financial

[1:37:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5821s) harnesses because there's a lot at

[1:37:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5824s) stake. You want a lot of feedback loops.

[1:37:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5826s) There's very specific things about those

[1:37:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5828s) use cases that somehow we're

[1:37:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5831s) representing here even at the personal

[1:37:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5834s) level or the company level. But imagine

[1:37:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5837s) you want to do you want to use AI just

[1:37:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5840s) to produce like social media content and

[1:37:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5843s) maybe one of your use cases is just

[1:37:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5846s) flooding Instagram and Tik Tok with AI

[1:37:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5848s) generated videos like everyone is doing.

[1:37:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5851s) There's some steps here that you

[1:37:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5852s) probably don't care that much. Evidence

[1:37:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5854s) gathering.

[1:37:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5857s) What for? H verification or

[1:37:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5860s) accountability even less so. So this is

[1:37:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5863s) specifically geared towards like

[1:37:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5866s) financial use cases and it can be

[1:37:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5868s) extended to different levels of it but

[1:37:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5872s) everything hinges on the fact that is a

[1:37:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5874s) trust system. H this recipe may not be

[1:37:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5878s) as useful if your use cases are not

[1:38:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5880s) necessarily like that and in terms of

[1:38:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5882s) how many harnesses you want to build. I

[1:38:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5886s) think Filipe said it better, but

[1:38:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5891s) you can easily fall victim for doing a

[1:38:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5893s) lot of axe sharpening and not actually

[1:38:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5896s) using the axe. H So, at some point, if

[1:38:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5899s) you're trying to use something and

[1:38:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5902s) you're not getting the right results,

[1:38:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5904s) you can try adding a couple of skills, a

[1:38:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5906s) couple of commands, a couple of extra

[1:38:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5908s) things, still not getting the results or

[1:38:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5910s) the experience you want. You're finding

[1:38:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5912s) yourself copy pasting text from one chat

[1:38:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5914s) to another, opening a new session. It

[1:38:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5917s) feels convoluted harness. So you all the

[1:38:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5920s) repeated things that you had to do three

[1:38:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5923s) five times. Okay. Just once and you

[1:38:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5926s) design it around it.

[1:38:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5929s) Yes. Yes.

[1:38:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5931s) Some more related question. So for your

[1:38:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5933s) wedding planning task, right? I think

[1:38:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5936s) one of like one of the naive way of

[1:38:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5938s) looking at this is like maybe spin of

[1:39:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5940s) agent, right? agents, venue agents. Uh,

[1:39:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5944s) have you have you have you tried that

[1:39:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5946s) and what's the outcome or like your

[1:39:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5948s) opinion on that?

[1:39:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5949s) I will start by confessing that I have

[1:39:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5952s) the most expensive cloth subscription

[1:39:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5954s) and I do max it out and not in a good

[1:39:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5957s) way as in Anthropic is definitely on the

[1:39:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5960s) on the red because of me.

[1:39:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5963s) I tried research everything for all the

[1:39:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5966s) venue, the flowers, the wedding, do the

[1:39:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5969s) planning, do the different like

[1:39:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5971s) scenarios for how much we will have to

[1:39:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5973s) pay etc. And what I found is that in

[1:39:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5976s) many cases it was good enough to give me

[1:39:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5979s) the list of the top five photographers

[1:39:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5981s) in Madrid with the new web search with

[1:39:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5984s) like a refined h tooling etc. Worked

[1:39:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5987s) like a charm. didn't have to do anything

[1:39:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5988s) else just spinning and having

[1:39:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5991s) comparisons and even reaching out to

[1:39:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5993s) them. But for some more nuance use cases

[1:39:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5995s) where I do want to have more control and

[1:39:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5997s) predictability and this is linking to

[1:39:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=5999s) the financial nature of them I did find

[1:40:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6002s) myself being like I need to build

[1:40:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6004s) something robust. I'm not recalculating

[1:40:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6007s) this again. Uh and to to make the pain

[1:40:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6011s) very real. I remember myself one day

[1:40:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6015s) because I was in the car and I was like,

[1:40:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6017s) "Yeah, I can dictate some stuff, voice

[1:40:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6019s) mode, and then I was driving and it was

[1:40:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6021s) about an hour drive back and forth, you

[1:40:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6024s) know, an hour and an hour and I re had I

[1:40:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6028s) had the same conversation three times in

[1:40:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6030s) that t in that trench just because every

[1:40:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6033s) time I was trying to do some financially

[1:40:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6035s) related or adjacent planning, something

[1:40:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6038s) will go wrong and I'll be like

[1:40:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6041s) so yes, that helps.

[1:40:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6047s) So getting back to your example of being

[1:40:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6049s) a BI, right? Like um

[1:40:53](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6053s) I don't think back in the day there were

[1:40:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6056s) people who were validating your SQL

[1:40:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6058s) queries, right?

[1:41:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6060s) Um so this seems like we are it's like

[1:41:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6065s) next level of SQL queries, right? Now

[1:41:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6067s) you're building tools because harnesses

[1:41:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6069s) are easier to build and stuff. Have you

[1:41:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6071s) found

[1:41:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6072s) ways to kind of make this point clear to

[1:41:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6076s) leadership in in conversations that like

[1:41:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6078s) because because to me it feels every

[1:41:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6080s) time there is an AI there is an

[1:41:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6082s) expectation that it will be 100% correct

[1:41:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6085s) like do you see what I'm getting at

[1:41:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6088s) so is your question about managing the

[1:41:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6091s) expectations so that when people do

[1:41:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6094s) agentic self surf

[1:41:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6096s) it's not enic sales more so like let's

[1:41:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6099s) say you build this and and you talking

[1:41:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6102s) to your senior leadership to make a

[1:41:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6103s) financial decision in your company,

[1:41:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6105s) right? And they start drilling you and

[1:41:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6107s) drilling you and and the moment that you

[1:41:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6109s) say, "Oh, I use AI and harnesses,"

[1:41:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6111s) they're like, "Well, this has to be 200%

[1:41:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6114s) correct or something like you know what

[1:41:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6116s) I'm saying? Have you found the way to to

[1:41:58](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6118s) kind of make sure that they understand

[1:42:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6120s) this is still SQL queries?"

[1:42:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6122s) Bit of a spoiler, but there's a talk

[1:42:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6125s) tomorrow. [laughter]

[1:42:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6128s) I don't know. Do you have any?

[1:42:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6130s) [clears throat]

[1:42:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6130s) Yeah. Yeah. No, I think um

[1:42:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6133s) it's

[1:42:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6137s) with the AI first world, we sometimes as

[1:42:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6141s) you were saying forget that what's that

[1:42:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6143s) that's only a two and a means to an end

[1:42:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6145s) be before like what's what's going on

[1:42:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6147s) right so I think what we would have

[1:42:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6150s) noticed and again from Monzo's

[1:42:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6152s) experience we've been approaching it

[1:42:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6154s) from an highly impactbased um point of

[1:42:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6157s) view so it's not about we should use AI

[1:42:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6161s) for this because AI can something

[1:42:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6164s) something something. It's like what's

[1:42:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6167s) the outcome we're looking for here? What

[1:42:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6169s) how do we measure success here? Are we

[1:42:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6171s) looking for less uh back and forth with

[1:42:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6174s) a BI team? Uh we don't even have a BI

[1:42:56](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6176s) team, but you get you get what I mean.

[1:42:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6177s) Um to validate a new dashboard. Okay.

[1:43:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6181s) So, we can track that and actually see

[1:43:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6183s) if AI can support us in in generating

[1:43:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6186s) that outcome. um are we looking for um

[1:43:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6190s) less I don't know internal ask tickets

[1:43:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6193s) because the number looks wrong. Okay, so

[1:43:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6196s) let's track that and then the approach

[1:43:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6199s) that we do the way we do that doesn't

[1:43:22](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6202s) really matter to a point if that makes

[1:43:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6204s) sense because it will be again a tool

[1:43:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6207s) that can get us to that um to that

[1:43:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6210s) result probably faster with better guard

[1:43:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6214s) rails and so on and so forth. So I think

[1:43:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6216s) it's less it's less about gaining trust

[1:43:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6220s) on the implementation on the AI itself

[1:43:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6224s) but more about aligning on what's the

[1:43:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6226s) outcome we're trying to drive with the

[1:43:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6228s) AI.

[1:43:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6231s) I think we had another one there.

[1:43:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6235s) My question is more you have show a

[1:43:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6237s) harness that has some complexity right

[1:44:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6240s) you have a UI you have databases say but

[1:44:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6244s) I have also build processes that invoke

[1:44:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6248s) some of these steps but they are all at

[1:44:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6250s) skill right that it ask you questions it

[1:44:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6255s) runs the number based on a Python script

[1:44:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6257s) and so on so my question is more like

[1:44:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6260s) where you see the boundaries between

[1:44:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6261s) like having a skill that does all these

[1:44:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6263s) creating like a separate tool or have

[1:44:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6266s) have you tried that?

[1:44:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6267s) Absolutely. Yeah, absolutely. Um I think

[1:44:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6270s) it it again very similarly to to what we

[1:44:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6272s) were saying about your user base and the

[1:44:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6275s) reproducibility you want is when this

[1:44:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6278s) becomes critical enough that the

[1:44:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6280s) tradeoffs we are getting with the

[1:44:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6283s) nondeterministic approach are are not uh

[1:44:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6287s) we're not willing to have them. I

[1:44:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6288s) usually describe the the skill command

[1:44:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6291s) like the the promptbased approach as hey

[1:44:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6295s) please follow all of this and make no

[1:44:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6297s) mistakes please please please like again

[1:45:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6300s) because this is what's going on right we

[1:45:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6302s) are we are giving a a lot of organized

[1:45:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6304s) very well organized instructions to a

[1:45:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6307s) machine and hoping that it it will

[1:45:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6309s) perform these instructions agents are

[1:45:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6311s) good enough nowadays they will probably

[1:45:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6314s) perform your instructions 99%

[1:45:17](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6317s) of the time. Uh, but what about that 1%?

[1:45:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6321s) What about an edge case that makes

[1:45:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6325s) things don't work? What about if you

[1:45:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6327s) distribute this to your internal users

[1:45:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6330s) and your internal user has a instruction

[1:45:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6333s) in their user level cloud.mmd file that

[1:45:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6337s) goes against what you are telling your

[1:45:39](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6339s) agent to do. You see my point is like um

[1:45:42](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6342s) there are many when you when you start

[1:45:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6345s) distributing it to a number of users and

[1:45:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6347s) use cases things can get messy

[1:45:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6351s) in places you are not able to predict

[1:45:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6354s) and then you can find yourself I we have

[1:45:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6357s) a at Moz we have an internal like

[1:45:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6359s) contribution for agent extensions where

[1:46:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6362s) multiple teams can well I've come across

[1:46:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6364s) something that works with me let me

[1:46:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6365s) share it internally um so like plugins

[1:46:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6368s) skills and so on and very frequent ly we

[1:46:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6370s) when reviewing the instructions we see

[1:46:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6372s) things trying to cover for if the user

[1:46:15](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6375s) tries to do this tell the user back that

[1:46:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6378s) this skill does not support that and you

[1:46:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6381s) see like it's again trying to cover for

[1:46:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6383s) many things that with the determinism

[1:46:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6386s) and when I say determinism is the

[1:46:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6388s) backend code like the python the

[1:46:29](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6389s) whatever we are using uh that the

[1:46:31](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6391s) harness provides that out of the like

[1:46:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6395s) out of the box is something we can cover

[1:46:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6398s) or

[1:46:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6403s) anyone has any other comments,

[1:46:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6405s) questions.

[1:46:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6409s) All right,

[1:46:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6409s) cool.

[1:46:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6412s) Where is the thingy? [laughter]

[1:46:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6414s) Never mind. Never mind. Never mind.

[1:47:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6420s) Cool, cool, cool, cool. Skipping ahead.

[1:47:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6421s) Skipping ahead. Ah, fun. Erh, I wanted

[1:47:05](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6425s) to make a joke and I completely missed

[1:47:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6427s) it for clicking this thing too quickly.

[1:47:09](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6429s) But finance is funny. And I say funny

[1:47:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6433s) because I find it to be a super

[1:47:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6436s) interesting domain with a the added

[1:47:19](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6439s) advantage that back in the day finance

[1:47:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6443s) felt like just working with

[1:47:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6445s) spreadsheets, very like boring work,

[1:47:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6448s) adding numbers here and there, formulas,

[1:47:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6450s) whatever. If you ask any accountant is

[1:47:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6453s) all about creativity and interpretation

[1:47:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6456s) because they will come up with the

[1:47:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6457s) weirdest things and I think in that

[1:47:40](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6460s) level of ambiguity AI is really good

[1:47:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6464s) really really good you have some bits

[1:47:46](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6466s) that are the spreadsheet and then you

[1:47:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6469s) realize some of the cash flow didn't

[1:47:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6471s) quite make sense because some of the

[1:47:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6472s) revenue was attributed a month later and

[1:47:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6475s) you're like ah creativity. So because we

[1:47:59](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6479s) are in this space of finance and

[1:48:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6482s) depending on where is your mind mindset

[1:48:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6484s) at you could look at a problem as

[1:48:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6487s) something very simple that you can just

[1:48:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6488s) get an agent to fix within 10 minutes or

[1:48:11](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6491s) you could look at it as it actually is

[1:48:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6494s) which is a mixture of things that need

[1:48:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6496s) to run reliably 100% of the time and a

[1:48:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6500s) lot of interpretation and freedom of

[1:48:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6503s) creativity in between the lines.

[1:48:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6506s) So yeah, sorry, very poorly delivered

[1:48:28](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6508s) joke, but that's what I wanted to go

[1:48:30](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6510s) through. Uh, and the bottom line being

[1:48:34](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6514s) the design of your harnesses needs to be

[1:48:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6517s) intentional to increase trust.

[1:48:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6521s) Don't assume that oh because this is

[1:48:43](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6523s) running on AI is going to be what was

[1:48:45](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6525s) the the quote 200% accurate.

[1:48:49](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6529s) It will likely not unless you design all

[1:48:51](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6531s) the mechanisms so that whatever process

[1:48:55](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6535s) runs you know you can trust the results

[1:48:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6537s) to a degree. Um and yeah models are

[1:49:01](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6541s) going to get better. We have seen the

[1:49:03](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6543s) last six months nine months in saying

[1:49:06](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6546s) I'm getting a little bit of whiplash

[1:49:08](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6548s) even with you know how fast things are

[1:49:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6550s) going but they still have to earn the

[1:49:13](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6553s) trust. it's not a 100% right and in when

[1:49:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6556s) we're talking about money 1% wrong

[1:49:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6558s) sometimes it's a lot of money

[1:49:21](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6561s) so yeah we kind of skipped into the

[1:49:24](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6564s) questions but if anyone has anything

[1:49:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6565s) else

[1:49:26](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6566s) yeah just really quick like in the UI

[1:49:27](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6567s) how do you demonstrate trust like

[1:49:32](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6572s) you just ask it show me how you

[1:49:33](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6573s) calculated this or does it show you the

[1:49:35](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6575s) code

[1:49:36](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6576s) can you show an example like how do you

[1:49:37](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6577s) prove it

[1:49:38](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6578s) the intent was that it was a aggregation

[1:49:41](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6581s) of everything so All the steps in the

[1:49:44](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6584s) framework should have felt like oh but I

[1:49:47](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6587s) know where this is coming from but I

[1:49:48](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6588s) know this is running as code but I know

[1:49:50](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6590s) this is tested. So it should have been

[1:49:52](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6592s) an experience to kind of like immerse

[1:49:54](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6594s) into this feels trustworthy. But again

[1:49:57](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6597s) it's one implementation we came up with.

[1:50:00](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6600s) So not not super direct.

[1:50:02](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6602s) Yeah. But I think the the takeaway is

[1:50:04](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6604s) there won't be a single thing that will

[1:50:07](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6607s) work to earn the user's trust at any

[1:50:10](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6610s) point. So I think that's for some people

[1:50:12](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6612s) the verification will be more important.

[1:50:14](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6614s) Some others are a bit more skeptic and

[1:50:16](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6616s) they want to see the actual calculations

[1:50:18](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6618s) and numbers and so on. So I think again

[1:50:20](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6620s) that the process guides and covers for a

[1:50:23](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6623s) multitude of u of possible

[1:50:25](https://www.youtube.com/watch?v=ATMcd9tkhhc&t=6625s) possibilities.
