AI Engineer World's Fair 2024
Which Jobs Can Be Replaced Today
Loading slides…
Read the talk
Which Jobs Can Be Replaced Today?
Start with the work an agent can pick up without being asked, then build its capabilities through better prompts, explicit state, fine-tuning and feedback.
From a talk by Fryderyk Wiatrowski and Peter Albert
Before you start: Familiarity with language-model prompts, tool calls and basic Python dictionaries will help with the implementation examples.
What would you hand to a reliable agent?
If you could hire a fully reliable, fast browser agent today, what would you ask it to handle? The audience’s answers are familiar: travel organization, expenses and calendars. Reliability and speed are assumptions in this opening question, leaving the workflow problem exposed: how should an agent take responsibility for that work?
Fryderyk Wiatrowski’s starting point is that sometimes you should not need to prompt the agent at all. A good employee receives onboarding and initial training, then identifies tasks independently instead of asking every morning what to do. The same expectation suggests an agent interface organized around ongoing responsibility, with less need for constant oversight.
That changes the product question. A box for one-off requests can demonstrate an agent’s capabilities, but it leaves the user responsible for noticing, describing and initiating every task. An independent agent needs a way to discover work inside the workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Automate the work around the valuable decision
Begin by separating activities that multiply your effectiveness from the administrative work surrounding them. For a founder, hiring is a high-leverage activity. Fryderyk illustrates the leverage by imagining hiring 10 people smarter than himself who could do the work 10 times better. Searching LinkedIn and coordinating meetings surround that valuable activity, but need not consume the founder’s attention in the same way.
An executive assistant can absorb meeting coordination and similar work. Fryderyk gives an illustrative salary of 50–70K, without specifying a currency or pay period. But the immediate need may simply be to arrange meetings, rather than to acquire an assistant’s entire bundle of skills. Start with that specific problem and give an agent responsibility for it.
The Jace AI example makes this concrete: CC the agent into an email conversation, and it handles meeting setup. In the demonstration, Jace replies to an investor interested in Zeta Labs using the founder’s availability. Fryderyk describes this narrow flow as needing calendar and availability access, without browser or tool interaction; he immediately adds that most other tasks do require integrations and tools. The CC flow is the interface shown in the recording, rather than a claim about today’s product interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the reactive layer
A useful first division is between reactive work, where an event initiates a reasonably well-defined procedure, and proactive work, where deciding what to do is itself difficult. A customer-support refund request fits the first category:
- Receive an email asking for a refund.
- Inspect application-use logs.
- Decide whether the customer qualifies under the refund rules.
- Use a Stripe integration to issue the refund when appropriate.
A founder’s job is harder to capture in a simple rule set, but it still contains this kind of reactive layer. The prediction is that reactive jobs will be automated first because their triggers and responses are easier to specify.
Once the rules and initial training are in place, each incoming event can initiate work without a new prompt. Complexity still limits how far that independence should extend. The talk claims that agents of the time could reliably perform hundreds to roughly 1,000 steps, then immediately warns that they still could not be fully trusted. No task definition, success threshold or evaluation protocol accompanies that estimate.
The proposed implementation starts with a pool of triggers: Slack messages, emails and phone calls. Rules determine which events an agent should pick up and what actions it should prepare. The agent can suggest a solution and do preliminary browser work, then ask for approval before performing the consequential action. This shifts the human’s role from initiating every task to reviewing a prepared proposal; preparation and execution remain distinct.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a trigger pool to browser actions
The boundary between reactive and proactive work is not absolute. A founder responding to a macroeconomic shift is reacting too, but the event is unusual and there are fewer established response templates than in customer support. More descriptive rules could cover a wider range of situations, gradually reaching work currently considered proactive.
The larger proposal is an aggregator that brings together market movements, emails, phone calls, Slack messages and Linear issues. Improving foundation-model reasoning would make more complex rules usable, allowing the system to decide what deserves action. A practical starting point is much narrower: let agents pick up tasks from Linear or Slack before attempting to aggregate every possible signal.
Task discovery still needs an execution surface. Browsers offer access to a broad range of actions through the interfaces people already use; Fryderyk contrasts that reach with APIs he describes as often limited or poorly implemented. Making browser agents reliable is therefore a central implementation challenge, which Peter Albert takes up next. The broad aggregator remains a proposed direction, not a demonstrated system that replaces an entire job.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Escalate only when the task requires it
Peter, who previously worked on Meta’s Llama 2 models, organizes agent development as a progression: prompting, cognitive architectures, fine-tuning and finally reinforcement learning. Apply that progression separately to each system according to its complexity and performance requirements.
Peter estimates that prompting and cognitive-architecture changes can take hours, while fine-tuning and reinforcement-learning projects take weeks to months. Each escalation slows iteration and makes subsequent changes more expensive. High performance on a demanding task may justify those costs, but it is useful to exhaust the faster methods first.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the prompt familiar and each decision smaller
Peter begins with model-assisted prompt rewriting. His rationale is that a language model can produce wording that is more predictable to another model—lower-perplexity text—and help expose misunderstandings in the original instructions. He also recommends XML tags to separate instructions from content, pointing to Anthropic’s use of that convention.
The broader heuristic is to stay close to familiar training patterns. For GPT-4, that means thinking about likely OpenAI fine-tuning conventions as well as pretraining data. Prefer common formats such as JSON, XML or Markdown to a custom syntax. Apply the same thinking to the division between system and user messages, and to whether material belongs in one message or several. These are Peter’s design heuristics about likely training distributions, not knowledge of a private training recipe.
Reduce the number of decisions the model must combine at each token. If a tool can accept a meaningful text value, requiring an arbitrary element number such as 5 or 4 adds a mapping step: the model must choose the target and translate that choice into an identifier. A textual target can remove that extra demand. For classification, the corresponding pattern is to choose a broad category first and then a narrower one, rather than jumping directly to the most specific class.
When you know how to approach a problem, put that structure into the instructions. A generic request to think step by step leaves the procedure unspecified. A more useful template can begin by summarizing the problem and then name the particular decisions needed to solve it. The aim is to give the model a usable sequence, not merely ask it to reason harder.
Context selection matters for the same reason. Every irrelevant passage creates another filtering task for the model. Long few-shot examples can often be shortened to the portions that demonstrate the important behavior without reproducing their entire inputs and outputs. Peter recommends starting with approximately 10–20 examples to test each prompt revision, alongside end-to-end evaluations. This gives prompt changes a repeatable check from the beginning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Track state explicitly and make updates easy to express
Once prompting improvements are exhausted, split the task into smaller pieces. A cognitive architecture supplies structure that a single prompt would otherwise have to maintain implicitly:
- State tracking: Record where a long task currently stands so the next call receives useful context.
- Planning and replanning: Generate a plan, revise it as conditions change and decide what comes next.
- Verification: Check work after an action or intermediate result.
- Notes and scratchpads: Preserve information needed by later steps.
Extra calls introduce latency. Where work is independent, parallel execution can preserve the decomposition without making every call wait for the previous one.
The state-update interface should also fit the model’s task. Peter recommends key-value updates resembling Python dictionary assignments: first identify the key, then generate its replacement value. For example, a scheduling agent can record that it has prepared a proposed slot without marking the meeting as booked:
python
notes = {
"task": "schedule investor meeting",
"status": "checking availability",
"proposed_slot": None,
"booked": False,
}
notes["proposed_slot"] = "Tuesday 14:00"
notes["status"] = "awaiting approval"
Here the slot is a teaching value. Updating the notes records a proposal; it does not create a calendar event. The task and booked flag remain unchanged while two targeted fields change.
Record a proposed slot without booking the meeting
Constructed example: The dictionary keys, status strings and Tuesday 14:00 slot are teaching values illustrating the stated key-value update mechanism, not records from the Jace demonstration.
schedule investor meeting
Operation: Set proposed_slot to Tuesday 14:00, then set status to awaiting approval.
task
schedule investor meeting
schedule investor meeting
status
checking availability
awaiting approval
proposed_slot
None
Tuesday 14:00
booked
False
False
The alternative is to regenerate the whole state or document. Peter reports that full rewrites often perform better because they give the model more generation time, at the cost of latency. He points to Cursor’s text rewriting as an example of preferring regeneration when models struggle to produce small diffs.
| Update form | Model’s immediate task | Tradeoff |
|---|---|---|
| Key-value update | Select a key, then a value | Targeted state changes |
| Full rewrite | Regenerate the complete object | More generation and latency |
| Small diff | Express only the precise edit | Harder for the models Peter describes |
The choice is part of the tool design: a compact edit representation is useful only if the model can produce it reliably.
Peter also advises avoiding recursively nested structures when possible, because they add complexity and, in his account, move farther from familiar training patterns. Treat that as a heuristic against unnecessary nesting: current Claude prompting guidance permits XML nesting when it reflects a natural hierarchy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use structure without multiplying failure points
Reasoning templates can encode the known structure of a problem. For visual inputs, Peter suggests an explicit intermediate representation: first have the model describe the image’s key points in text, then reason over that description. His rationale is that models have seen trillions of text tokens but substantially less paired image-text data, making text a more practiced medium for the reasoning step. He presents this as an approach to try, rather than a measured comparison for a named vision task.
Components should also compensate for one another’s errors. If one stage can produce a particular mistake, another should be able to detect or handle it. But every additional component introduces another opportunity for the system to become brittle. Use enough structure to reduce each model call’s burden, while keeping the number of components as small as the task allows. The benefit of decomposition and the cost of coordination have to be considered together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fine-tune on varied, demanding interactions
Fine-tuning becomes relevant when the system still falls short of its performance target or when inference cost needs to come down. Peter’s proposed data-generation method keeps the application in the loop:
- Create templates describing different human roles.
- Instruct a language model to act as a person in one of those roles.
- Let the rest of the application respond normally.
- Collect the resulting interactions as synthetic training data.
The simulator varies the user side of the interaction while exercising the system you actually want to improve. Prompt diversity and difficulty are central to the usefulness of that data.
Peter attributes weaknesses in early Alpaca models partly to prompts that were too easy. His recommendation is to add conditions and complexity even when the eventual task looks simple, giving the model more relationships and constraints to learn from each example. That causal diagnosis is his interpretation; Stanford’s original account does not establish easy prompts as the cause of Alpaca’s weaknesses.
Fine-tuning can also remove intermediate stages from an existing pipeline. Pair the original inputs with the pipeline’s final outputs, then train a model to map directly between them. This distillation skips work that previously required several steps. Peter expects lower latency in exchange for some performance loss, making it a way to trade part of an elaborate architecture’s quality for a faster path through the task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Filter the data, then consider reinforcement learning
Before moving to a more expensive training method, improve the examples used for fine-tuning through rejection sampling or filtering. Peter claims that ordinary fine-tuning can reach GPT-4 performance on a specific task and that filtering can take it further. He supplies no named task, dataset, GPT-4 version or metric for that comparison.
When execution feedback is unavailable, a language-model judge can evaluate the output of an individual model, a larger subsystem or the complete application. Use those judgments to remove examples that likely failed. The judge does not have to be perfect to be useful in Peter’s account: the practical objective is to exclude substantial amounts of poor training data before fitting the model to it.
Reinforcement learning is the final escalation because it can optimize behavior across multiple steps, an especially relevant capability for agents. Execution feedback or language-model judgments of system components can supply the learning signal. Peter reserves this approach for cases where the earlier methods do not work: the setup cost is substantial, and the resulting training investment makes it harder to switch to a different model. The decision to use reinforcement learning therefore includes both the performance target and the cost of changing the system afterward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
An email assistant offering contextual drafts, custom rules and integrations.
The Llama 2 research paper, coauthored by Peter Albert, introducing pretrained and dialogue-tuned models.
Stanford's original account of Alpaca's synthetic instruction data, training recipe, preliminary evaluation and limitations.
Further reading
Cursor explains full-file rewriting, apply-model evaluation and speculative edits for reducing editing latency.
Updates since the talk
Current Claude guidance on prompt structure, examples, XML tags and agent workflows.
Full transcript
Loading transcript…