AI Engineer World's Fair 2024

Which Jobs Can Be Replaced Today

Fryderyk Wiatrowski· Co-founder, Zeta LabsPeter Albert· Co-founder, Zeta Labs19:59

Loading slides…

Read the talk

Which Jobs Can Be Replaced Today?

Start with the work an agent can pick up without being asked, then build its capabilities through better prompts, explicit state, fine-tuning and feedback.

From a talk by Fryderyk Wiatrowski and Peter Albert

Before you start: Familiarity with language-model prompts, tool calls and basic Python dictionaries will help with the implementation examples.

What would you hand to a reliable agent?

If you could hire a fully reliable, fast browser agent today, what would you ask it to handle? The audience’s answers are familiar: travel organization, expenses and calendars. Reliability and speed are assumptions in this opening question, leaving the workflow problem exposed: how should an agent take responsibility for that work?

Fryderyk Wiatrowski’s starting point is that sometimes you should not need to prompt the agent at all. A good employee receives onboarding and initial training, then identifies tasks independently instead of asking every morning what to do. The same expectation suggests an agent interface organized around ongoing responsibility, with less need for constant oversight.

Slide lists human employees, solid onboarding followed by independence, and building UIs for AI employees, with the presenter inset below.Open full source frame
Ideal employees don’t need to be prompted.

That changes the product question. A box for one-off requests can demonstrate an agent’s capabilities, but it leaves the user responsible for noticing, describing and initiating every task. An independent agent needs a way to discover work inside the workflow.

0:330:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:33 · section reference included

Automate the work around the valuable decision

Begin by separating activities that multiply your effectiveness from the administrative work surrounding them. For a founder, hiring is a high-leverage activity. Fryderyk illustrates the leverage by imagining hiring 10 people smarter than himself who could do the work 10 times better. Searching LinkedIn and coordinating meetings surround that valuable activity, but need not consume the founder’s attention in the same way.

An executive assistant can absorb meeting coordination and similar work. Fryderyk gives an illustrative salary of 50–70K, without specifying a currency or pay period. But the immediate need may simply be to arrange meetings, rather than to acquire an assistant’s entire bundle of skills. Start with that specific problem and give an agent responsibility for it.

Slide asks about high-leverage activities versus noise, uses scheduling as an example, and asks whether to hire or simply solve the problem.Open full source frame
How to embed agents into workflows: distinguish high-leverage activities from noise.

The Jace AI example makes this concrete: CC the agent into an email conversation, and it handles meeting setup. In the demonstration, Jace replies to an investor interested in Zeta Labs using the founder’s availability. Fryderyk describes this narrow flow as needing calendar and availability access, without browser or tool interaction; he immediately adds that most other tasks do require integrations and tools. The CC flow is the interface shown in the recording, rather than a claim about today’s product interface.

2:272:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:27 · section reference included

Find the reactive layer

A useful first division is between reactive work, where an event initiates a reasonably well-defined procedure, and proactive work, where deciding what to do is itself difficult. A customer-support refund request fits the first category:

  1. Receive an email asking for a refund.
  2. Inspect application-use logs.
  3. Decide whether the customer qualifies under the refund rules.
  4. Use a Stripe integration to issue the refund when appropriate.

A founder’s job is harder to capture in a simple rule set, but it still contains this kind of reactive layer. The prediction is that reactive jobs will be automated first because their triggers and responses are easier to specify.

Once the rules and initial training are in place, each incoming event can initiate work without a new prompt. Complexity still limits how far that independence should extend. The talk claims that agents of the time could reliably perform hundreds to roughly 1,000 steps, then immediately warns that they still could not be fully trusted. No task definition, success threshold or evaluation protocol accompanies that estimate.

The proposed implementation starts with a pool of triggers: Slack messages, emails and phone calls. Rules determine which events an agent should pick up and what actions it should prepare. The agent can suggest a solution and do preliminary browser work, then ask for approval before performing the consequential action. This shifts the human’s role from initiating every task to reviewing a prepared proposal; preparation and execution remain distinct.

4:454:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:45 · section reference included

From a trigger pool to browser actions

The boundary between reactive and proactive work is not absolute. A founder responding to a macroeconomic shift is reacting too, but the event is unusual and there are fewer established response templates than in customer support. More descriptive rules could cover a wider range of situations, gradually reaching work currently considered proactive.

The larger proposal is an aggregator that brings together market movements, emails, phone calls, Slack messages and Linear issues. Improving foundation-model reasoning would make more complex rules usable, allowing the system to decide what deserves action. A practical starting point is much narrower: let agents pick up tasks from Linear or Slack before attempting to aggregate every possible signal.

Task discovery still needs an execution surface. Browsers offer access to a broad range of actions through the interfaces people already use; Fryderyk contrasts that reach with APIs he describes as often limited or poorly implemented. Making browser agents reliable is therefore a central implementation challenge, which Peter Albert takes up next. The broad aggregator remains a proposed direction, not a demonstrated system that replaces an entire job.

Slide titled “The path forward” lists trigger aggregation, improving AI cognitive abilities, Zeta Labs’ aggregator approach, and the question of agents acting.Open full source frame
The path forward: aggregate triggers and improve general trigger-handling rules.
7:287:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:28 · section reference included

Escalate only when the task requires it

Peter, who previously worked on Meta’s Llama 2 models, organizes agent development as a progression: prompting, cognitive architectures, fine-tuning and finally reinforcement learning. Apply that progression separately to each system according to its complexity and performance requirements.

Peter estimates that prompting and cognitive-architecture changes can take hours, while fine-tuning and reinforcement-learning projects take weeks to months. Each escalation slows iteration and makes subsequent changes more expensive. High performance on a demanding task may justify those costs, but it is useful to exhaust the faster methods first.

9:349:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:34 · section reference included

Make the prompt familiar and each decision smaller

Peter begins with model-assisted prompt rewriting. His rationale is that a language model can produce wording that is more predictable to another model—lower-perplexity text—and help expose misunderstandings in the original instructions. He also recommends XML tags to separate instructions from content, pointing to Anthropic’s use of that convention.

The broader heuristic is to stay close to familiar training patterns. For GPT-4, that means thinking about likely OpenAI fine-tuning conventions as well as pretraining data. Prefer common formats such as JSON, XML or Markdown to a custom syntax. Apply the same thinking to the division between system and user messages, and to whether material belongs in one message or several. These are Peter’s design heuristics about likely training distributions, not knowledge of a private training recipe.

Reduce the number of decisions the model must combine at each token. If a tool can accept a meaningful text value, requiring an arbitrary element number such as 5 or 4 adds a mapping step: the model must choose the target and translate that choice into an identifier. A textual target can remove that extra demand. For classification, the corresponding pattern is to choose a broad category first and then a narrower one, rather than jumping directly to the most specific class.

When you know how to approach a problem, put that structure into the instructions. A generic request to think step by step leaves the procedure unspecified. A more useful template can begin by summarizing the problem and then name the particular decisions needed to solve it. The aim is to give the model a usable sequence, not merely ask it to reason harder.

Context selection matters for the same reason. Every irrelevant passage creates another filtering task for the model. Long few-shot examples can often be shortened to the portions that demonstrate the important behavior without reproducing their entire inputs and outputs. Peter recommends starting with approximately 10–20 examples to test each prompt revision, alongside end-to-end evaluations. This gives prompt changes a repeatable check from the beginning.

10:4510:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:45 · section reference included

Track state explicitly and make updates easy to express

Once prompting improvements are exhausted, split the task into smaller pieces. A cognitive architecture supplies structure that a single prompt would otherwise have to maintain implicitly:

  • State tracking: Record where a long task currently stands so the next call receives useful context.
  • Planning and replanning: Generate a plan, revise it as conditions change and decide what comes next.
  • Verification: Check work after an action or intermediate result.
  • Notes and scratchpads: Preserve information needed by later steps.

Extra calls introduce latency. Where work is independent, parallel execution can preserve the decomposition without making every call wait for the previous one.

The state-update interface should also fit the model’s task. Peter recommends key-value updates resembling Python dictionary assignments: first identify the key, then generate its replacement value. For example, a scheduling agent can record that it has prepared a proposed slot without marking the meeting as booked:

python

notes = {
    "task": "schedule investor meeting",
    "status": "checking availability",
    "proposed_slot": None,
    "booked": False,
}

notes["proposed_slot"] = "Tuesday 14:00"
notes["status"] = "awaiting approval"

Here the slot is a teaching value. Updating the notes records a proposal; it does not create a calendar event. The task and booked flag remain unchanged while two targeted fields change.

Record a proposed slot without booking the meeting

Constructed example: The dictionary keys, status strings and Tuesday 14:00 slot are teaching values illustrating the stated key-value update mechanism, not records from the Jace demonstration.

Scheduling task — unchanged
schedule investor meeting

Operation: Set proposed_slot to Tuesday 14:00, then set status to awaiting approval.

task

Before: Before the notes update
schedule investor meeting
After: After the notes update · Unchanged
schedule investor meeting

status

Before: Before the notes update
checking availability
After: After the notes update · Changed
awaiting approval

proposed_slot

Before: Before the notes update
None
After: After the notes update · Changed
Tuesday 14:00

booked

Before: Before the notes update
False
After: After the notes update · Unchanged
False
Targeted dictionary updates change the proposal fields while preserving the task and unbooked status.

The alternative is to regenerate the whole state or document. Peter reports that full rewrites often perform better because they give the model more generation time, at the cost of latency. He points to Cursor’s text rewriting as an example of preferring regeneration when models struggle to produce small diffs.

Update formModel’s immediate taskTradeoff
Key-value updateSelect a key, then a valueTargeted state changes
Full rewriteRegenerate the complete objectMore generation and latency
Small diffExpress only the precise editHarder for the models Peter describes

The choice is part of the tool design: a compact edit representation is useful only if the model can produce it reliably.

Peter also advises avoiding recursively nested structures when possible, because they add complexity and, in his account, move farther from familiar training patterns. Treat that as a heuristic against unnecessary nesting: current Claude prompting guidance permits XML nesting when it reflects a natural hierarchy.

Slide recommends parallelizing calls, key-value state updates, full rewrites over diffs, and caution with recursively nested structures.Open full source frame
Cognitive architecture considerations: latency, state-update interfaces, and nested structures.
14:0214:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:02 · section reference included

Use structure without multiplying failure points

Reasoning templates can encode the known structure of a problem. For visual inputs, Peter suggests an explicit intermediate representation: first have the model describe the image’s key points in text, then reason over that description. His rationale is that models have seen trillions of text tokens but substantially less paired image-text data, making text a more practiced medium for the reasoning step. He presents this as an approach to try, rather than a measured comparison for a named vision task.

Components should also compensate for one another’s errors. If one stage can produce a particular mistake, another should be able to detect or handle it. But every additional component introduces another opportunity for the system to become brittle. Use enough structure to reduce each model call’s burden, while keeping the number of components as small as the task allows. The benefit of decomposition and the cost of coordination have to be considered together.

16:0616:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:06 · section reference included

Fine-tune on varied, demanding interactions

Fine-tuning becomes relevant when the system still falls short of its performance target or when inference cost needs to come down. Peter’s proposed data-generation method keeps the application in the loop:

  1. Create templates describing different human roles.
  2. Instruct a language model to act as a person in one of those roles.
  3. Let the rest of the application respond normally.
  4. Collect the resulting interactions as synthetic training data.

The simulator varies the user side of the interaction while exercising the system you actually want to improve. Prompt diversity and difficulty are central to the usefulness of that data.

Peter attributes weaknesses in early Alpaca models partly to prompts that were too easy. His recommendation is to add conditions and complexity even when the eventual task looks simple, giving the model more relationships and constraints to learn from each example. That causal diagnosis is his interpretation; Stanford’s original account does not establish easy prompts as the cause of Alpaca’s weaknesses.

Fine-tuning can also remove intermediate stages from an existing pipeline. Pair the original inputs with the pipeline’s final outputs, then train a model to map directly between them. This distillation skips work that previously required several steps. Peter expects lower latency in exchange for some performance loss, making it a way to trade part of an elaborate architecture’s quality for a faster path through the task.

17:0817:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:08 · section reference included

Filter the data, then consider reinforcement learning

Before moving to a more expensive training method, improve the examples used for fine-tuning through rejection sampling or filtering. Peter claims that ordinary fine-tuning can reach GPT-4 performance on a specific task and that filtering can take it further. He supplies no named task, dataset, GPT-4 version or metric for that comparison.

When execution feedback is unavailable, a language-model judge can evaluate the output of an individual model, a larger subsystem or the complete application. Use those judgments to remove examples that likely failed. The judge does not have to be perfect to be useful in Peter’s account: the practical objective is to exclude substantial amounts of poor training data before fitting the model to it.

Reinforcement learning is the final escalation because it can optimize behavior across multiple steps, an especially relevant capability for agents. Execution feedback or language-model judgments of system components can supply the learning signal. Peter reserves this approach for cases where the earlier methods do not work: the setup cost is substantial, and the resulting training investment makes it harder to switch to a different model. The decision to use reinforcement learning therefore includes both the performance target and the cost of changing the system afterward.

18:3018:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:30 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…