AI Engineer Summit 2025
What does it take to build a personal, local, private AI Agent that augments you deeply?
Loading slides…
Read the talk
What does it take to build a personal, local, private AI Agent that augments you deeply?
A useful personal agent needs more than intelligence: it needs the right context, dependable actions, and an owner who controls what it knows and does.
From a talk by Soumith Chintala
From keeping up with AI to delegating everyday work
How do you keep up with AI when following its developments is part of your job? For Soumith Chintala, that problem comes with working on the infrastructure beneath many AI applications. He co-founded PyTorch and works at Meta, which he describes as the project's majority funder. Although he also works a little on Llama, this talk is about personal agents, not model-release announcements.
Chintala reports previously spending three to five hours a day checking AI sources for work. AI News, swyx's news aggregator, made that daily information gathering much easier. Its usefulness prompted a larger question: where else could AI become a deeper part of everyday life?
Aggregation is only the starting point. AI News gathers information; an agent can act on it. Chintala's other motivation comes from robotics: he wants home robots that handle errands. Bringing those two ambitions together creates a demanding design problem. A deeply useful personal agent would hold extensive context about your life and have authority to act on your behalf. That combination makes local ownership and privacy central requirements. What follows is a sketch of how to get there, not a finished system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Intelligence cannot recover context it never received
Chintala defines an agent by its ability to act in the world. Merely collecting and processing context does not meet that definition. But the ability to act is not sufficient either: a highly intelligent agent without the right context can still be useless.
Suppose your agent can read Gmail, WhatsApp, and your calendar. You ask whether your prescription was renewed, and it answers no. The renewal confirmation actually arrived as a CVS text in iMessage—a source the agent cannot access. Its answer looks like a lie, but the underlying failure is missing evidence. A similar failure occurs when an agent sees one bank account but misses money received through Venmo.
The practical damage extends beyond one wrong answer. If you cannot predict when the agent has enough information, you have to investigate its answers yourself. Delegation only saves effort when reliability is predictable enough to stop routinely rechecking the work. More reasoning over an incomplete view does not remove that dependency on context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A practical place to gather context
Context collection is a separate problem from choosing an inference provider. Whether the agent uses an OpenAI API, another service, or a local model, it still needs access to the parts of your life relevant to its tasks. A wearable could, in principle, see what you see and hear what you hear. Chintala considers battery life the obstacle to making that continuous capture practical.
A phone is the next obvious candidate because so much personal activity already happens there. But an agent continuously running in the background and watching the screen conflicts with the execution restrictions Chintala encounters in Apple's ecosystem. The limitation is restricted background execution, rather than a blanket inability to perform asynchronous work.
His practical alternative is an internet-connected Mac mini at home:
- Keep the computer plugged in and online.
- Log into the personal services the agent needs.
- Run the agent asynchronously on that machine.
This removes the wearable's battery constraint and provides a desktop environment for accessing services. Chintala also describes this approach as connecting with Android ecosystems, which he considers more open. It is a feasible host for an agent, not a claim that every personal data source becomes automatically accessible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trust changes when software acts for you
Why not subscribe to a cloud agent and let it run your life? After all, cloud email already holds sensitive information, including tax documents. Chintala's distinction is behavioral predictability. Email has a simple mental model: messages arrive, and you send replies. You can understand what the service does on your behalf.
Now let the email service automatically reply whenever it feels confident. The question immediately becomes what its worst possible action might be. An offensive reply to your boss is consequential in a way that receiving or displaying a message is not. As the action space grows more powerful and less predictable, direct user control matters more.
Control also includes whose incentives guide a decision. Chintala imagines a shopping agent whose provider changes its behavior to buy only from vendors that pay kickbacks. The agent could still complete a purchase while no longer serving the user's preferences faithfully. Local ownership is his proposed way to retain control over an assistant intimate enough to make such decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Avoiding dependence across an entire life
The second reason for local agents is decentralization. Walled gardens already restrict interoperability between services. Accepting an ecosystem's limits for a compartmentalized tool such as maps or email is one decision; placing an agent that takes many kinds of everyday actions inside that ecosystem is a much broader commitment. Chintala wants local, personalized agents to become the norm rather than another layer of dependence on a single provider.
The third reason concerns private deliberation. A deeply personal assistant may receive questions you would never say aloud. Chintala calls his concern “thought crimes”: the possibility of prosecution or persecution for thoughts shared with an intimate assistant.
He invokes enterprise cloud API contracts, mentions Perplexity, and contrasts enterprise arrangements with consumer services. His concern is that even stronger contracts can leave exposure through logging and safety checks, which he describes as legally mandated. That is his personal risk rationale; the talk does not establish a provider-specific legal obligation. For his most intimate augmentation, he does not want to take that risk, which brings him back to local and private operation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Running the model is only the first technical challenge
The technical assessment belongs to the talk's February 2025 AI Engineer Summit context. Chintala points to vLLM and SGLang as strong open-source projects for running models, both built on PyTorch.
That dependency prompts an infrastructure anecdote. After introducing a PyTorch bug, Chintala and colleagues who owned Teslas worried about the possibility of writing bugs that could affect themselves, because Tesla uses PyTorch. He immediately qualifies the story: things were fine, and the bug was not that bad. The anecdote illustrates how close foundational software can sit to the systems its developers depend on.
At recording time, Chintala describes local inference as slower and more limited than cloud services, even with an expensive machine. His qualitative comparison is between a roughly 20-billion-parameter or distilled model that can run fairly fast locally and full, unquantized DeepSeek-R1, which he describes as very slow. He supplies no hardware configuration, task, or latency measurements for that comparison.
He expects the infrastructure to improve, while accepting that local users may not always run the newest, most capable model. The harder remaining questions are research and product questions: can the model operate software dependably, understand your preferences, and recognize when an action is too consequential to take?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Understanding a screen—and a particular sofa
Chintala finds open multimodal models promising but insufficient for dependable computer use. Even leading paid closed models, in his experience, frequently break while operating software. Better access to applications does not help much if the agent cannot reliably navigate them.
Shopping reveals a different weakness. Recommendations for clothes, shoes, and furniture tend toward generic choices. Giving increasingly precise preferences does not necessarily produce increasingly precise matches. His example is a request for a red velvet sofa with oak wooden legs that returns a green velvet sofa without oak legs.
| Attribute | Requested | Returned in the example |
|---|---|---|
| Color | Red | Green |
| Upholstery | Velvet | Velvet |
| Legs | Oak wood | Not oak wood |
Matching one attribute is not enough to satisfy the request. Chintala's diagnosis is that the models rely heavily on text matching rather than actually identifying the requested visual properties. A personal shopping agent needs to preserve the conjunction of preferences, not merely retrieve something nearby in description.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognize dangerous actions before taking them
Another missing component is a dependable catastrophic action classifier. Not all mistakes deserve the same response:
- Recoverable navigation: The agent follows the wrong Wikipedia link. It can backtrack and open the intended page.
- Consequential purchase: The user asks to replenish Tide Pods, but the agent buys a Tesla. The mistake has escaped the realm of harmless exploration.
The distinction is about what happens when the action reaches the world, not just whether the model selected the wrong next step.
The required mechanism is detection before execution: identify a potentially catastrophic action and, where appropriate, notify the user instead of taking it. Chintala acknowledges existing research but considers it insufficient for the trust people need. Local operation does not solve this problem by itself; both local and cloud agents need better judgment at the action boundary.
Voice is another practical gap. Chintala describes open-source voice mode as underdeveloped, yet considers it necessary for a personal assistant. Sometimes the natural interaction is to speak, not to type every request. A useful local agent needs that everyday interface alongside better perception and action safety.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why open models could compound faster
Despite those gaps, Chintala is optimistic about open models. He contrasts OpenAI and Anthropic improving their respective models with improvements shared across an open ecosystem. In his account, successive releases from Llama, Mistral, Grok, and DeepSeek have repeatedly challenged skepticism about what open models can achieve.
The proposed mechanism is coordination. Early in an open-source project's life, there may not be enough contributors to coordinate effectively. Once a critical mass exists, participants can build on one another's work, and progress compounds. Linux is his analogy for what becomes possible after that threshold. Chintala forecasts that open models will outperform closed models per dollar invested; he presents this as an expectation grounded in open-source coordination, not an investment-normalized benchmark.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The work extends from reasoning data to systems engineering
A concrete example closes the research agenda: General Reasoning, associated with Ross Taylor. Taylor worked on Galactica, an open science language model released before ChatGPT. Chintala recalls the criticism surrounding its release and the subsequent retreat from its public rollout, contrasting that reception with the later acceptance of using language models for science. This should not be read as the disappearance of all model access: the Galactica repository still documents models and weights. Taylor's newer work, Chintala says, includes open reasoning data intended to narrow the reasoning gap between open and closed models.
The infrastructure work remains concrete too. Chintala says PyTorch is working to enable local agents and, at the time of the talk, is hiring engineers who combine AI expertise with systems engineering. He ends with an invitation to LlamaCon on April 29, 2025, promising more Llama developments. He tentatively places it in California while acknowledging that he has not checked the location. The closing invitations point toward two complementary contributions: improving the intelligence available to local agents and making that intelligence practical to run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Daily summaries of AI discussions across Discord, Reddit and X.
Open-source inference and serving engine for language models.
Serving framework for language and multimodal models.
Official repository for the DeepSeek-R1 reasoning model family.
Ross Taylor and collaborators' paper on language models trained for scientific knowledge and tasks.
Installation instructions, model sizes and inference examples for GALACTICA.
Further reading
Apple's guidance on scheduling background work and handling runtime limits.
Full transcript
Loading transcript…