AI Engineer World's Fair 2024

The Adversarial Path to the Personal Assistant

Sumit Agarwal· CEO and cofounder, Ario18:46

Loading slides…

Read the talk

The Adversarial Path to the Personal Assistant

A useful personal assistant needs more than a capable model: it needs access to your data, editable context, and a way to turn scattered records into fewer chores.

From a talk by Sumit Agarwal

Before you start: Familiarity with LLMs and basic data pipelines is helpful; no knowledge of Ario is required.

What would it take to give someone an hour back?

What would an assistant have to do to give you a real hour back—not another hour of generated content to review, but an hour of tasks you no longer have to perform? Ario’s stated goal is to return one hour a day to each person who wants its help. That is an ambition, not a measured result. The target is the ordinary work that still needs doing even when nobody enjoys it.

The opening example makes that priority concrete: let AI handle the laundry so people have time for poetry and other creative work. Sumit Agarwal paraphrases an author named Joanna, whose sentiment Ario obtained permission to feature on its homepage. The value comes from removing chores, leaving people free to choose what to do with their time.

A quote card over a family photograph on Ario’s homepage contrasts AI making art with AI doing laundry.Open full source frame
Ario’s homepage features a quote about AI doing laundry so people can write poetry and make art.

That leads to a personal assistant for ordinary people, extending beyond executive assistance for people with substantial budgets. Agarwal introduces this direction alongside Ario’s announcement that it had raised $16 million. The funding supports the ambition; the rest of the demonstration explores what the assistant actually needs to know and do.

0:340:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:34 · section reference included

Start beneath the model: obtain the user’s data

The foundation is adversarial ETL: extracting, transforming, and loading personal data from systems that do not necessarily want to make extraction easy. Search histories, Facebook posts, DoorDash orders, Strava runs, and Instagram activity all contain context an assistant could use. Agarwal’s informal challenge to the audience—how many have ever exported hundreds of megabytes of their own data?—illustrates the access problem, rather than establishing an adoption statistic.

Ario aggregates those records on the user’s behalf, then builds an agent architecture above them. The diagram places the LLM, retrieval-augmented generation (RAG), and data management above adversarial ETL. These components become useful only when packaged into tasks people recognize and want help completing. Data access is the foundation; the assistant is the application built on top.

Diagram titled “Ario: Your Personal, Proactive, and Private Helper,” with an AI Assistant box containing LLM, RAG, and Data Management above Adversarial ETL.Open full source frame
Ario’s assistant stack places adversarial ETL beneath LLM, RAG, and data management.

The product is presented as something the audience can try, with an iOS app available at the time of the talk. Agarwal also qualifies the reliability: the capabilities mostly work, and a user may need to try more than once. That matters when interpreting the polished examples that follow.

2:002:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:00 · section reference included

Ownership does not remove connection friction

The first practical step is connecting sources such as Google Calendar, Amazon, and DoorDash. Agarwal describes Ario’s ownership commitments before showing the resulting profiles: the company takes no rights in the data, and users retain CRUD controls—create, read, update, and delete. He also says Ario does not move the data and operates on it at the user’s instruction. These are the presenter’s commitments; this part of the talk does not supply a storage architecture that explains the broad data-location statement.

Connecting the sources is still painful. Authentication and authorization introduce friction before the assistant can deliver anything useful. A system that promises to save effort therefore begins by asking the user to do some work, making the first result after connection especially important.

3:223:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:22 · section reference included

Turn records into a profile the user can correct

Ario’s first result is a data portrait. In cofounder Mengmeng Chen’s example, the generated image reflects interests and family context: an SLR camera for photography, drone photography, toys and small bikes associated with a young child, and hiking with her husband. The portrait gives a visible form to what the system has inferred from connected records.

Selecting Read More exposes the textual understanding behind the portrait. Daycare appointments suggest an active search for childcare; restaurant visits and DoorDash orders reveal food preferences, including Japanese restaurants she enjoys with her husband. These details become a profile that can inform later responses. Users can edit the profile and correct mistakes, but they do not have to write it from scratch. Instead of an intake form that becomes stale after completion, the intended behavior is an evolving profile updated from new data.

The family details show both the usefulness and the limits of inference. Her son Hunter likes Lego and trains. Purchases of sand toys and rash guards suggest beach plans. But the system also guesses that he attends kindergarten, although he is younger than that. A dietary constraint concerning her mother’s glucose intake shows how sensitive the resulting context can become. Searches, purchases, and explicit conversations can support useful assistance, while still producing conclusions that need correction.

Each source contributes a different view. Amazon and DoorDash supply purchase and ordering context; Twitter contributes reading interests and social connections, including whom the user follows and who follows them. The profile is an abstraction over those records, rather than a requirement that every future question inspect every transaction or post.

Conversational Memory adds a separate channel. An explicit request to remember something, or a statement of interest during a conversation, can populate a Memory section that the user can inspect, edit, and update. Ario tunes which facts deserve to persist. Agarwal considers its memory more robust than ChatGPT’s, but presents that as his assessment rather than a benchmark result.

4:314:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:31 · section reference included

The same travel question, with personal context

The demonstrations are prerecorded videos of the real product, not live executions. The first takes a promoted Copilot prompt: “Where’s a sunny, warm place I can travel to right now?” Copilot suggests Belize, Mexico, and Key West, Florida. Those may be attractive destinations, but the response shown does not connect them to this particular traveler. Agarwal calls this style of answer Google Plus Plus: a natural-language interface to a generic search for warm places.

Ario runs the same query with personal context and explains the connections behind its suggestions.

DestinationPersonal connection in the demonstration
MauiA preference for Asian food and relevant activities
San DiegoThe child’s Lego purchases and a visit to Legoland
MiamiRecent activities resembling things the user does in the Bay Area

The difference is not simply a new destination list. The response supplies reasons tied to the user’s records, making it easier to see why a recommendation appeared.

Slide titled “Demo 1: Copilot vs Ario” with an Ario phone response mentioning LEGOLAND, an interest in LEGO, and Miami food, activities, and weather.Open full source frame
Ario’s travel response connects San Diego activities to an interest in LEGO and lists Miami recommendations.

The next step would require more than matching an interest to an attraction. Agarwal proposes finding trips that fit the user’s calendar, reconcile the needs of Grandma, husband, and user, and respect a known budget. Those are future directions, not demonstrated capabilities here. He acknowledges that recommending San Diego because a child likes Lego is close to string matching; understanding a household means coordinating multiple constraints at once.

7:177:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:17 · section reference included

A morning routine without another questionnaire

The next comparison starts with a request for a personalized morning routine. In the ChatGPT example, the answer is a series of questions. Agarwal calls this the 20-questions failure mode: the user asks for help and receives an intake form to complete. Gathering missing information can be necessary, but it also transfers work back to the person asking for assistance.

Ario receives the prompt, “Help me create a personalized morning routine that will boost my productivity.” It responds with a schedule grounded in context it already has:

  • Wake time: A 7:00 AM start, described as matching this user’s existing wake time.
  • Exercise: Hydration and light stretching informed by exercise history, followed by a 20-minute workout from 7:10 to 7:30.
  • Food: After showering and getting ready, recommendations based on food and ingredients the user likes, inferred from DoorDash orders.

This illustrates how existing context can reduce repeated questioning and make an answer feel more relevant. It does not measure whether the proposed routine improves productivity.

9:239:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:23 · section reference included

Import a school schedule without losing its uncertainty

For a busy household, a dense school schedule is another source of work. Ario’s demonstrated workflow begins with a photo: upload the schedule, extract the events, and publish them to the calendar. Extraction handles the transcription, but the assistant adds value by comparing those events with the user’s existing commitments. Agarwal says Ario identifies calendar conflicts and notifies the user 24 hours, 72 hours, and seven days in advance. The imported information becomes something the system watches, rather than another document the parent must remember to check.

One detail makes this more than a clean document-to-calendar example: the school has not settled its first day and will confirm it later by email. Ario holds both possible dates on the calendar. An unresolved date should remain unresolved in the assistant’s representation. Keeping both holds protects the user’s availability until there is enough information to release one.

Agarwal then describes an update planned for four to six weeks after the talk. It would monitor email for this category of scheduling information, rather than attempting broad email monitoring. Once the school confirmed its start date, the planned workflow would remove the incorrect calendar entry and notify the user. The existing demonstration preserves both possibilities; the future workflow would resolve them from a later message. That distinction keeps a proposed automatic action separate from what the product is shown doing.

11:1311:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:13 · section reference included

Compute the obvious before asking the model

The demonstrations lead into a practical rule for building the pipeline: do as much enrichment and processing as possible with inexpensive, reliable conventional tools before involving an LLM. The Strava example is small enough to expose why. Ario’s extracted dataset contained a start time and an end time, but no total-runtime column. A same-day interval from 1:05 PM to 2:00 PM is 55 minutes of elapsed time.

Asked how long the user ran, the model initially failed to derive an answer. The team then added a general instruction to manipulate columns and look for useful relationships, rather than prescribing the exact subtraction. With that hint, it produced the duration. But the hint consumed prompt space to make the model discover a deterministic operation that preprocessing could have supplied directly.

The enrichment operation preserves the original timestamps and adds the derived value:

telapsed=tend−tstart=(14×60)−(13×60+5)=55 minutes\begin{aligned} t_{\text{elapsed}} &= t_{\text{end}} - t_{\text{start}} \\ &= (14 \times 60) - (13 \times 60 + 5) \\ &= 55\ \text{minutes} \end{aligned}

This assumes both times refer to the same day and clock. The missing duration field describes Ario’s particular dataset, not a general limitation of Strava: the current Strava API reference includes elapsed_time and moving_time, both in seconds. Strava’s duration definitions also distinguish elapsed time, which includes stops, from active moving time. Subtracting the timestamps supplies the former; it does not establish how long the runner was moving. The engineering lesson remains to compute a known feature before asking the model to reason with it.

Add elapsed duration without changing the source times

Constructed example: Field names and the enriched-record layout are teaching choices, not an observed Ario schema. The source provides the times and duration; the comparison illustrates recommended preprocessing rather than an executed transformation. Same-day, same-clock timestamps are assumed.

Activity record — unchanged
Start: 1:05 PM; end: 2:00 PM; same day and clock.

Operation: Compute end time minus start time and add elapsed_minutes before supplying the record to the LLM.

start_time

Before: Source fields
1:05 PM
After: Illustrative enriched record · Unchanged
1:05 PM

end_time

Before: Source fields
2:00 PM
After: Illustrative enriched record · Unchanged
2:00 PM

elapsed_minutes

Before: Source fields
Not present
After: Illustrative enriched record · Added
55
Supply the deterministic result as data instead of spending prompt space teaching the model to derive it.

Agarwal turns this into an operating principle: use the LLM last. His suggestion to try five, seven, or nine other ways is emphasis, not a fixed evaluation procedure. The point is to reserve model capacity and prompt space for work that benefits from them, instead of spending both on trivial arithmetic.

13:1113:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:11 · section reference included

Retrieve refined context before raw records

At larger scale, the next problem is search. Simply embedding raw records and making them available to the assistant does not, in Agarwal’s experience, produce good enough results. Ario instead transforms sources such as DoorDash and Amazon into refined profiles, then gives those profiles to the LLM so it can determine how to answer a question. Only when the task requires more detail does it descend into the raw records.

This creates a useful division of labor:

LayerRole
Raw recordsPreserve detailed source information
Refined profilesSupply an abstracted understanding of the user
LLM contextPresent information useful for the current question

A profile can answer many questions without a fresh search through every underlying record. But even refined data needs selection: giving every pipeline stage everything the system knows can overwhelm it. Ario spends effort withholding unnecessary context as well as retrieving relevant context. These are high-level design lessons; the talk does not specify a ranking algorithm, chunking strategy, or retrieval implementation, and points to later technical posts and videos for more detail.

15:0915:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:09 · section reference included

Use the extraction layer without adopting the assistant

The closing walkthrough exposes the foundation as a tool other developers can use. Ario Boost is a browser extension that does not require adopting the full assistant. Its two entry paths serve different uses:

Entry pathDescribed behavior
Login with ArioMake accessed data available in the mobile app
Skip Login for NowEnter offline/developer mode without an Ario account

The second option is explicitly for someone who does not want to entrust an account to Ario before using its extraction tools.

Annotated Ario Boost page with “Login with Ario” and “Skip Login for Now” buttons, labeled mobile-app connection and offline/developer mode, above a dashboard and phone illustration.Open full source frame
Ario Boost presents login connected to the mobile app and a separate offline/developer option.

Agarwal describes support for roughly 15–20 data sources at the time of the talk. In developer mode, skipping Ario login does not mean skipping authentication to the source services. The extraction system still logs into those services on the user’s behalf, combining screen scraping, APIs, and automated GDPR data downloads where each is available. Adversarial ETL is therefore a collection of access techniques adapted to different sources, rather than one universal API.

The final example is Agarwal’s Sutter health data. He describes logging into the source, retrieving after-visit summaries, downloading them to the device, and making the local records searchable. In this developer mode, he says, the extracted data stays on the user’s device. From there, the user can choose to send it to another LLM, use Llama, or build an application of their own. The assistant is one destination for the data; making the data available to its owner also enables others.

16:2216:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:22 · section reference included

Resources

From the talk

  • Chrome extension listing for connecting Amazon, DoorDash and Twitter context to the Ario mobile app.

Transcript download
Full transcript

Loading transcript…