AI Engineer World's Fair 2025
Grounded Reasoning Systems for Cloud Architecture
Loading slides…
Read the talk
Grounded Reasoning Systems for Cloud Architecture
Catio’s architecture copilot connects requirements to infrastructure graphs, coordinates specialist agents, and uses human review to turn proposed changes into inspectable design decisions.
From a talk by Iman Makaremi
Before you start: Familiarity with cloud components, vector retrieval, and LLM agents will help; no specific orchestration framework is required.
Reasoning from scattered requirements
How should an architecture change when its requirements, constraints, and current state are scattered across different sources? More users, developers, tools, and expectations create more decisions than a static automation workflow can anticipate. A useful system must understand the situation, consider competing options, justify a choice, and plan the work. This is the problem behind Catio’s cloud-architecture copilot, introduced by Iman Makaremi, its co-founding head of AI.
Architects negotiate trade-offs among requirement clarity, delivery time, and available resources. Much of the context behind those decisions is implicit. Capturing it for AI requires more than inventorying infrastructure: the system must understand how an architect uses that context to make a decision.
Catio divides the work into three connected problems:
- Requirement understanding: Identify where requirements come from, how they are represented, which parts matter, and whether they apply globally or to a particular part of the system.
- Architecture identification: Determine what each component does. Its type alone is insufficient; its position and role in the architecture affect its function.
- Architecture recommendation: Compare the requirements with the current architecture, then propose changes that close gaps or bring the system closer to best practices.
A recommendation therefore depends on both an interpretation of intent and an account of what the infrastructure actually supports.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connecting text, graphs, and questions
The first AI challenge is that the two main sources of evidence have different shapes. Requirements mostly live in text documents; architecture is inherently graph data. The system must connect statements about desired behavior to components and relationships in the deployed design.
The second challenge is decomposition. A broad or vague architecture question rarely maps to one retrieval call: it needs to be broken into smaller questions and a plan for answering them. The third is evaluation. When many moving parts contribute to a recommendation, feedback must help identify which part of the system needs to improve.
Grounding starts with getting the right context into the reasoning process. Translating a natural-language request into a meaningful architecture retrieval task is already difficult; doing it quickly adds another constraint. Catio consequently explored architecture retrieval and requirement retrieval as separate, related problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Enriching retrieval without losing relationships
Catio first improved the information available to its retrievers in several ways:
- Semantic component enrichment: Add relevant descriptive information to each architecture component so vector search can find it more easily.
- Graph-enhanced component search: Use graph algorithms alongside searches for a component or component type to retrieve the appropriate architectural context.
- Requirement scoring: Identify concepts and requirements important to the organization, then score documents against them before retrieval.
Makaremi reports that scoring requirement documents in advance made retrieval faster. He does not provide a measured speedup.
Semantic grounding helped, but it did not solve every retrieval problem. Catio still encountered limits when scaling searches or seeking detailed answers. Soft grounding also depended on prompt design: the agent needed instructions about what to focus on and what to retrieve when the system could not supply an exact structure for the task.
Graph memory supports continuity as well as accuracy. Finding individually relevant nodes is only part of the job. The system also needs to connect those nodes and explain why they appeared together, so subsequent reasoning retains the relationships that made the evidence meaningful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two retrieval pipelines
The early architecture retriever separated storage from querying. Its storage path prepared a searchable representation of each component:
- Break the architecture into individual components.
- Convert each component’s JSON data into natural language.
- Enrich that description with connection data.
- Embed the enriched description and store it in a vector database.
At query time, the system embedded the search request and used it to retrieve matching components. The connection enrichment mattered because the indexed object needed to convey more than its isolated properties.
This pipeline produced useful results, but semantic similarity did not adequately handle the graph-shaped data. Catio first augmented the vector approach with graph searches, then began moving toward a design based more heavily on a knowledge graph. That progression preserves a useful distinction: describing a component’s connections in text is not the same operation as searching the relationships themselves.
The requirements pipeline began with a familiar document-processing sequence: preprocess the input, split it, and embed the resulting pieces. Catio added requirement templates to define the structure of extracted information. The templates made extraction answer to the needs of downstream recommendation agents, rather than merely producing arbitrary snippets of relevant text.
Makaremi reports that requirement templates helped organize business requirements and retrieve useful pieces faster. But as document counts grew, larger searches sometimes lost context. Graph analysis was a possible way to recover relationships across the corpus, extending the same concern that had appeared in architecture retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structured communication and isolated memory
Retrieving evidence does not remove conflicting architectural goals. Good design still requires debate and trade-offs. Catio built a multi-agent system, or MAS, with role-specific agents that could collaborate and converge on justified recommendations. Structured messages made those interactions easier to coordinate across longer chains of work. Early versions generated and parsed XML; Makaremi describes that as a burdensome implementation stage the team had moved beyond.
Multiple levels of interaction required explicit memory boundaries. Catio isolated conversations to avoid spending tokens on unrelated exchanges. Makaremi reports that Catio’s experiments associated larger agent memory with more hallucination. No quantitative effect or evaluation protocol is supplied, but the observation influenced the decision to keep agent memories isolated.
Parallel processing introduced another memory decision: when cloning an agent, what should be copied, and from which point in its history? A clone needed the context relevant to its task without allowing all concurrent conversations to become one growing shared transcript.
Structured output provided clarity and programmatic control over those workflows. Makaremi acknowledges a potential trade-off between output constraints and a model’s reasoning abilities, while reporting good results at Catio’s chosen level of structure. The engineering question is how much structure the workflow needs to remain controllable while leaving room for the agent to solve the task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Dynamic reasoning inside explicit control flow
The orchestration does more than execute a fixed plan. Agents can resolve trade-offs dynamically, and Catio continues to experiment with what each agent does at each interaction step. Makaremi associates more dynamic orchestration with greater creativity in planning. But that flexibility still needs explicit control flow: letting agents converse without governing how they reach a result is not sufficient.
Makaremi describes the recommendation engine as a working production part of Catio’s stack, while identifying the displayed recommendations as using mock data. The interface groups recommendations into categories such as data architecture, messaging and queuing, and API integration.
Each recommendation is a structured design artifact:
| Part | Purpose |
|---|---|
| Description | Explain the recommended change |
| Target state | Describe the intended architecture |
| Gap analysis | Compare requirements with discovered support |
| Recommended actions | Propose how to close the gaps |
The selected recommendation in the screenshot is titled Introduce a Streaming Data Processing Layer. The gap analysis captures requirements whose support the agents could not find in the current architecture. That wording matters: a retrieval result establishes what the system found, so its evidence must remain inspectable before a proposed change is accepted.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From specialist recommendations to independent proposals
A chief architect agent oversees the high-level work. Beneath it are ten staff architect agents specializing in domains such as infrastructure, API, and IAM. Two retrieval agents provide the evidence: one accesses requirements, while the other answers questions about the current architecture and its components.
The recommendation workflow has three sequential stages:
- List generation: Produce candidate recommendation topics.
- Conflict resolution: Have the chief architect inspect the combined list, identify conflicts and redundancies, and prune it.
- Design proposal: Expand each retained topic into a full proposal, including gap analysis and architectural improvements.
Resolving conflicts before detailed drafting prevents the workflow from treating every specialist’s initial suggestion as an independently valid final plan.
During list generation, the chief asks a staff architect for recommendations. That specialist queries the architecture-state and requirements agents in parallel, repeating the exchanges within its available budget before returning a candidate list. Multiple staff architects do this concurrently; Makaremi describes tens of calls happening at the same time. After conflict resolution, the chief returns the surviving topics to the specialists for proposal writing.
Proposal writing uses a deliberate fork in memory. Each staff architect is cloned according to the number of recommendations it must develop. Every clone inherits the prior history, but its new conversation history is separate from the other clones. The common evidence survives; proposal-specific work does not leak into neighboring drafts.
For a concrete illustration, suppose one staff architect has gathered evidence about an API and must draft two proposals: add request throttling and add API monitoring. The JSON below represents the memory boundary after each clone receives its assignment. Both retain the same prior evidence; neither receives the other clone’s new assignment. These are proposal-writing tasks, not executed infrastructure changes.
json
{
"staffHistory": [
"Requirement: protect the API from request bursts.",
"Architecture: an API gateway routes to the service."
],
"clones": [
{
"id": "throttling-proposal",
"inheritedHistory": [
"Requirement: protect the API from request bursts.",
"Architecture: an API gateway routes to the service."
],
"currentHistory": [
"Draft a proposal to add request throttling."
]
},
{
"id": "monitoring-proposal",
"inheritedHistory": [
"Requirement: protect the API from request bursts.",
"Architecture: an API gateway routes to the service."
],
"currentHistory": [
"Draft a proposal to add API monitoring."
]
}
]
}
The implementation must preserve that distinction between inherited context and each clone’s subsequent history, regardless of how the histories are stored.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluating the path to a recommendation
A final recommendation alone does not explain whether a many-agent workflow behaved well. There may be repeated conversations behind it, each introducing an opportunity for an irrelevant retrieval, unsupported inference, or mistaken handoff. Catio closed the improvement loop with human scoring, structured feedback, and revision cycles. At the development stage Makaremi describes, LLM evaluations were useful but did not provide the guidance the team needed to make its intended improvements.
The team built an internal evaluation tool called Eagle Eye. For each case, reviewers can inspect the architecture, extracted requirements, inter-agent conversations, and generated recommendations. They score relevance, feasibility, and clarity, then use those judgments to decide which improvements to prioritize. This puts the recommendation beside the evidence and exchanges that produced it.
LLM confidence is not correctness. Human feedback was especially important while building the system from scratch, and evaluation became part of system design rather than a later addition. For every new AI system, Catio began asking how it would be evaluated as it developed. Depending on the task, that infrastructure could be a human review tool, a monitoring dashboard, or an LLM-based feedback loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When a retriever is asked to schedule a workshop
Inspecting intermediate conversations exposed a distinctive failure. In an early case, the network-security staff architect sent the requirements retriever a request to schedule a workshop, complete with possible dates, as part of trying to solve its architecture task. The agent had moved from seeking requirements to requesting an organizational action from a retrieval role.
The example shows why a plausible conversational next step can still violate the workflow’s purpose. A requirements retriever is there to supply evidence; a workshop request does not perform that function. Makaremi identifies prompts and structured outputs as places to address this kind of hallucination. Reviewing the exchanges makes the failure visible at the point where the agent leaves its intended role, rather than only through whatever recommendation eventually emerges.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Coordinating workflows and task-specific memory
The intended scope is larger than answering an isolated architecture question. Makaremi describes a copilot that can reason across thousands or millions of components and many documents, serving stakeholders from developers to CTOs. Those quantities describe the problem the system aims to address, rather than a demonstrated scale benchmark.
Supporting that scope takes roles, workflows, memory, and structure, with experimentation tied to the available data. Graphs are becoming more important in Catio’s designs. How agents interact, and how much autonomy each receives, remain consequential design choices: they determine which evidence can reach a decision and how the system keeps the reasoning process under control.
After experimenting with several frameworks, Catio is converging on LangGraph for some agent workflows, with a manager above them to coordinate multiple workflows. Flyte handles some of the higher-level management. Alongside that orchestration, the team is using graphs to capture as much memory as possible so each task receives the context it needs. The workflow graph and the graph holding architectural knowledge serve different purposes; naming the orchestration framework does not by itself specify the knowledge representation.
Makaremi closes with the view that this work is only beginning, and that systems built around these mechanisms will become a way for AI to design software. The ambition rests on the whole process: retrieving the relevant context, coordinating specialists, preserving useful memory, and making the resulting proposals open to inspection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current documentation for the named framework's thread checkpoints and cross-thread memory stores.
Workflow orchestration project used for higher-level management. The current README presents Flyte 2 and links to Flyte 1.
Further reading
A contemporaneous account of the chief architect, ten specialists, retrieval agents, cloning and EagleEye inspection system.
Makaremi explains Catio's use of conversation limits, hierarchical coordination and conflict resolution.
Updates since the talk
- Introducing GraphQAArticle
Makaremi's later account of combining language-model reasoning with graph algorithms for architectural questions.
Open-source natural-language graph analysis with NetworkX algorithms, installation instructions and examples.
Full transcript
Loading transcript…