AI Engineer World's Fair 2025
Building Agentic Applications with Heroku Managed Inference and Agents — Julián Duque and Anush DSouza
Loading slides…
Read the talk
Building Agentic Applications with Heroku Managed Inference and Agents
Build up from a hosted Jupyter notebook to streaming inference, executable tools, read-only database analysis and MCP servers that other clients can discover and call.
From a talk by Julián Duque and Anush DSouza
Before you start: Basic familiarity with Python, HTTP requests and environment variables will help; following the deployment steps also requires a Heroku account with access to the relevant services.
Start with a notebook that needs only a browser
How far can you get building an agentic application without installing anything locally? This workshop starts by deploying Jupyter on Heroku, loading a notebook into it and attaching managed inference. The same notebook then becomes a client for model responses, tool execution and MCP servers. Heroku Managed Inference and Agents supplies the AI services; the browser supplies the working environment.
Julián Duque introduces himself as Heroku’s principal developer advocate, joined by Heroku AI product manager Anush DSouza. Participants get the slides and links through the Workshop Heroku AI channel in the AI Engineer Slack workspace. The workshop site’s QR code leads to the content and a form accepting the email associated with a Heroku account. At the event, that form provides temporary access to deployment and services without entering credit-card information. The hands-on work will happen inside the deployed notebook.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the agent loop an application resource
Heroku’s precedent is the Ruby on Rails application that was easy to write but difficult to deploy, operate and scale. git push heroku main compressed much of that operational work into a familiar developer action. DSouza applies the same framing to AI: getting a model response is only the beginning. Operating the application means choosing models, scaling execution, controlling tool access and understanding whether the system behaves as intended.
The proposed unit of integration is a managed agent loop attached to an application. Curated models sit inside a control loop with access to code execution and data; Model Context Protocol provides extensions. Model selection, evaluation and tracing remain operational concerns, but the platform aims to supply useful defaults rather than require developers to assemble every component. DSouza presents heroku ai models create as the CLI entry point for attaching AI as a resource.
The platform combines three kinds of primitives:
| Primitive | Role in the application |
|---|---|
| Managed inference | Access a curated model from application code |
| MCP servers | Extend agents with remote or stdio tools |
| Postgres with pgvector | Store embeddings for vector retrieval |
Heroku dynos provide the compute for tool execution. A first-party tool can run there and stream its result back to the agent, while eligible tool execution scales to zero when idle. Developers can bring their own MCP tools into the same arrangement. At the time of the workshop, first-party web search is planned and memory is described as a possible future offering.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy Jupyter with notebook persistence
The workshop website supplies step-by-step instructions for continuing later. Accepting the event invitation adds participants to the AI Engineer World’s Fair team in the Heroku dashboard; access is described as lasting through the seventh, over the weekend, with an extension possible. The team already contains a Jupyter workshop application, a Brave Search MCP and ai-engineer-data, whose PostgreSQL database will support the later analysis example.
To create the notebook application, Duque uses the Heroku-Jupyter template:
- Open the repository and select its Deploy to Heroku button.
- Enter a unique application name. The initial name is already taken in the demonstration, so Duque adds a workshop suffix.
- Select AI Engineer World’s Fair as the application owner to use the event’s funding.
- Set the notebook password through the deployment form’s environment variable, then deploy. Duque uses
labfor the demonstration; a publicly reachable notebook needs a strong password.
The deployment creates a dyno for Jupyter and a Heroku Postgres database for notebook persistence. It fetches the template source and builds its dependencies. While deployment and the conference network take time, Duque uses an existing deployment to continue. The persisted object here is the notebook, not every piece of transient runtime state.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attach inference and load the execution context
The next resource is Managed Inference and Agents. Duque describes its appeal as keeping inference within Heroku’s infrastructure instead of directly calling an external model-provider API. That should not be read as a guarantee that inference runs inside the application’s own network: the later documentation describes Amazon Bedrock hosting and a boundary of secure AWS accounts.
In the dashboard, open the application’s management page, select Resources, and add Heroku Managed Inference and Agents. The catalog presented in the recording includes Claude 3.5, Claude 3.7 and Claude 4 for text, Cohere Embed for embeddings, and Stable Image Ultra for image generation. Duque selects Claude 4. This historical provisioning flow supplies an API URL, API key and model ID, usable through HTTP or an OpenAI-compatible SDK. Later standard-plan documentation instead lists the connection URL and key, with the model selected in requests.
Once Jupyter is ready, File → Open from URL imports the workshop notebook. In Heroku, Settings → Reveal config vars exposes the workshop’s three connection settings: INFERENCE_URL, INFERENCE_KEY and INFERENCE_MODEL_ID. The notebook proceeds through environment setup, basic and streaming inference, native agent tools, and finally MCP deployment and client access. Tool execution uses a one-off dyno: it starts, performs the operation and shuts down. Duque describes this as paying for the tool’s compute rather than keeping a tool application running continuously.
The Python setup also needs the application targeted by tools:
python
import os
inference_url = os.environ["INFERENCE_URL"]
inference_key = os.environ["INFERENCE_KEY"]
model_id = os.environ["INFERENCE_MODEL_ID"]
target_app_name = "ai-engineer-workshop"
Replace ai-engineer-workshop with the deployed application’s name. This selects where commands and compute run; naming an application does not itself grant access. The managed tools documentation requires the inference add-on to be attached to the target application before it can start dynos there. Database access will require its own attachment setup.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect a completion, then consume its deltas
Basic chat completions take a message or an array of messages and return a model response. This endpoint can also request custom tool calls, but the caller owns their execution: application code must interpret the function call, execute it and provide the result. Frameworks such as ADK or LangChain can handle that loop. The first notebook example avoids tools and makes a direct HTTP inference request.
The payload contains the provisioned Claude 4 model ID and a user message asking for a one-sentence explanation of managed inference. Parameters such as temperature, maximum tokens and top_p tune generation, subject to the endpoint’s documented support. The response is a JSON chat-completion object: choices contains an assistant message explaining that managed inference handles deployment, scaling and maintenance of models for predictions.
Streaming changes how the response arrives. Instead of waiting for one complete JSON answer, the client receives server-sent events containing completion chunks. Duque first prints the raw events so their structure is visible. Each chunk carries a message delta, and an initial delta may have no text. The next example extracts those content fragments, prints them incrementally and renders the accumulated response as Markdown when generation finishes.
For decoded completion chunks, the notebook’s accumulation step can be written as a small Python function:
python
from IPython.display import Markdown, display
def render_completion(chunks):
parts = []
for chunk in chunks:
for choice in chunk.get("choices", []):
content = choice.get("delta", {}).get("content")
if content:
print(content, end="", flush=True)
parts.append(content)
print()
answer = "".join(parts)
display(Markdown(answer))
return answer
Incremental printing supplies feedback while generation is underway. The final Markdown pass formats the completed answer, including any code blocks the model returns.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move tool execution into the managed loop
The agents endpoint adds managed execution around inference. The available tools cover several distinct operations:
| Tool | Operation |
|---|---|
| Dyno Run Command | Run a Unix command or an existing application script |
| Postgres Get Schema | Retrieve the database structure |
| Postgres Run Query | Execute SQL and return data |
| HTML to Markdown / PDF to Markdown | Extract text from a supplied URL |
| Code Execution | Generate and run Python, Node, Ruby or Go code |
Dyno Run Command is useful for trusted code already deployed with the application. The database tools form a natural sequence: discover the real schema before generating a query. Document conversion provides text the model can work with, while code execution returns a computed result to inference.
Generated-code tools also accept a packages array, allowing dependencies such as Pandas or NumPy to be installed before execution. Duque describes the native tools as MCPs maintained by Heroku; custom MCPs extend the same endpoint. In the demonstrated API, agent responses are streaming only because tool work may take time. Application and database permissions depend on the tool being invoked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask the server for its current time
The first agent question asks for the server’s current date and time. A model’s training data cannot establish the live clock of this particular server, so the request enables Dyno Run Command. The payload includes the model ID and message, identifies the tool as a Heroku tool, selects the target application, and supplies date as the command with a description of its output.
The notebook sends the payload to the agents endpoint and extracts tool-call information from the stream. Execution targets ai-engineer-workshop; the notebook waits for the command, receives the date and time, and then displays the inference response based on that result. Replacing date with a deployed script would use the same mechanism for data extraction or access to an external service. The tool supplies current evidence before the model composes the answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the computation, change the runtime
Code execution lets the model produce the program as well as request its execution. Duque enables CodeExec Node and asks for the 13th Fibonacci number. The notebook displays the generated JavaScript with syntax highlighting, then receives the result from execution on Heroku. The Node demonstration returns 233 for the 13th Fibonacci number.
Using the convention F₀ = 0 and F₁ = 1, the computation fits in a short JavaScript program:
javascript
function fibonacci(n) {
let a = 0;
let b = 1;
for (let i = 0; i < n; i++) {
[a, b] = [b, a + b];
}
return a;
}
console.log(fibonacci(13));
Duque then changes the selected tool from Node to Go. The model generates Go code, Heroku compiles and runs it on a dyno, and the response gives the same result with an explanation. Python and Ruby are further runtime choices; participants are encouraged to change the requested operation as well as the language.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve an implementation before running it
The next example gives the agent two tools: HTML to Markdown and Code Execution Python. The request is specific about both the source and the task: use the Python snippet from Wikipedia’s Euclidean algorithm page to calculate the greatest common divisor of 252 and 105. That instruction makes retrieval part of the computation instead of leaving the model to produce an implementation from memory.
The managed loop now crosses a dependency between tools:
- Fetch the Euclidean algorithm page as Markdown.
- Read the page and extract its Python implementation.
- Prepare and execute Python using the requested inputs.
- Return an explanation grounded in the retrieved implementation and execution result.
The notebook shows page retrieval, then code generation and execution. Duque points out the additional wait involved in multiple tool runs, invites participants to inspect and modify the code, and finally receives an explanation describing the implementation and calculation. The important extension is that the output of retrieval becomes input to execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attach a read-only database before querying it
The notebook application initially lacks access to the shared solar-energy database. Before enabling database tools, open the team’s ai-engineer-data application and its Resources page. Heroku’s maintained Postgres tools operate only on read-only followers. Duque explicitly leaves the primary database untouched; write access through a separately deployed MCP would be a different capability with risks borne by its operator.
On the follower, select Manage Attachments and add the Jupyter application. Then inspect the consuming application’s Resources page to find the database’s attachment name. One application uses DATABASE; Duque’s ai-engineer-workshop uses HEROKU_POSTGRESQL_AQUA. These application-specific names identify the connection configuration that the tool should use. The selected dashboard frame shows this attachment mapping.
An attachment makes the database available to the application; a name selects that available database. Tools cannot reach another application’s database merely by naming it. If the appropriate database is already attached locally, no additional attachment is needed. This keeps resource access separate from the natural-language question the agent will receive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover the schema, then investigate energy savings
Postgres Get Schema and Postgres Run Query receive the target application and database identifiers. Duque describes the example dataset as roughly two months of hourly energy consumption and production for a solar-energy company. The question is how much energy has been saved in the last 30 days. To answer it, the agent must discover the relevant tables and fields, generate SQL, execute the query and interpret the returned data.
The first tool retrieves the schema, revealing tables including metrics, products, systems and users. The agent then generates and runs its query. Duque expresses complete confidence that schema knowledge will make the query run, but schema discovery grounds table and column references; it does not guarantee correct SQL or a correct interpretation of energy savings. The demonstration proceeds from returned data to further investigation.
The agent asks for additional queries to break the result down across three systems with different performance. Its final report includes savings, per-system results and an energy deficit. This is the useful behavior of the loop: one user question can trigger schema discovery and several queries before the final answer. Duque then proposes adding Python execution to generate a Matplotlib chart from the data; that chart is an extension, not a completed part of this demonstration. More tool calls also increase the total time required. A further exercise combines PDF extraction with code execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add Brave Search through MCP
Custom tools enter through the model configuration’s MCP server list and Toolkit Integration. Manage MCP Servers attaches a deployed MCP application, much as a database attachment connects another resource. The Brave Search server exposes web search and local search. In the agent request, an MCP tool type replaces the Heroku-native tool type, while the MCP Brave namespace identifies the server and Brave Web Search identifies the operation. Namespacing allows several MCP servers to coexist.
The Brave API key is already configured on the MCP application. Duque plans to remove the shared workshop key and warns that heavy search usage can incur charges. With authentication in place, the prompt asks for the most recent news about AI agents. Heroku starts a dyno, runs the MCP over standard input/output, returns the search results to inference and scales the execution back to zero. The notebook receives the tool’s search output before the model finishes rendering its answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give an existing MCP a Heroku entry point
For deployment, Duque uses a fork of the official Perplexity Ask MCP. The main adaptation is a Procfile, Heroku’s declaration of how application processes start. He adds an MCP entry point and explains that process types beginning with mcp are recognized as MCP processes. The server continues to communicate over standard input/output. A deployment button packages this into a Node.js application that can be attached to the consuming app.
Duque does not have a valid Perplexity API key and enters a placeholder for the deployment exercise. That lets him demonstrate building and making the application available, but it does not establish authenticated Perplexity execution. Participants are left with an exercise: attach the deployed server and write the code to invoke it with working credentials.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Call the same tools from an MCP client
After the earlier Brave news response completes, the workshop turns to consumers outside the managed agent endpoint. Cursor, Claude Desktop and agents built on other platforms can use the hosted tools through Toolkit Integration. In the recording, this gateway aggregates attached MCPs behind a server-sent events endpoint authenticated with a bearer token; OAuth support is described as work in progress.
Duque adds the newly deployed Perplexity server to the toolkit and uses the same token as inference. The notebook installs the Python MCP package, creates a client for the toolkit URL and sends the key in an Authorization header. The client then lists the tools and calls Brave Web Search. Although the notebook itself is hosted on Heroku, this demonstrates access through an ordinary MCP client rather than through Heroku’s managed agent loop.
The client connects and discovers Brave Local Search, Brave Web Search and Perplexity Ask. It executes Brave Web Search and prints the returned search content without further processing. Perplexity’s appearance in discovery shows that the server is exposed through the toolkit, while the successful invocation shown here is Brave’s. Duque also describes using the same gateway approach from Cursor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the OpenAI SDK, then continue experimenting
The final technical example returns to inference through the OpenAI SDK compatibility interface. Duque estimates chat-completion compatibility at 95%, with 99% as a goal; he supplies no measurement method. Some parameters are unsupported, so field-level compatibility matters more than that estimate when porting an application. The demonstrated integration configures the OpenAI SDK with Heroku’s API key and URL.
In Python, the client configuration and basic request have this shape:
python
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["INFERENCE_KEY"],
base_url=os.environ["INFERENCE_URL"].rstrip("/") + "/v1",
)
response = client.chat.completions.create(
model=os.environ["INFERENCE_MODEL_ID"],
messages=[{
"role": "user",
"content": "Explain the concept of Managed Inference in one sentence.",
}],
)
print(response.choices[0].message.content)
The SDK call in the recording returns an inference response. By this point, the notebook has exercised both interfaces: familiar chat completions for model output, and a managed agent endpoint that runs tools and feeds their results back into inference. The toolkit additionally exposes those tools to independent MCP clients.
The workshop closes with continued experimentation rather than another deployment step. Duque extends the event team’s access through the weekend and explains that Heroku has no free tier at the time of the talk, with a return under consideration. He points participants to Dev Center documentation, the Heroku AI website, the Heroku community on Twitter/X and the slides shared in Slack, with questions welcome through social follow-up.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Deploy Jupyter Notebook or JupyterLab on Heroku with password protection and PostgreSQL-backed notebook storage.
Updates since the talk
Current provisioning, connection settings, model hosting, security guidance and usage documentation.
Reference for command execution, read-only database tools, generated-code execution and HTML/PDF conversion.
Configure the OpenAI SDK for Heroku and check supported, ignored and unsupported API fields.
Full transcript
Loading transcript…