AI Engineer World's Fair 2025

Building Agentic Applications with Heroku Managed Inference and Agents — Julián Duque and Anush DSouza

Loading slides…

Read the talk

Building Agentic Applications with Heroku Managed Inference and Agents

Build up from a hosted Jupyter notebook to streaming inference, executable tools, read-only database analysis and MCP servers that other clients can discover and call.

From a talk by Julián Duque and Anush DSouza

Before you start: Basic familiarity with Python, HTTP requests and environment variables will help; following the deployment steps also requires a Heroku account with access to the relevant services.

Start with a notebook that needs only a browser

How far can you get building an agentic application without installing anything locally? This workshop starts by deploying Jupyter on Heroku, loading a notebook into it and attaching managed inference. The same notebook then becomes a client for model responses, tool execution and MCP servers. Heroku Managed Inference and Agents supplies the AI services; the browser supplies the working environment.

Julián Duque introduces himself as Heroku’s principal developer advocate, joined by Heroku AI product manager Anush DSouza. Participants get the slides and links through the Workshop Heroku AI channel in the AI Engineer Slack workspace. The workshop site’s QR code leads to the content and a form accepting the email associated with a Heroku account. At the event, that form provides temporary access to deployment and services without entering credit-card information. The hands-on work will happen inside the deployed notebook.

0:180:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:18 · section reference included

Make the agent loop an application resource

Heroku’s precedent is the Ruby on Rails application that was easy to write but difficult to deploy, operate and scale. git push heroku main compressed much of that operational work into a familiar developer action. DSouza applies the same framing to AI: getting a model response is only the beginning. Operating the application means choosing models, scaling execution, controlling tool access and understanding whether the system behaves as intended.

The proposed unit of integration is a managed agent loop attached to an application. Curated models sit inside a control loop with access to code execution and data; Model Context Protocol provides extensions. Model selection, evaluation and tracing remain operational concerns, but the platform aims to supply useful defaults rather than require developers to assemble every component. DSouza presents heroku ai models create as the CLI entry point for attaching AI as a resource.

Two columns list model selection, app integration, secure data access and ongoing operation alongside an integrated platform, Heroku DevEx and OpEx, curated AI models and MCP extensibility.Open full source frame
AI application challenges alongside Heroku’s proposed platform capabilities.

The platform combines three kinds of primitives:

PrimitiveRole in the application
Managed inferenceAccess a curated model from application code
MCP serversExtend agents with remote or stdio tools
Postgres with pgvectorStore embeddings for vector retrieval

Heroku dynos provide the compute for tool execution. A first-party tool can run there and stream its result back to the agent, while eligible tool execution scales to zero when idle. Developers can bring their own MCP tools into the same arrangement. At the time of the workshop, first-party web search is planned and memory is described as a possible future offering.

3:544:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:54 · section reference included

Deploy Jupyter with notebook persistence

The workshop website supplies step-by-step instructions for continuing later. Accepting the event invitation adds participants to the AI Engineer World’s Fair team in the Heroku dashboard; access is described as lasting through the seventh, over the weekend, with an extension possible. The team already contains a Jupyter workshop application, a Brave Search MCP and ai-engineer-data, whose PostgreSQL database will support the later analysis example.

To create the notebook application, Duque uses the Heroku-Jupyter template:

  1. Open the repository and select its Deploy to Heroku button.
  2. Enter a unique application name. The initial name is already taken in the demonstration, so Duque adds a workshop suffix.
  3. Select AI Engineer World’s Fair as the application owner to use the event’s funding.
  4. Set the notebook password through the deployment form’s environment variable, then deploy. Duque uses lab for the demonstration; a publicly reachable notebook needs a strong password.

The deployment creates a dyno for Jupyter and a Heroku Postgres database for notebook persistence. It fetches the template source and builds its dependencies. While deployment and the conference network take time, Duque uses an existing deployment to continue. The persisted object here is the notebook, not every piece of transient runtime state.

8:028:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:02 · section reference included

Attach inference and load the execution context

The next resource is Managed Inference and Agents. Duque describes its appeal as keeping inference within Heroku’s infrastructure instead of directly calling an external model-provider API. That should not be read as a guarantee that inference runs inside the application’s own network: the later documentation describes Amazon Bedrock hosting and a boundary of secure AWS accounts.

In the dashboard, open the application’s management page, select Resources, and add Heroku Managed Inference and Agents. The catalog presented in the recording includes Claude 3.5, Claude 3.7 and Claude 4 for text, Cohere Embed for embeddings, and Stable Image Ultra for image generation. Duque selects Claude 4. This historical provisioning flow supplies an API URL, API key and model ID, usable through HTTP or an OpenAI-compatible SDK. Later standard-plan documentation instead lists the connection URL and key, with the model selected in requests.

Once Jupyter is ready, File → Open from URL imports the workshop notebook. In Heroku, Settings → Reveal config vars exposes the workshop’s three connection settings: INFERENCE_URL, INFERENCE_KEY and INFERENCE_MODEL_ID. The notebook proceeds through environment setup, basic and streaming inference, native agent tools, and finally MCP deployment and client access. Tool execution uses a one-off dyno: it starts, performs the operation and shuts down. Duque describes this as paying for the tool’s compute rather than keeping a tool application running continuously.

The Python setup also needs the application targeted by tools:

python

import os

inference_url = os.environ["INFERENCE_URL"]
inference_key = os.environ["INFERENCE_KEY"]
model_id = os.environ["INFERENCE_MODEL_ID"]
target_app_name = "ai-engineer-workshop"

Replace ai-engineer-workshop with the deployed application’s name. This selects where commands and compute run; naming an application does not itself grant access. The managed tools documentation requires the inference add-on to be attached to the target application before it can start dynos there. Database access will require its own attachment setup.

13:0313:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:03 · section reference included

Inspect a completion, then consume its deltas

Basic chat completions take a message or an array of messages and return a model response. This endpoint can also request custom tool calls, but the caller owns their execution: application code must interpret the function call, execute it and provide the result. Frameworks such as ADK or LangChain can handle that loop. The first notebook example avoids tools and makes a direct HTTP inference request.

The payload contains the provisioned Claude 4 model ID and a user message asking for a one-sentence explanation of managed inference. Parameters such as temperature, maximum tokens and top_p tune generation, subject to the endpoint’s documented support. The response is a JSON chat-completion object: choices contains an assistant message explaining that managed inference handles deployment, scaling and maintenance of models for predictions.

Streaming changes how the response arrives. Instead of waiting for one complete JSON answer, the client receives server-sent events containing completion chunks. Duque first prints the raw events so their structure is visible. Each chunk carries a message delta, and an initial delta may have no text. The next example extracts those content fragments, prints them incrementally and renders the accumulated response as Markdown when generation finishes.

For decoded completion chunks, the notebook’s accumulation step can be written as a small Python function:

python

from IPython.display import Markdown, display


def render_completion(chunks):
    parts = []
    for chunk in chunks:
        for choice in chunk.get("choices", []):
            content = choice.get("delta", {}).get("content")
            if content:
                print(content, end="", flush=True)
                parts.append(content)

    print()
    answer = "".join(parts)
    display(Markdown(answer))
    return answer

Incremental printing supplies feedback while generation is underway. The final Markdown pass formats the completed answer, including any code blocks the model returns.

20:1820:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:18 · section reference included

Move tool execution into the managed loop

The agents endpoint adds managed execution around inference. The available tools cover several distinct operations:

ToolOperation
Dyno Run CommandRun a Unix command or an existing application script
Postgres Get SchemaRetrieve the database structure
Postgres Run QueryExecute SQL and return data
HTML to Markdown / PDF to MarkdownExtract text from a supplied URL
Code ExecutionGenerate and run Python, Node, Ruby or Go code

Dyno Run Command is useful for trusted code already deployed with the application. The database tools form a natural sequence: discover the real schema before generating a query. Document conversion provides text the model can work with, while code execution returns a computed result to inference.

Generated-code tools also accept a packages array, allowing dependencies such as Pandas or NumPy to be installed before execution. Duque describes the native tools as MCPs maintained by Heroku; custom MCPs extend the same endpoint. In the demonstrated API, agent responses are streaming only because tool work may take time. Application and database permissions depend on the tool being invoked.

24:4624:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:46 · section reference included

Ask the server for its current time

The first agent question asks for the server’s current date and time. A model’s training data cannot establish the live clock of this particular server, so the request enables Dyno Run Command. The payload includes the model ID and message, identifies the tool as a Heroku tool, selects the target application, and supplies date as the command with a description of its output.

The notebook sends the payload to the agents endpoint and extracts tool-call information from the stream. Execution targets ai-engineer-workshop; the notebook waits for the command, receives the date and time, and then displays the inference response based on that result. Replacing date with a deployed script would use the same mechanism for data extraction or access to an external service. The tool supplies current evidence before the model composes the answer.

Jupyter notebook output shows agent tool execution, a message about checking the server time, and the returned date and time above the next Code Execution demo.Open full source frame
The notebook displays the server date and time returned by the agent’s tool call.
28:2728:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

28:27 · section reference included

Keep the computation, change the runtime

Code execution lets the model produce the program as well as request its execution. Duque enables CodeExec Node and asks for the 13th Fibonacci number. The notebook displays the generated JavaScript with syntax highlighting, then receives the result from execution on Heroku. The Node demonstration returns 233 for the 13th Fibonacci number.

Using the convention F₀ = 0 and F₁ = 1, the computation fits in a short JavaScript program:

javascript

function fibonacci(n) {
  let a = 0;
  let b = 1;
  for (let i = 0; i < n; i++) {
    [a, b] = [b, a + b];
  }
  return a;
}

console.log(fibonacci(13));

Duque then changes the selected tool from Node to Go. The model generates Go code, Heroku compiles and runs it on a dyno, and the response gives the same result with an explanation. Python and Ruby are further runtime choices; participants are encouraged to change the requested operation as well as the language.

30:5631:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:56 · section reference included

Retrieve an implementation before running it

The next example gives the agent two tools: HTML to Markdown and Code Execution Python. The request is specific about both the source and the task: use the Python snippet from Wikipedia’s Euclidean algorithm page to calculate the greatest common divisor of 252 and 105. That instruction makes retrieval part of the computation instead of leaving the model to produce an implementation from memory.

The managed loop now crosses a dependency between tools:

  1. Fetch the Euclidean algorithm page as Markdown.
  2. Read the page and extract its Python implementation.
  3. Prepare and execute Python using the requested inputs.
  4. Return an explanation grounded in the retrieved implementation and execution result.

The notebook shows page retrieval, then code generation and execution. Duque points out the additional wait involved in multiple tool runs, invites participants to inspect and modify the code, and finally receives an explanation describing the implementation and calculation. The important extension is that the output of retrieval becomes input to execution.

33:1933:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

33:19 · section reference included

Attach a read-only database before querying it

The notebook application initially lacks access to the shared solar-energy database. Before enabling database tools, open the team’s ai-engineer-data application and its Resources page. Heroku’s maintained Postgres tools operate only on read-only followers. Duque explicitly leaves the primary database untouched; write access through a separately deployed MCP would be a different capability with risks borne by its operator.

On the follower, select Manage Attachments and add the Jupyter application. Then inspect the consuming application’s Resources page to find the database’s attachment name. One application uses DATABASE; Duque’s ai-engineer-workshop uses HEROKU_POSTGRESQL_AQUA. These application-specific names identify the connection configuration that the tool should use. The selected dashboard frame shows this attachment mapping.

Heroku Resources page with a Postgres attachment popover listing four application attachments, including HEROKU_POSTGRESQL_AQUA and DATABASE.Open full source frame
The Heroku dashboard shows database attachments and their application-specific names.

An attachment makes the database available to the application; a name selects that available database. Tools cannot reach another application’s database merely by naming it. If the appropriate database is already attached locally, no additional attachment is needed. This keeps resource access separate from the natural-language question the agent will receive.

35:4636:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

35:46 · section reference included

Discover the schema, then investigate energy savings

Postgres Get Schema and Postgres Run Query receive the target application and database identifiers. Duque describes the example dataset as roughly two months of hourly energy consumption and production for a solar-energy company. The question is how much energy has been saved in the last 30 days. To answer it, the agent must discover the relevant tables and fields, generate SQL, execute the query and interpret the returned data.

The first tool retrieves the schema, revealing tables including metrics, products, systems and users. The agent then generates and runs its query. Duque expresses complete confidence that schema knowledge will make the query run, but schema discovery grounds table and column references; it does not guarantee correct SQL or a correct interpretation of energy savings. The demonstration proceeds from returned data to further investigation.

The agent asks for additional queries to break the result down across three systems with different performance. Its final report includes savings, per-system results and an energy deficit. This is the useful behavior of the loop: one user question can trigger schema discovery and several queries before the final answer. Duque then proposes adding Python execution to generate a Matplotlib chart from the data; that chart is an extension, not a completed part of this demonstration. More tool calls also increase the total time required. A further exercise combines PDF extraction with code execution.

39:1639:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

39:16 · section reference included

Give an existing MCP a Heroku entry point

For deployment, Duque uses a fork of the official Perplexity Ask MCP. The main adaptation is a Procfile, Heroku’s declaration of how application processes start. He adds an MCP entry point and explains that process types beginning with mcp are recognized as MCP processes. The server continues to communicate over standard input/output. A deployment button packages this into a Node.js application that can be attached to the consuming app.

Duque does not have a valid Perplexity API key and enters a placeholder for the deployment exercise. That lets him demonstrate building and making the application available, but it does not establish authenticated Perplexity execution. Participants are left with an exercise: attach the deployed server and write the code to invoke it with working credentials.

46:0746:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

46:07 · section reference included

Call the same tools from an MCP client

After the earlier Brave news response completes, the workshop turns to consumers outside the managed agent endpoint. Cursor, Claude Desktop and agents built on other platforms can use the hosted tools through Toolkit Integration. In the recording, this gateway aggregates attached MCPs behind a server-sent events endpoint authenticated with a bearer token; OAuth support is described as work in progress.

Duque adds the newly deployed Perplexity server to the toolkit and uses the same token as inference. The notebook installs the Python MCP package, creates a client for the toolkit URL and sends the key in an Authorization header. The client then lists the tools and calls Brave Web Search. Although the notebook itself is hosted on Heroku, this demonstrates access through an ordinary MCP client rather than through Heroku’s managed agent loop.

The client connects and discovers Brave Local Search, Brave Web Search and Perplexity Ask. It executes Brave Web Search and prints the returned search content without further processing. Perplexity’s appearance in discovery shows that the server is exposed through the toolkit, while the successful invocation shown here is Brave’s. Duque also describes using the same gateway approach from Cursor.

Jupyter notebook output shows “MCP Client connected!”, a list of available tools and a block of search-result text beneath the client code.Open full source frame
The MCP client displays its available tools and a returned search result.
47:5848:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

47:58 · section reference included

Use the OpenAI SDK, then continue experimenting

The final technical example returns to inference through the OpenAI SDK compatibility interface. Duque estimates chat-completion compatibility at 95%, with 99% as a goal; he supplies no measurement method. Some parameters are unsupported, so field-level compatibility matters more than that estimate when porting an application. The demonstrated integration configures the OpenAI SDK with Heroku’s API key and URL.

In Python, the client configuration and basic request have this shape:

python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["INFERENCE_KEY"],
    base_url=os.environ["INFERENCE_URL"].rstrip("/") + "/v1",
)

response = client.chat.completions.create(
    model=os.environ["INFERENCE_MODEL_ID"],
    messages=[{
        "role": "user",
        "content": "Explain the concept of Managed Inference in one sentence.",
    }],
)
print(response.choices[0].message.content)

The SDK call in the recording returns an inference response. By this point, the notebook has exercised both interfaces: familiar chat completions for model output, and a managed agent endpoint that runs tools and feeds their results back into inference. The toolkit additionally exposes those tools to independent MCP clients.

The workshop closes with continued experimentation rather than another deployment step. Duque extends the event team’s access through the weekend and explains that Heroku has no free tier at the time of the talk, with a return under consideration. He points participants to Dev Center documentation, the Heroku AI website, the Heroku community on Twitter/X and the slides shared in Slack, with questions welcome through social follow-up.

50:5051:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

50:50 · section reference included

Resources

From the talk

  • Deploy Jupyter Notebook or JupyterLab on Heroku with password protection and PostgreSQL-backed notebook storage.

Updates since the talk

Transcript download
Full transcript

Loading transcript…