AI Engineer World's Fair 2024
Substrate Launch: the API for modular AI
Loading slides…
Read the talk
Substrate Launch: the API for modular AI
Substrate couples a graph SDK with a coordinated inference engine, making dependencies explicit and reducing the overhead of moving between model calls.
From a talk by Rob Cheung
Before you start: Basic familiarity with model API calls and Python is enough to follow the graph example.
When one inference run is not enough
What does it take to turn a model into a product that reliably summarizes documents or solves coding problems end to end? Substrate entered this problem space after about a year working with private clients. At his June 2024 AI Engineer World’s Fair session, Rob Cheung described the launch the previous week as the beginning of the team’s public introduction.
The starting observation is that successful AI products often use multiple inference runs, sometimes combining different types of models to perform a specific task. Even document summarization can benefit from a system of calls; end-to-end coding makes the need for a logical structure more apparent. The foundation model supplies a capability, while the surrounding system organizes how that capability gets used.
An explicit structure makes intermediate behavior inspectable. Substrate’s premise is that modular intelligence will be more effective than monolithic intelligence because developers can understand the system’s parts and relationships. That makes it easier to debug a failing step, extend the workflow, and evaluate intermediate decisions. When the decision tree is explicit, evaluation can ask where an error entered the process instead of judging only the final answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Describe the graph, then execute it together
Substrate’s proposed combination is speed, flexibility across AI products, and scale by default. Its architecture couples two components: a developer SDK for describing a computation graph and an inference engine built to execute that graph. Nodes cover image generation, speech transcription, text generation, JSON, embeddings, and code execution. The graph captures both the tasks and their dependencies.
A document-summary workflow illustrates the dependency model. One node generates a summary; another turns that summary into a title. In the Substrate Python SDK, a future output can become part of a downstream prompt:
python
import os
from substrate import Substrate, ComputeText, sb
substrate = Substrate(api_key=os.environ["SUBSTRATE_API_KEY"])
document = (
"The team moved the release to Friday. "
"The extra time will be used to test the payment flow."
)
summary = ComputeText(
prompt=f"Summarize this document in one sentence:\n{document}"
)
title = ComputeText(
prompt=sb.concat(
"Write a short title for this summary:\n",
summary.future.text,
)
)
response = substrate.run(summary, title)
print(response.get(summary).text)
print(response.get(title).text)
Here, summary.future.text expresses a dependency before the summary exists. sb.concat constructs a prompt containing that future value, and substrate.run submits the connected work. This example shows the graph-authoring pattern: the application describes how outputs will be consumed rather than retrieving each result and manually dispatching the next call.
The execution engine is the other half of the design. Because it receives tasks and their relationships, a coordinated compute cluster can optimize batching, caching, networking, concurrency, and physical placement, both statically and dynamically. Dependencies identify work that must wait; independent branches expose opportunities for concurrency. The graph therefore serves both as a readable application structure and as information the execution system can use to schedule work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The cost between model calls
A workflow assembled from separately dispatched API calls repeatedly crosses service boundaries. Cheung identifies DNS resolution, proxies, authentication, and balance checks as work incurred along that path. Coordinated execution aims to keep intermediate transfers inside the compute system instead.
| Execution pattern | Inter-step path |
|---|---|
| Separately dispatched APIs | Repeated service entry and request checks |
| Substrate graph | Internal node-to-node, process-to-process transfer |
Cheung reports hundreds of milliseconds of overhead per separately dispatched API step, versus microsecond-scale internal transfers in Substrate—a roughly 10,000× difference in inter-step transfer overhead. This is a comparison of handoff costs, not total model inference time. The recording supplies no payload sizes, hardware configuration, concurrency level, or latency statistic; the launch-era investor account describes output handoffs in a few milliseconds without enough detail to reconcile the measurement scopes.
The practical ambition is to make online applications with dozens of nodes feasible. If each handoff is expensive, adding useful intermediate steps can make an application too slow. Reducing that overhead gives developers more room to decompose the task without spending the interaction budget on communication between steps.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structured outputs make the connections useful
Fast transfers address the cost of connecting nodes; JSON decoding addresses a useful format for those connections. Cheung identifies JSON as a particularly valuable pattern in workflows involving multiple inference runs and says Substrate invested heavily in its JSON mode. He positions its reliability and speed as best in class, though the recording provides no supporting measurements.
The concluding product claim ties these mechanisms to higher-quality outcomes: a flexible graph lets developers compose capabilities, a legible structure exposes how the system works, and intermediate outputs make individual steps verifiable. The intended benefit is the ability to build and improve a system of inference runs whose behavior can be examined along the way.
The launch closes with an invitation to obtain credits through a QR code, visit substrate.run, or contact Cheung by email. Treat that as the historical access invitation: the Python package is now marked archived, and the former documentation destination redirects to Zo. The SDK example above illustrates the documented graph pattern, rather than establishing that hosted access or launch credits remain available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source and examples for connecting inference nodes through future outputs and executing computation graphs. Hosted-service availability is unverified.
Further reading
Launch-era investor account of Substrate’s graph SDK, atomic compute nodes and coordinated execution.
- Substrate SDK: June 2024 releaseDocumentation
Historical Python package release with usage examples and release metadata. PyPI now marks the project archived.
Full transcript
Loading transcript…