AI Engineer Summit 2023

AI Engineering 101

Noah Hein· Instructor, Latent Space University3:02:24

Loading slides…

Read the talk

AI Engineering 101: Build a Telegram Bot Across Five AI Capabilities

Build from a working Telegram bot to conversational memory, document retrieval, code generation, images and voice transcription, while tracing the engineering choices behind each capability.

From a talk by Noah Hein

Before you start: Basic Python, terminal commands and API concepts will help; following the bot exercises also requires Telegram Messenger and appropriate API credentials.

What should a new AI engineer know how to build?

What should you know before walking into an AI engineering conference where every session seems to assume you have read every paper and tried every API? The opening problem is a shared baseline: learn techniques that already work, understand how to turn an AI idea into software, and know where to look when an implementation is unfamiliar. This Latent Space University workshop, taught by Noah Hein, is a sampler rather than a promise of expertise in one day. It precedes a Gradient lunch-and-learn, an Anthropic prompting workshop and Charles Frye’s AI Engineering 201 session. Following the hands-on work requires Python and Telegram Messenger.

The project moves through five capabilities: calling a chat model, understanding tokens and embeddings, generating code, generating images, and transcribing speech. The connecting thread is programmatic access: taking a model response and doing something useful with it inside an application. DALL-E 3 is still described here as an early-access release. Hein positions that work as an opportunity distinct from building the underlying model infrastructure. His search-trend slide uses “AI” as a proxy and extrapolates beyond data ending in 2023; it is a motivation for learning these skills, not a measured forecast of engineering jobs.

Slide pairing a software estimation comic with bullets about AI capabilities, faster development, and practical AI applications.Open full source frame
The rise of the AI engineer.
0:281:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:28 · section reference included

Start with a bot that only answers /start

All the capabilities will live in one Telegram bot. The workshop’s broadcast channel distributes repository links and resources, while checkpoint branches let participants resume at a working stage instead of reproducing every edit in real time. BotFather provisions the bot itself:

  1. Send /newbot to BotFather.
  2. Choose a display name and a unique bot username, such as one based on your GitHub handle.
  3. Save the HTTP API token and open the bot’s shareable chat link.
  4. Start with the workshop repository’s main branch and follow its README to install Python dependencies and prepare the environment.

The repository named in the recording is inheingit/ai101; it was unavailable when checked on August 28, 2026. Its checkpoint branches are part of the original workshop workflow.

The application needs two credentials: TG_BOT_TOKEN identifies the Telegram bot, and OPENAI_API_KEY authorizes model calls. The setup turns .env.example into .env, where those values belong. Hein walks through obtaining an OpenAI key and shares a temporary workshop key to keep the room moving. That shortcut does not change the architecture: the Python process connects independently to Telegram and OpenAI.

Inside src/main.py, imports and environment loading come first, followed by a messages array and logging. Telegram callbacks receive an update, containing information about the incoming message and chat, and a context, which provides access to the bot. The initial start callback calls context.bot.send_message with update.effective_chat.id and a greeting. A CommandHandler maps /start to that function; registering it on the token-configured application makes the behavior available. Polling then retrieves incoming updates. Run python src/main.py from the repository root and press Start in the bot’s Telegram chat. The application is running on your machine, even though its interface is on Telegram.

The setup questions expose three separate failure modes. A different file or line number can mean the wrong repository state; an import error can mean the Python requirements were never installed; an unresponsive ordinary message can simply mean there is no handler for it yet. Installing Telegram Messenger does not install its Python client library, and copying main.py alone does not install dependencies. BotFather is Telegram’s first-party credential service, but the token only connects the application—it does not supply application behavior. At this checkpoint, repeated /start commands work and “hello” does nothing.

9:0610:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:06 · section reference included

Conversation memory is an array you resend

The first checkpoint adds a chat callback. Import openai, load the API key, and append the incoming Telegram text to messages with the role user. The array starts with a system message describing the assistant’s job. Subsequent turns alternate between user and assistant. GPT-3.5 Turbo receives that array through openai.ChatCompletion.create, with model="gpt-3.5-turbo". The snippets here retain the recording’s historical OpenAI SDK interface; the later Python SDK v1 rewrite introduced explicit clients and changed these calling conventions.

The API returns an object containing both the answer and metadata. Extract choices[0].message.content, append that answer to the history as an assistant message, and send the text back to the originating Telegram chat. The model only sees the conversation you include in the request. Keeping both sides of the exchange is what makes follow-up questions possible.

python

import os
import openai

openai.api_key = os.environ["OPENAI_API_KEY"]
messages = [
    {"role": "system", "content": "You are a helpful assistant that answers questions."}
]

async def chat(update, context):
    messages.append({"role": "user", "content": update.message.text})
    completion = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=messages,
    )
    answer = completion["choices"][0]["message"]["content"]
    messages.append({"role": "assistant", "content": answer})
    await context.bot.send_message(
        chat_id=update.effective_chat.id,
        text=answer,
    )

This reproduces the workshop’s in-process history pattern. A bot shared by unrelated users would also need to separate their histories; one global list does not provide that isolation.

Python editor showing the async chat function above a terminal: append the user message, call OpenAI, append the assistant response, and send the response through the bot.Open full source frame
The chat handler saves user and assistant messages, requests a model response, and sends the reply to Telegram.

Register the callback with MessageHandler(filters.TEXT & ~filters.COMMAND, chat). The lowercase filters module selects text messages, and ~filters.COMMAND excludes slash commands so ordinary conversation does not consume /start or later feature commands. Defining the function, constructing its handler and adding that handler to the application is the pattern repeated throughout the workshop. The first checkpoint also contains a question handler, but that belongs to the retrieval feature explained next.

The live test makes the memory mechanism concrete. Hein asks “Who is Simon Cowell?” and receives a description. He then asks “What is his net worth?” The second request includes the first question and answer, allowing the model to resolve “his.” Restarting the Python process recreates the initial messages array and loses that reference. Persistent chat memory would therefore require storing and restoring the history, not merely keeping the same Telegram conversation open.

System prompts also change behavior: Hein points to two bots conversing as a pirate and a nobleman, and to a game where progressively stronger instructions try to prevent disclosure of a secret. But continuously appending conversation introduces a different limit. Eventually the request exceeds the model’s context capacity. Hein expects this simple implementation to log an error and leave the user without an answer because it has no failure response. Retrieval offers a way to select relevant earlier material instead of resending an unlimited history.

29:2130:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

29:21 · section reference included

Tokens measure input; embeddings support search

A token is a unit the language model processes and generates. Token counts affect both billing and context capacity. Hein illustrates the boundary with a hypothetical model that accepts 100 tokens: a 101-token question already exceeds it. His rough estimate of four English characters per token is useful for intuition, but actual counts depend on the tokenizer, language and text. He contrasts Spanish tokenization costs with English and mentions Replit’s code-oriented tokenizer.

Token boundaries can also be surprising. Hein cites rawdownloadcloneembedreportprint as an anomalous single-token example discussed on LessWrong, then recommends experimenting with the OpenAI tokenizer playground to see how text is split. The broader lesson is that a tokenizer belongs with the model trained to use its token IDs. You still write a system prompt as ordinary text, but you cannot freely replace the model’s tokenizer with an unrelated one.

An embedding is a vector of floating-point numbers representing an input for operations such as semantic comparison. The slide places dogs near cats and strawberries near blueberries to illustrate related meanings. This enables a different approach to a large corpus: instead of putting all of Wikipedia into a prompt to answer a question about Taylor Swift, find relevant passages first. The original text remains necessary; the vector helps locate it.

The preparation job lives in embedding/embed.py and runs from the repository root with python3 embedding/embed.py. It uses Pandas for tabular data, tiktoken for token counting and LangChain’s RecursiveCharacterTextSplitter for splitting text. Hein warns that it may spend several minutes without useful terminal feedback. A participant offers an intuition for why splitting matters: representing a whole book with one fixed-length embedding is like asking for the average color of a film. A scene-level representation tells you more about a particular moment. That analogy concerns loss of detail; it does not mean an embedding model can accept an arbitrarily long book.

LangChain supplies useful components, but the underlying operations do not require the framework. The immediate operational issue is a shared API key: simultaneous embedding jobs can hit rate limits. Hein suggests retrying, then using a personal key if necessary. Hein estimates the entire workshop at about $0.05 in API charges. For participants blocked by the embedding job, a supplied embedding.csv contains its precomputed output. The corpus is a raw scrape of MDN documentation, so everyone can continue to retrieval without independently regenerating the same vectors.

46:5447:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

46:54 · section reference included

Clean, chunk and embed the MDN corpus

Ingestion begins with ordinary data preparation. The script replaces newlines with spaces and reconstructs source paths from filenames whose slashes were encoded as underscores. It includes the restored URL with the page text, preserving enough provenance for an answer to cite its source later. Contributor files and unusable scrape results—such as login or JavaScript-required pages—are omitted. The remaining filename/text pairs become a Pandas dataframe and a scraped CSV.

Next, tiktoken loads cl100k_base and counts tokens for each text row. A separate splitting step turns oversized pages into smaller inputs, skips empty rows, collects the shortened text and counts tokens again. Keep the units straight: recursive character splitting with length_function=len measures characters. The demonstrated size setting of 1000 therefore does not establish 1,000-token chunks. Nor does a 3,000-token page necessarily become exactly three chunks, despite the spoken illustration. The token count and the splitter’s length function are distinct controls.

Each resulting text chunk goes to text-embedding-ada-002 through openai.Embedding.create. The script extracts data[0].embedding, adds it as a dataframe column and writes the processed CSV. The model’s published specifications are 1,536 dimensions and an 8,192-token input context. Those limits describe the model; the smaller chunking choice serves retrieval as well as input safety.

A chunk can fit the API and still be a poor search unit. Cutting after “Little Red Riding Hood ran to the...” removes the completion of the thought. A retrieved fragment then lacks information needed to answer the question. Libraries can automate splitting, but data boundaries remain an application decision: embedding choice, chunk size and preservation of complete meanings all affect what the generator eventually receives.

1:00:051:00:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:00:05 · section reference included

Retrieve text before asking for an answer

The online half of the pipeline lives in src/questions.py. It loads the saved dataframe and converts stored embedding representations into NumPy arrays for numerical operations. An array with 1,536 entries is still a vector in a 1,536-dimensional embedding space; NumPy does not compress that space into one dimension. The workshop uses openai.embeddings_utils.distances_from_embeddings to compare the question vector with document vectors.

create_context takes a question, the dataframe and a maximum context length. It embeds the entire question—for example, “What is an event in JavaScript?”—and computes a cosine distance against each chunk. Smaller distances rank first. Hein illustrates this with related berries versus unrelated objects, then applies the same idea to relevant JavaScript material versus less relevant CSS material. The application adds a distance column, sorts it and collects the associated text, not the vectors, for the prompt.

The workshop sets a 1,800-token budget for retrieved context. It walks ranked chunks until that budget is reached, producing a bounded text block for answer_question. A compact implementation of that ranking-and-packing operation makes the separation visible:

python

import numpy as np

def select_context(question_embedding, rows, max_tokens=1800):
    query = np.asarray(question_embedding, dtype=float)
    vectors = np.asarray([row["embedding"] for row in rows], dtype=float)
    similarities = (vectors @ query) / (
        np.linalg.norm(vectors, axis=1) * np.linalg.norm(query)
    )
    order = np.argsort(1.0 - similarities)

    selected = []
    used_tokens = 0
    for index in order:
        row = rows[int(index)]
        cost = row["n_tokens"] + 4
        if used_tokens + cost > max_tokens:
            break
        selected.append(row["text"])
        used_tokens += cost
    return "\n\n###\n\n".join(selected)

Here each row holds a chunk’s text, token count and embedding, with a small separator allowance. The retrieved-context budget is only part of the eventual request: instructions, the question and the generated answer also need room.

The embedding model and the answer model have different jobs. Ada supplies the representation used for retrieval; GPT-3.5 Turbo receives the selected text and writes the answer. The prompt asks it to answer from the supplied context and respond “I don’t know” if that context is insufficient. Because ingestion preserved source URLs, the prompt can also request a source link. The application has already filtered the large corpus before generation begins.

The answer call introduces temperature, max_tokens, top_p, frequency and presence penalties, and stop conditions. These control generation; they do not improve the retrieved documents. Hein initially describes sampling through analogies to answer variety and curated search results. The important distinction, clarified in the following discussion, is that sampling acts on candidate next tokens, not on a ranked list of complete answers. A fluent response can still invent a fact, such as placing Neptune closer to the Sun than Earth.

1:09:401:09:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:09:40 · section reference included

Choose semantic search only where it helps

An audience question about model choice reveals an essential boundary. The generator need not share a model family with the embedding system, because it receives retrieved text. But query and document embeddings must occupy a compatible vector space; arbitrary vectors from different models are not interchangeable merely because they contain numbers. A participant recommends trying open-source alternatives and names Alibaba’s GTE as a then-leading option. Its published model card supports comparing it with ada-002, though it does not establish the exact live leaderboard ordering at the workshop. Changing the embedding model means reconsidering both corpus and query encoding.

The cost discussion is similarly historical. A participant recalls an OpenAI claim that embedding the internet could cost $50 million. Hein estimates about $0.04 to embed a scrape he describes as roughly 80% of MDN. These are attributed estimates without a billing record, corpus count or complete calculation, rather than prices to budget against today.

The audience then separates three sampling concepts:

ControlWhat it changes
top_kNumber of candidate next tokens retained
top_pCumulative probability mass of retained candidates
temperatureHow concentrated or spread out token probabilities are

Thus top_p=0.1 does not mean “the best ten percent of answers.” Lower temperature reduces randomness, but it does not provide a guaranteed repeatability percentage. The API parameter definitions formalize this probability-based interpretation.

Structured metadata creates a different retrieval problem. Suppose records have descriptions plus Boolean pass/fail values, IDs or categories. An audience participant recommends filtering exact attributes first, then applying semantic search to the remaining text. Embeddings are lossy; an exact constraint should remain exact. If failure is instead buried in a natural-language description, combining lexical search with embedding search may recover useful signals that either method misses alone. The self-querying diagram shown during this discussion makes the separation visible: a query constructor and translator produce a search query and metadata filter for the vector store.

LangChain documentation showing a self-querying diagram with a query constructor, query translator, metadata filter, and vector store.Open full source frame
Self-querying retrieval combines a query with metadata filters.

Abstention and chunk sizing also depend on the corpus. Hein reports that an event-emitter question returned “I don’t know,” and suspects the scrape omitted the relevant material. He also reports frequent abstention outside the corpus, without measuring a reliability rate. For chunk size, dense reference documentation may need less surrounding text than a narrative: one page from the middle of The Lord of the Rings can be insufficient for a chapter summary. The discussion points to LangChain’s many splitting strategies and its text-splitter playground, but the decision still requires trying representative questions against actual chunks.

1:20:091:20:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:20:09 · section reference included

Make /question a separate application path

Back in main.py, import answer_question, load the embedding CSV and convert its vectors at startup. This increases restart time, but keeps the example easy to inspect. The new question callback passes the dataframe and update.message.text into the answer function, then sends its result to Telegram. Registering CommandHandler("question", question) completes the route. Hein recommends keeping the local NumPy solution until storage or search performance becomes a real bottleneck; pgvector and Pinecone are possible later steps, not prerequisites for understanding retrieval.

The demonstration now has two distinct behaviors:

Input pathContext suppliedHistory behavior
Ordinary textSystem prompt and chat historyAppends the conversation
/questionRetrieved MDN chunks and answer instructionsDoes not append to ordinary chat history

/question What is CSS? produces an explanation of Cascading Style Sheets. The event-emitter question returns “I don’t know,” which Hein again attributes tentatively to incomplete scraping. An ordinary question about Taylor Swift takes the general chat route instead. Retrieval has not changed the model’s underlying knowledge; it has changed the request assembled by the application. This same pipeline can power search over a product’s own documentation.

That separation explains why a question can receive a useful answer in ordinary chat and an abstention under /question. The latter is asked to stay within supplied evidence. It also exposes a security boundary: an attendee reports bypassing those context-only instructions by asking the bot to ignore previous instructions and use its general knowledge. Hein tentatively suggests stronger system prompts while acknowledging limited experience with prompt-injection defenses. A context-only prompt is not an enforceable security boundary, and good retrieval does not prevent the generator from hallucinating.

Sampling choices should follow the task. A poem benefits from variation; a CSS explanation usually benefits from consistency. Hein says this example uses a temperature of 0.5, and describes changing either temperature or top_p at a time, trying several prompts and inspecting the outputs. The discussion finally corrects the temperature range to 0–2, with a default of 1 for the API under discussion. These settings govern sampling behavior, not factual certainty.

1:28:071:28:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:28:07 · section reference included

Preserve meaning at boundaries and across questions

The manual pipeline exposes controls that a higher-level library call can hide. A text-splitter preview shows boundaries, but evaluating those boundaries still requires embedding the chunks and checking downstream answers. Overlap repeats text around a boundary so at least one neighboring chunk is more likely to contain a complete thought. Returning to Little Red Riding Hood, the purpose is to avoid retrieving only the sentence prefix while its meaning lives in the next chunk.

Hein has not compared distance metrics; he keeps cosine fixed while experimenting with other parts of the pipeline. Multi-part questions create a more obvious constraint. Asking about bitwise operations, CSS and events in one request embeds several meanings into a single query and makes their retrieved evidence compete for the same context budget. A participant proposes decomposing the question, retrieving context separately for each part, then combining the results. Hein considers that a reasonable extension but has not tried it himself.

This is where application engineering can add value beyond a bare chat interface: several carefully arranged retrieval and prompting steps can serve a specific task better than one undifferentiated request. The baseline remains simple, however. One whole user question produces one embedding for comparison with the document chunks. An audience member’s final question about confusing tokens with embeddings ends as the session moves into its break; the useful distinction already established is that tokenization produces the model’s input units, while embedding produces a vector representing an input.

1:38:201:38:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:38:20 · section reference included

Use examples to specify generated code

After the break, the workshop treats code as another kind of generated text and switches to GPT-4. Hein uses an AP-test comparison chart to motivate the switch on reasoning grounds, then explains why repeated code structures are useful to next-token prediction: an opening div tag strongly suggests a corresponding closing tag. Familiar frameworks also provide established conventions. His example is Rails and its Devise authentication package, where a configuration request can draw on common file layouts and generated templates.

He frames this as a combination of handwritten software and learned behavior: Software 1.0 expresses instructions directly, Software 2.0 changes behavior through training data, and his Software 3.0 framing combines the two. The immediate implementation is few-shot prompting. Give the model example inputs paired with the outputs you want, then ask it to continue using the same conventions. Running or rendering the resulting code is a separate optional step.

The second checkpoint adds a code prompt containing two examples: a blue box with three yellow circles and a red outline, and a red button labeled “Click Me.” Both use JSX with inline styling. That prompt becomes an additional system message, and a /code handler sends the user’s request to GPT-4. The live request asks for a red box containing a blue “yes” button. The response follows the established JSX and inline-style format without the user restating those requirements. To target Vue or Nuxt instead, change the instructions and examples to those conventions.

A proposed extension turns generated HTML into an SVG string, converts the SVG to PNG bytes and returns the image to Telegram. Each stage introduces another opportunity for failure. Even the smaller centered-div demonstration deserves inspection: its response combines flex alignment with absolute positioning and offsets. Hein accepts that it follows the request while calling the code imperfect. He then describes building a Rails RAG application without prior Ruby experience, asking for incremental changes and receiving a favorable review from an experienced former boss. The practical workflow is to supply a bounded request, inspect the result and continue from there.

2:01:382:01:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:01:38 · section reference included

Delegate computation and improve the prompt

The bonus rendering example makes iteration visible. A request for yes/no buttons first produces a blank screen; another correction still fails. Repeating the intended green yes button and red no button eventually produces a useful rendering. The accompanying arithmetic example uses generated expressions with a Python evaluation function shown as math.evaluate. Hein explicitly warns against placing unrestricted eval in a production application. The useful principle is to let a computational tool perform arithmetic rather than rely on a language-only answer, while keeping execution under application control.

Homework / Bonus slide with three suggestions and a chat screenshot containing yellow circles on blue and green and red buttons.Open full source frame
Homework examples for function calling, HTML rendering, and math.

A question about Vertex AI raises a portability issue: providers may offer similar chat or completion capabilities under different API names. A continuation model can complete “yellow brick” with “road”; a chat interface packages role-labeled messages. Neither label, by itself, creates memory. As the earlier bot showed, a follow-up about Mark Cuban’s net worth requires the relevant prior context to be supplied somehow. The request structure and history handling matter more than whether an endpoint is named chat_completion.

For better prompts, Hein suggests asking the model what additional information it needs when a response misses the target. Its answer can reveal missing constraints for the next attempt. He cites Kent C. Dodds’s preference-setting prompt as an example: concise answers, no unnecessary reminders that the assistant is an AI, and no unwanted politeness or emojis. The goal is not to collect an immutable library of magic phrases. It is to learn which instructions and examples change output in useful ways as models and APIs evolve.

2:13:202:13:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:13:20 · section reference included

An avocado armchair is easy to request—and hard to guarantee

Image generation begins with the memorable DALL-E avocado armchair: an ordinary sentence requests a visual object most people could imagine but could not draw. The failures are equally instructive. “Salmon in the river” yields an interpretation that satisfies the words without satisfying the intended scene. Hands can look malformed, prompting users to avoid them through instructions such as placing hands behind a subject’s back. Counting and lettering introduce other constraints.

For the prompt asking for four people in a forest, Hein reports that one of four displayed results has the requested count—a 25% hit rate for that small example. Hein says he spent roughly 30 minutes trying to obtain satisfactory “fox” lettering. These demonstrations distinguish a plausible-looking image from one that satisfies a specific brief. DeepFloyd IF appears as a stronger text-rendering example; it also uses diffusion, specifically cascaded pixel-space diffusion with a T5 text encoder, rather than representing a non-diffusion alternative.

Personalization supplies the product connection. Hein cites Lensa’s avatar business and Pieter Levels’s image applications as revenue success stories, then describes RoomGPT: upload a photograph of a living room and request a different style, such as mid-century decor. Midjourney offers another route to image creation. These products let a nontechnical user turn a visual intention into something they can see, rather than merely receive information they might also find through search.

2:20:112:20:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:20:11 · section reference included

Return image bytes, then consider inferred routing

The third checkpoint adds requests and an image callback. openai.Image.create receives a prompt, an image count n and a size. The response contains a URL; an HTTP GET retrieves the image bytes, and context.bot.send_photo delivers them to the originating chat. A /image command handler connects this callback to the application. The essential transport change is from sending text to sending media content.

Code editor showing an image handler that requests an image, retrieves its URL, and sends the image content as a photo, beside a small implementation slide.Open full source frame
Image-generation code retrieves an image and sends it to Telegram.

Function calling proposes an operation; application code executes it. Describe an add function and the model can return its name with arguments such as 2 and 4. It has not itself invoked your Python function. The application decides whether and how to perform that call. The same mechanism could replace explicit slash commands: a request for a photorealistic robot hugging a koala could produce an image-generation call with the relevant prompt. Hein calls this a kind of smart if/else and offers it as an extension to the existing handlers, not behavior already implemented in the basic bot.

The live image request is “The Iron Giant sitting in the forest, 4K.” Here 4K is part of the prompt, not evidence of the returned image’s resolution. Hein clarifies that the workshop endpoint uses DALL-E 2, while DALL-E 3 is rolling out. He recommends exploring alternatives, including locally runnable Stable Diffusion, and contrasts the result with the widely circulated fashion image of the Pope. A simpler request—“A koala sitting in a tree”—produces a result he finds more satisfying.

Images make prompt specificity especially consequential because many users cannot repair the result themselves. Hein asks groups of three to five to spend ten minutes experimenting, with suggestions to separate ideas using commas and describe the setting as well as the subject. The exercise turns a working API connection into a way to compare intent with output.

2:24:542:25:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:24:54 · section reference included

From generated characters to codebase diagrams

The break conversations extend the workshop into participants’ own workflows. One vision result appears to recognize code handlers, though some are wrong; adding the fact that the code belongs to a Telegram bot is proposed as useful context. A participant describes generating team members’ characters from choices such as hero versus villain and fantasy versus science fiction. A CTO’s magic-elephant character then becomes input to Runway ML for animation. Hein compares the shown result favorably with his own less-smooth Mage experiments. These are participant examples of composing outputs across tools.

Diagrams offer a more directly technical composition. Hein describes asking for Mermaid syntax from his function-calling code: a text model can produce a structured representation that another tool renders visually. A participant explains a Cursor workflow that indexes documentation from a URL, references it alongside a codebase, and generates Mermaid or UML architecture diagrams. They use this both for onboarding and for understanding unfamiliar agent repositories before explaining them to a team. The participant also reports a familiar VS Code-style environment, imported extensions and citations to the documentation used.

Heavy experimentation changes both attention and cost. One participant warns that exploring the assistant can displace the work originally planned. One participant reports $140 in API usage; another reports more than $100 compared with a usual $5–$10 per month. They discuss a $20-per-month Cursor subscription as attractive relative to their own-key spending, but these are personal historical expenditures, not matched usage comparisons. A participant also describes using different models for different outputs, including GPT-3.5 function calling and fine-tuning.

Hein contrasts the low cost of workshop experimentation with a crypto transfer that cost him $6 to send $10. His challenge to exceed a dollar experimenting in the workshop is immediately qualified by participants pointing to GPT-4, especially its larger-context variant. One participant selectively uses GPT-4 32K for larger files and codebase searches. The group exercise ends with a request to send favorite image prompts and outputs to Hein for sharing in the broadcast channel.

2:33:382:33:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:33:38 · section reference included

Context confusion and delivery failures are application problems

A later code-generation question qualifies the earlier optimism. An attendee has encountered architectural context limits and uncertainty about which code in a long conversation the model is referencing. Hein has not personally hit the same limit, but recommends starting a new conversation when switching to a different code path. Mixing byte-processing work with a separate encoding problem can bring irrelevant prior assumptions into the answer. File uploads through GPT-4’s Code Interpreter are another option he has found useful, though sometimes frustrating; potential Replit improvements are discussed speculatively.

The attendee describes enterprise work using and fine-tuning StarCoder, including its developer opt-out provisions. Hein suggests exploring Llama and Llama 2, while acknowledging he lacks hands-on code-generation experience with them. A question about combining structured and unstructured information in embeddings is raised but deferred. These exchanges leave model selection tied to the actual environment and task rather than resolving it with a universal recommendation.

The image feature also encounters a concrete integration failure. An attendee reports a timeout while posting an image to Telegram, while posting the image URL instead appears to work. Hein acknowledges the report without diagnosing or fixing it. Another attendee asks for a detector or postprocessor that could identify unnaturally rendered images and make them look more normal. Hein knows no such tool; he offers only a tentative analogy to some systems’ post-generation safety filtering and regeneration. That discussion does not establish an automatic image-quality repair stage.

2:43:392:43:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:43:39 · section reference included

Turn voice into input for the existing stack

Automatic speech recognition, or ASR, turns audio into text—the same kind of input the bot already knows how to process. Alexa, Siri and Google’s voice interfaces provide familiar examples. Hein introduces Whisper and its transformer architecture, contrasting it with earlier recurrent-network approaches. Training speech systems also requires audio paired with useful labels, a different data-collection problem from gathering ordinary text.

The workshop uses the already-configured hosted API, but local and browser-based paths are also available:

  • Browser exploration: A Hugging Face Space can provide a low-friction speech-recognition demo.
  • Local inference: Open-source Whisper runs on local hardware; whisper.cpp is a C/C++ implementation Hein recommends exploring for CPU execution.
  • Speaker separation: WhisperX combines transcription with alignment and optional diarization.
  • The reverse direction: Tortoise TTS is mentioned for generating speech from text.

Diarization asks who spoke when within one audio file. In Hein’s Noah-and-Justin podcast example, the system can assign speaker IDs to turns; associating those IDs with real names requires additional information. Overlapping speech and speaker assignment can still be imperfect.

2:51:112:51:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:51:11 · section reference included

Transcribe a voice note and show the work in progress

Telegram voice memos provide the final input type. The workflow is straightforward: record a note, retrieve the audio file, send it to Whisper and return its transcription. This is the fourth checkpoint. During the live walkthrough, Hein corrects git branch to git checkout so the editor actually switches to the implementation.

Speech-To-Text Implementation slide listing creation of a Telegram voice file, downloading it, passing it to Whisper, and returning the transcription.Open full source frame
The voice-note transcription workflow.

The callback reads update.message.voice.file_id, asks context.bot.get_file for the file, and uses download_to_drive to save it locally. The filename includes the voice ID to avoid collisions and ends in .ogg. These recordings remain on the application server in the demonstrated implementation, making file retention a real operational consideration rather than an invisible API detail.

Before transcription completes, the bot tells the user that work is underway. Hein uses the unpleasant alternative—a silent wait followed by a large block of text—to explain why streaming or progress feedback matters. Streaming is not implemented here, so acknowledgment provides the minimum useful feedback. The historical transcription call is openai.Audio.transcribe("whisper-1", audio_file), followed by a reply containing the transcript. A compact callback expresses the same sequence:

python

from pathlib import Path

async def transcribe_message(update, context):
    voice_id = update.message.voice.file_id
    voice_file = await context.bot.get_file(voice_id)
    path = Path(f"voice_note_{voice_id}.ogg")
    await voice_file.download_to_drive(custom_path=path)

    await update.message.reply_text("Downloaded. Transcribing your voice note…")
    with path.open("rb") as audio_file:
        transcript = openai.Audio.transcribe("whisper-1", audio_file)
    await update.message.reply_text(
        "Transcript finished:\n" + transcript["text"]
    )

Unlike /question or /image, voice transcription uses MessageHandler(filters.VOICE, transcribe_message). The audio attachment itself selects the behavior, so no /voice or /transcribe text command is necessary.

For the final test, Hein records a message on his phone introducing himself to the room and saying “rubber baby buggy bumpers.” Telegram delivers it to the running bot, which reports the downloaded, transcribing and finished stages. The returned transcript gets the speech largely right but spells his surname Hine instead of Hein. The result is useful without being error-free, especially for a proper name.

The ending returns to what makes these APIs valuable together: once speech becomes text, it can enter the same retrieval, conversation or generation paths already built. The phone demonstration completes a working interface across modalities, while Hein looks ahead to more personalized and interactive applications. The next product decision is what those connected capabilities should let a person accomplish.

2:55:072:55:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:55:07 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…