AI Engineer World's Fair 2024

Multi-model, multimodal, and multi-agent innovations in Azure AI

Cedric Vidal· Principal AI Advocate, Microsoft28:56

Loading slides…

Read the talk

Multi-model, multimodal, and multi-agent innovations in Azure AI

What is vegan on a photographed menu? That small question opens a progression through multimodal reasoning, local inference, retrieval grounding and agents that execute Python.

From a talk by Cedric Vidal

Before you start: Basic familiarity with language-model prompts and APIs is helpful; the Python example assumes you can read a short script.

Beyond a text-only application

What changes when an AI application can work across models, images, speech and executable tools? In this AI Engineer World’s Fair 2024 walkthrough, Cédric Vidal, Microsoft’s principal AI advocate, explores those capabilities through demonstrations. Azure AI Studio provides the common environment, bringing together AI services, machine learning and responsible AI tooling. Its model offerings extend beyond OpenAI to other providers of text, vision and speech capabilities.

Azure AI overview with a foundation-model banner above three columns for Azure AI Services, Azure Machine Learning, and Responsible AI Tooling, with Azure AI Studio below.Open full source frame
Azure AI services, machine learning, and responsible AI tooling within Azure AI Studio.

The interface and availability statements belong to that 2024 setting. Vidal introduces the Build announcements: Azure AI Studio’s general availability; GPT-4o, with text and vision available and speech described as forthcoming; Microsoft Research’s Phi-3 family; GPT-4 Turbo with Vision; DALL·E 3; and Whisper. The development features include the Assistants API, GPT-4 fine-tuning, batch inference and video translation. These establish the toolkit for the demonstrations, rather than a current availability guide.

0:160:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

From image descriptions to monitoring signals

A discussion with an energy-industry customer motivates the next example: a photograph of fallen utility poles and power lines. Asked what is happening, GPT-4o describes poles that have fallen or are leaning and identifies infrastructure damage. Vidal remarks on the quick response, but supplies no latency measurement. The useful change from the restaurant example is the question’s operational purpose: a description could help someone assess an electricity network.

Vidal then proposes continuous inspection of camera feeds across a network, potentially at national scale and at the edge. Asking whether electricity is working produces an assessment that service is highly unlikely in the pictured area. This is an inference from visible damage, not an electrical measurement. He suggests requesting JSON output instead of prose so software could consume the assessment in dashboards and monitoring workflows; the continuous monitoring system itself is not demonstrated.

The insurance example follows the same pattern with a different image. A house appears collapsed and severely tilted. The response describes that damage and offers hurricanes, earthquakes or landslides as possible causes. Visible condition and possible explanation remain distinct: the photograph supports damage assessment more directly than it establishes which event caused the damage.

8:378:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:37 · section reference included

Translating speech without flattening delivery

The next modality is speech. Vidal plays a video of himself transformed by Azure AI’s video translation service. The sequence moves through German and Spanish, then whispered Italian and loudly delivered Japanese. The demonstration is about more than changing the words: the translated speech retains his voice and follows the expressive delivery of the original.

Vidal highlights that whispering stays quiet and yelling stays loud. He also separates the service’s work from his own editing: he manually stitched the separately translated videos into the multilingual sequence, while attributing the voice and language transformations to the service.

An audience question asks about embedding models. Vidal says he will return to them, then moves into the model catalog; the walkthrough does not return to an explicit embeddings explanation.

11:3611:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:36 · section reference included

Choosing a model also means choosing how to run it

Opening the catalog and removing a filter makes the expanded model choice tangible. Vidal reports 1,600 models in the Azure AI catalog at recording time. That is the whole catalog, not the number available through serverless APIs. The deployment-options filter narrows the selection to models offered in that form.

Deployment optionWhat you pay forInfrastructure responsibility
Rented GPU infrastructureGPU capacity, including idle timeYou provision the capacity
Serverless APITokens consumedVendor manages infrastructure

The serverless option extends a familiar OpenAI-style consumption model to additional marketplace vendors. Vidal also points to GPU scarcity: avoiding the need to obtain and manage capacity can be as useful as avoiding idle charges.

More choice creates a selection problem. Azure AI Studio’s model benchmarks compare many—but not all—catalog models using accuracy and other metrics. These comparisons help narrow candidates for a use case; the next demonstration explores whether a smaller model can handle a task already shown with GPT-4o.

13:3113:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:31 · section reference included

Repeat the infrastructure question with Phi-3 Vision

Microsoft Research’s Phi-3 Vision 128k provides another text-and-image model within the Phi-3 family. Vidal uploads the same damaged-infrastructure photograph and asks the same question about whether electricity is working. Keeping the image and question fixed makes the difference in responses easier to inspect.

The smaller model gives a simpler explanation, but still identifies a likely outage or disruption. Vidal judges that answer useful for the task. This is a qualitative comparison of the two responses, with no comparative accuracy score: it shows a reason to evaluate the smaller model, rather than establishing that the models are interchangeable.

Zoomed chat response beneath “is electricity working here?” says downed utility lines and a disrupted power pole suggest an outage or disruption; the Phi-3 vision model identifier appears below.Open full source frame
Phi-3 Vision identifies signs of a possible power outage or disruption.
16:1816:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:18 · section reference included

Move inference into the browser

The next Phi-3 demonstration changes the execution location. Vidal describes a 3.8-billion-parameter Phi-3 model, quantized to four bits, with an approximately 2 GB browser download. Parameters and download size are different quantities; this is a separate small-model example from the preceding vision demonstration. After downloading, the model runs locally through WebGPU, which gives browser applications access to the device’s GPU.

The prompt asks what the model knows about the Rivian R2. Text is generated inside the browser on Vidal’s Apple-silicon laptop, which he tentatively identifies as a MacBook Pro M2. He points out the apparent speed, without measuring throughput.

Local execution changes the network dependency and where sensitive inputs can be processed. Vidal says the downloaded model could continue running with Wi-Fi disconnected, although he leaves the connection on during the demonstration. The practical attraction is inference that can happen on the user’s device rather than requiring every input to be sent to a hosted model.

17:4718:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:47 · section reference included

Give the model newer Rivian information

The Rivian question carries into the next demonstration. Asked for the different Rivian models, GPT-4 Turbo, described here as last updated in April 2023, lists the R1T, R1S and Amazon delivery truck. The answer illustrates a knowledge-freshness problem: the application needs information beyond what this deployment supplies unaided.

Vidal selects a prebuilt index containing Wikipedia pages about Rivian models. This enables retrieval-augmented generation, or RAG: documents become grounding material for the answer without retraining the model. The pages were uploaded before the session; the live operation is selecting the index and asking the same question again.

The query stalls. Vidal clears the conversation, reruns it and refreshes the page, with the index still selected. He then switches models, calling the replacement “Phi-3 Turbo,” and the request succeeds. He attributes the earlier failure to an apparent bug with the other model.

The grounded answer includes R1T, R1S and the newly announced R3, but omits R2. A follow-up asking specifically about R2 produces information about it. The sequence shows both the benefit and the remaining application problem: retrieval can make newer information available, while an initial answer may still leave out a relevant item.

19:5020:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:50 · section reference included

Check behavior after changing prompts or models

Switching models raises the next engineering question: does the application still behave as intended? Vidal opens Azure AI Studio’s Evaluation feature, which exposes coherence, groundedness and relevance. The same need arises after editing a system prompt. For RAG, groundedness is especially useful because it assesses whether the answer is supported by the supplied documents.

For the single evaluation question about Rivian models, Vidal reports a coherence score of 4 on a 1–5 scale. He also describes the answer as well grounded, without clearly supplying a separate groundedness score. With only one dataset entry, this demonstrates how to inspect an evaluation result, not broad coverage of application behavior.

22:4523:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:45 · section reference included

Ask questions of a kitesurfing GPS track

The agent demonstration begins with a personal file: a kitesurfing session, recorded on Vidal’s watch the preceding Sunday and exported as GPX. After uploading it to Code Interpreter, he asks how long he was on the water. The agent’s job is to turn that natural-language request into an operation over the file.

Code Interpreter generates Python and executes it in a sandbox. Here the input is XML-based GPX, rather than the CSV commonly used in data-analysis demonstrations. For the duration question, the central operation is extracting track-point timestamps and calculating the elapsed interval. A compact Python implementation of that operation is:

python

from datetime import datetime
from pathlib import Path
import xml.etree.ElementTree as ET


def track_duration_minutes(path: Path) -> float:
    root = ET.parse(path).getroot()
    timestamps = []

    for point in root.findall(".//{*}trkpt"):
        text = point.findtext("{*}time")
        if text:
            timestamps.append(
                datetime.fromisoformat(text.strip().replace("Z", "+00:00"))
            )

    if len(timestamps) < 2:
        raise ValueError("The track needs at least two timestamped points.")

    elapsed = max(timestamps) - min(timestamps)
    return elapsed.total_seconds() / 60


if __name__ == "__main__":
    minutes = track_duration_minutes(Path("session.gpx"))
    print(f"Recorded track span: {minutes:.1f} minutes")

This computes the recorded track’s elapsed span; whether that exactly matches time on the water depends on when recording began and ended. In the demonstration, Code Interpreter returns approximately 42 minutes, which Vidal says matches his recollection.

He then asks how many turns, or tacks, he made. This is a more interpretive operation than subtracting timestamps:

  1. Read the GPS coordinates along the track.
  2. Determine the direction of successive movements.
  3. Calculate angular changes along the path.
  4. Apply a threshold to classify changes as turns.

The agent reports 217 turns, without an independent count or a stated threshold. The threshold matters because the procedure must distinguish a meaningful maneuver from smaller changes in the recorded path.

Next, Vidal asks the agent to draw the session on a map. While it works, he emphasizes that he did not explain GPX’s structure or write the parsing code himself. The resulting map exposes the track visually. He then shows a prepared follow-up result with red crosses marking turns; that additional generation is skipped live. The progression moves from a scalar answer, to a derived count, to a plot that makes the analysis inspectable.

23:5624:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:56 · section reference included

From a repository request to an editable plan

The final demonstration moves from interpreting a data file to understanding a code repository. Vidal opens a preview he calls GitHub Workspaces; the interface is labeled Copilot Workspace. The repository contains Python code, and the request is to add a Java GUI frontend. Before proposing changes, the system examines what the repository currently contains and which requested capabilities are missing.

The workflow first writes a specification, separating the current state from the proposed state. From that specification, the user can request a plan. Corrections are possible at each step, so the feature request becomes an inspectable sequence of decisions rather than an opaque jump to a code change.

Copilot Workspace shows the request “Can you add a Java GUI frontend?”, completed Current and Proposed specification lists, and a Generate plan button.Open full source frame
Copilot Workspace specifies a proposed Java GUI frontend for a Python repository.

Vidal presents this as a preview, not a generally available product at the time. The demonstrated endpoint is specification and planning, not a completed Java frontend or a deployment. Where Code Interpreter executed a bounded analysis over an uploaded file, this last workflow makes the proposed repository changes available for correction before implementation.

27:2227:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

27:22 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…