AI Engineer World's Fair 2024

Covalent Launch: The GPU Cheatcode: Fine-tune 20 Llama Models in 5 Minutes

Santosh Radha· Head of Product/Research, Agnostiq (Covalent)6:26

Loading slides…

Read the talk

Covalent Launch: The GPU Cheatcode: Fine-tune 20 Llama Models in 5 Minutes

Covalent’s launch demo follows locally written Python through remote GPU execution, model evaluation, CPU-based selection, and deployment behind a text-generation endpoint.

From a talk by Santosh Radha

Before you start: Familiarity with Python functions, decorators, and the distinction between model training and inference will help you follow the workflow.

Send locally written Python to remote compute

How do you fine-tune a collection of models from Python without first managing Docker containers or a Kubernetes cluster? Santosh Radha opens with that problem. The launch title names twenty Llama models, but he calls twenty arbitrary and describes an ambition to support hundreds. The five minutes refer to his presentation budget; the demonstration does not establish a timed twenty-model training result. He points viewers to a QR code for the accompanying code and examples, then introduces Covalent as an open-source/open-core product that sends locally written Python to a chosen compute backend.

The starting point is a function in a local Jupyter notebook. Add a decorator specifying the resources it needs, then press Shift+Enter. Radha’s first illustrative request selects an H100, 36 GB of memory, and a maximum runtime of two days. The function executes on the remote GPU backend, and its result comes back to the notebook. The memory quantities here and later are stated resource requests; the talk does not establish that they describe GPU VRAM.

The execution destination depends on how Covalent is used:

ModeWhere the code runs
Open sourceYour own attached compute
Hosted cloudCovalent’s GPU cluster
Bring your own infrastructureYour cloud or on-premises systems, orchestrated through Covalent

Radha describes hosted billing in terms of GPU execution time: a five-minute H100 run incurs five minutes of usage, while a ten-second run incurs ten seconds. The local programming interface stays the same idea across these arrangements: specify compute, send the function, receive its result.

Diagram connecting laptop Python code through Covalent to a compute cluster, with open-source and cloud options.Open full source frame
Develop locally, run on GPUs remotely with Covalent.
0:130:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:13 · section reference included

Function jobs and inference services

A job is a single function with an execution destination. Its decorator describes the compute requirement; the backend runs the function and returns a Python object. Radha describes the charge as belonging to that function’s execution. This makes the function boundary both a programming boundary and a resource-allocation boundary.

An inference service adds a different lifecycle. Define an initializer, define a Python handler for an endpoint such as /generate, and call cc.deploy from the notebook. Instead of only receiving a function result, the developer receives an API endpoint. In the launch presentation, Radha describes that endpoint as supporting scale-to-zero and scaling as requests arrive. These are the service capabilities described in the recording, rather than guarantees established here for a current SDK version.

Radha then gives three examples of custom scaling policies:

  • Scheduled capacity: increase to ten GPUs at nine o’clock each day.
  • Utilization trigger: scale when GPU utilization reaches 80%.
  • Request trigger: scale when the request count reaches 1,000.

He also describes configurable authentication. The developer supplies Python functions and service policy; the platform handles the infrastructure work without requiring the developer to write Docker or Kubernetes configuration.

Run Inference slide with Python service and endpoint definitions above a curl request.Open full source frame
Python code for an inference service and a sample request.
1:532:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:53 · section reference included

A broader set of Python workloads

Before moving into the fine-tuning code, Radha points to the example library. His examples range from real-time time-series analysis and inverted transformers for forecasting—which he describes as state of the art—to large language model serving and an entire AI model foundry built with Python. Fine-tuning is the small demonstration of a broader capability: coordinating compute for several kinds of workloads through the same language used to implement them.

Tutorial cards beside a QR code labeled Examples and Documentation.Open full source frame
Covalent’s example library spans multi-agent workflows and LLM benchmarking.
3:073:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:07 · section reference included

Give each task the resources it needs

The code demonstration begins with ordinary training and evaluation functions in a local Python package. A training task wraps the fine-tuning function: it accepts a model and data, then returns a fine-tuned model. Radha assigns that task 24 CPU cores, one H100 GPU, 48 GB of memory, and an eighteen-hour maximum runtime. The resource specification surrounds the training function rather than changing its model-and-data interface.

Once training finishes, an evaluation task accepts the trained model and measures its accuracy. The workflow then ranks the candidates and picks the best-performing model for deployment. The selection task runs on a CPU machine: there is no reason to reserve a GPU merely to sort model scores. Resource placement follows the work each function performs.

3:514:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:51 · section reference included

Connect training to deployment

The selected model becomes an inference service. An initialization decorator marks the service setup, and a /generate endpoint generates text and returns a prediction. This completes the deployment target before the demonstration connects the individual tasks into a workflow.

The workflow itself uses a familiar Python structure: loop over candidate models, fine-tune each one, evaluate the result, and collect model–accuracy pairs. Sort those pairs, then deploy the winner. The orchestration logic can be expressed with the task functions passed explicitly:

python

def train_select_deploy(
    models, data, fine_tune_task, evaluate_task,
    select_task, deploy_task,
):
    scored_models = []
    for model in models:
        trained_model = fine_tune_task(model, data)
        accuracy = evaluate_task(trained_model)
        scored_models.append((trained_model, accuracy))

    best_model = select_task(scored_models)
    return deploy_task(best_model)


def select_best(scored_models):
    ranked = sorted(
        scored_models,
        key=lambda item: item[1],
        reverse=True,
    )
    return ranked[0][0]

Here, select_task represents the CPU task wrapping select_best; the other task arguments represent the training, evaluation, and deployment boundaries. The collected pairs preserve the association between each trained model and its score, so selection returns the model that should be deployed.

A single-line dispatch submits the workflow to Covalent and creates a job in its application. Each function runs on the device assigned to it. The frontend then lets Radha inspect an individual evaluation step and the machine allocated to that step, connecting the Python task boundary to an actual execution record.

4:515:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:51 · section reference included

Inspect the runs behind the workflow

Radha closes by reporting two runtime and cost examples from the application:

RunReported deviceReported runtimeReported cost
EvaluationL14Six minutes$0.87
Another model run; task unspecifiedV100Six minutes$0.11

The first device name is transcribed as L14; its accelerator identity remains unclear. The second run’s task is not specified, and the presentation does not supply the model sizes, datasets, accuracy definition, or billing conditions needed to compare the two costs. These are reported individual runs, not a controlled accelerator comparison.

The completed path is nevertheless concrete: locally written Python defines training, evaluation, selection, and deployment; each function receives its own compute assignment; and the selected model becomes a service. Radha ends with that training-to-deployment workflow completed without the developer managing Docker or Kubernetes.

5:496:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:49 · section reference included

Resources

From the talk

  • Open-source Python workflow orchestration with executor plugins for cloud and on-premises compute.

Transcript download
Full transcript

Loading transcript…