AI Engineer Summit 2023

Building AI For All

Amjad Masad· CEO, ReplitMichele Catasta· VP of AI, Replit25:13

Loading slides…

Read the talk

Building AI for All: The Small-Model Economics of Code Completion

Making AI assistance a default feature requires more than a capable model: Replit’s approach connects curated training data, compact architecture, fast inference, and responsive deployment.

From a talk by Amjad Masad and Michele Catasta

Before you start: Basic familiarity with code editors, model training, and inference will help; the article explains the evaluation and completion formats as they appear.

Making better programming tools available to everyone

How do you make the fastest path from an idea to deployed, scalable software available to everyone? For Replit co-founder Amjad Masad, that question begins with the interfaces used to create software. His opening slide shows a punch-card machine: an example of the physical work that preceded modern programming tools. The ENIAC history needs one distinction: original ENIAC programming used wiring and switches, while its later stored-instruction conversion ran in 1948. It was not originally a von Neumann machine programmed simply by punched cards. The broader starting point remains the difficulty of expressing instructions before accessible programming languages and editors.

Black-and-white photograph of a punch-card machine beside the text “We created software with punch cards...”Open full source frame
Software creation with punch cards.

Assembly, compilers, and higher-level languages such as C moved more of that work into text. JavaScript supplies Masad’s punchline: after its invention, everything went downhill. He estimates that text-based programming represented a 10× to 100× improvement over earlier methods. That is his historical framing for a possible new jump in productivity, rather than a measured comparison presented in the talk.

As software grew, IDEs brought more assistance into the editing loop. Masad recalls Replit adding Language Server Protocol, or LSP, support across its programming environments around 2017–2018, giving every account access to IntelliSense. The access came with an infrastructure bill: language analysis consumes CPU, and a TypeScript server can consume substantial RAM. Replit accepted that cost to put professional programming tools in more people’s hands.

0:160:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

From Ghostwriter to default AI assistance

Early experiments with GPT-2 suggested that a model could take a piece of code and continue it. Replit developed that possibility into Ghostwriter, combining autocomplete, chat, and other assistance inside the IDE. As coding assistants spread, they began improving the productivity of engineers already equipped to use them. Masad estimates gains of 50%, 80%, or perhaps 100% for some engineers at the time of the talk. He explicitly places that below an order-of-magnitude improvement; his forecast of 10×, 100×, or perhaps 1,000× over the following decade is an expectation, not an observed result.

For a company whose mission is to empower the next billion developers, a separately purchased assistant creates an access problem. The proposed change is therefore a product default: AI should be part of every programming interaction, rather than another named add-on a user must discover. Masad announces AI for All, extending basic AI assistance to Replit’s millions of users. The announcement concerns basic access; advanced features remain differentiated, and the new V1.5 model has a gradual rollout.

Masad describes the rollout as potentially the largest deployment of AI-enhanced coding in the world, then jokes that Replit will burn as much GPU as CPU. The reach is concrete: people coding on Android phones are included. Just as LSP made editor intelligence a shared infrastructure expense, default AI makes model serving part of the cost of providing the programming environment.

2:232:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:23 · section reference included

Putting inference inside the application

An assistant changes how a developer writes software. A model call inside an application changes what that software can do. Model Farm addresses this second layer by exposing models directly from the Replit IDE. Masad presents it as a way to begin inference with three lines of code, launching with Google Cloud LLMs and announcing Llama and Stable Diffusion as planned additions. He also invites other model providers to participate.

The on-screen example pairs code in the editor with generated console output about learning AI. Masad promises everyone free-tier access at least through the end of 2023, the announcement year. That is the historical Model Farm offer shown in the recording, not a statement of current API availability or pricing. Together, default assistance and accessible inference cover both the software-creation stack and the application’s call stack.

Replit editor and console showing a response about learning AI, beside an announcement of extended free ModelFarm access and upcoming models.Open full source frame
Replit ModelFarm code alongside generated console output.
5:145:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:14 · section reference included

A small model with an interactive latency budget

Michele Catasta, introduced as Replit’s head of AI, takes over with the engineering question behind that rollout: is it worthwhile to train a relatively small model on a limited GPU cluster? He characterizes a recent SemiAnalysis study as dismissive of that approach. Replit’s experience training its own models since May had suggested otherwise. Its bespoke completion model learns from open-source code on GitHub and code developed by Replit users, with a product requirement that rewards responsiveness.

Catasta states a P95 code-completion latency target below 250 milliseconds. At that speed, assistance can fit into the developer’s typing rhythm rather than becoming a separate interaction. He describes Replit’s preceding models as state of the art within their size class over the previous few months. The May release, Replit Code V1 3B, brought adoption and community contributions; that response, and Replit’s own dependence on open source, motivate the release of Replit Code V1.5 3B.

6:176:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:17 · section reference included

Spend more during training to serve a smaller model

V1 followed a Llama-style training recipe. For V1.5, Catasta reports training at roughly 300 tokens per parameter. He connects this choice to DeepMind’s Chinchilla paper, Training Compute-Optimal Large Language Models, which showed how a fixed training-compute budget should be divided between model size and training data. The deployment implication goes a step further: spending more compute training a smaller model can yield a useful model that is cheaper and faster to run repeatedly. Training cost can be amortized across inference.

Catasta reports doubling the amount of high-quality data relative to V1 and training V1.5 on approximately one trillion tokens: roughly 200 billion corpus tokens across five epochs, with a linear cooldown at the end. An epoch is one pass through the corpus, so repeated exposure must be distinguished from unique data. The mixture covers 30 programming languages and adds developer-oriented Stack Exchange posts, including coding and software-engineering questions.

8:589:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:58 · section reference included

Curate the corpus, then reuse it

The code corpus begins with The Stack, BigCode’s collection built from GitHub repositories. Catasta describes a pipeline that selects repositories, cleans their contents, and retains permissively licensed code, including MIT, BSD, and Apache 2.0 material. Replit selects 30 leading languages from that mixture and invests further effort in filtering. This matters especially because the released models are trained from scratch: their learning begins with the first token Replit supplies, rather than with an existing pretrained model.

Inspired by the quality pipelines for Codex and PaLM, Replit adds heuristics to remove several kinds of undesirable training material:

  • Autogenerated code: output that does not reflect the code a developer would ordinarily write.
  • Minified code: compressed source that is a poor target for readable editor suggestions.
  • Non-parseable code: source that fails the relevant syntax checks.
  • Toxic content: material excluded by content filtering.

The pipeline runs on Spark. Catasta’s practical invitation is that much of the infrastructure needed to prepare data, train a model, and serve it is now available as open-source components. He contrasts that ecosystem with the much harder task of assembling the same system in 2022.

A curated corpus creates a new constraint: what happens when the model needs more training exposure than the available high-quality data supplies? Scaling Data-Constrained Language Models, published shortly after V1, supports the intuition Replit had already used—repeat a good subset instead of expanding it with worse data. Catasta summarizes the plots as showing comparable loss after four or five repetitions; the paper’s stated result is negligible loss differences through four epochs in its fixed-compute, data-constrained experiments, with diminishing returns afterward.

Repetition therefore serves two goals. It lets Replit spend more training compute without relaxing its quality filters, and it reduces pressure to broaden the corpus beyond permissively licensed material. Catasta announces V1.5 as an open-source release with a commercially permissive license. The data decision and the distribution decision are connected: the team wants other developers to be able to build on the model.

10:3610:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:36 · section reference included

Pack more code into a compact architecture

Despite the 3B name, Catasta describes V1.5 as a 3.3-billion-parameter model. Its context expands from V1’s 2K tokens to 4K, and it uses a newly trained, code-specific 32K vocabulary. Vocabulary design changes how much source code fits in a fixed token budget: if common code fragments require fewer tokens, the model can see more source within the same context and training sequence. Catasta reports a few percent of additional compression and says the vocabulary compresses code better than those used by StarCoder and Code Llama; he does not present the tokenizer comparison’s measurement details.

Catasta reports training on 128 H100 GPUs with 80 GB of memory each for one week on MosaicML. To the team’s knowledge, it was the first officially announced model trained on H100s and released open source. The architecture also incorporates FlashAttention, grouped-query attention, and ALiBi positional embeddings, along with newer optimizers that he does not name. These choices support the broader objective: retain useful coding capability while making inference fast enough for an editor.

14:0514:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:05 · section reference included

Evaluate generated programs by running them

HumanEval Pass@1 evaluates whether a generated solution passes a task’s tests. Catasta describes the basic procedure:

  1. Give the model an English description of a programming task.
  2. Generate a self-contained Python solution.
  3. Execute the solution in a test harness.
  4. Check whether its outputs match the expected results.

The important distinction is that the output is tested as a program, rather than judged only as plausible-looking text. BigCode’s multilingual evaluation harness extends this approach beyond Python, and its leaderboard runs evaluations across models from different organizations. Catasta expects Replit’s entry to appear within days.

Catasta reports that the V1.5 base model beats StarCoder 3B across all six displayed languages and comes close to Code Llama 7B. The table covers Python, Java, JavaScript, C++, Rust, and PHP. His more consequential comparison is with the larger model: Code Llama 7B builds on a Llama 2 model trained on two trillion natural-language tokens, followed by another 500 billion code tokens. On the accounting presented here, that means roughly twice the parameters and 2.5× the training-token exposure of Replit’s base model, with substantially more GPU compute. The comparison tests whether intensive training of a small code model can approach a larger model’s task performance.

Evaluation table comparing StarCoder 3B, replit-code-v1.5-3b, and CodeLlama 7B on Python, Java, JavaScript, C++, Rust, and PHP.Open full source frame
Pass@1 results across six programming languages.
15:2715:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:27 · section reference included

Adapt the model to the code users actually write

The next model is a repl-tuned variant, produced through further pretraining on public Replit projects. Catasta explains that public repls were automatically published under MIT at the time, making that code available for this training stage. The team reuses its filtering pipeline to extract higher-quality code. He reports 200 billion additional training tokens, but also describes a 30-billion-token filtered corpus, three epochs, and a linear cooldown; those quantities do not fully specify a consistent training schedule.

The mixture shifts toward the languages Replit users favor. Catasta estimates that 95% of users mostly write Python and JavaScript. Recency matters alongside language choice: the adaptation’s data cutoff is only a few weeks before the talk, which he says helps the model generate code for recently popular libraries. The team intends to keep refreshing the models so completions follow the ecosystem developers are actually using.

In the next comparison, the base model remains on the left, Code Llama sits in the middle, and the repl-tuned model is added on the right. StarCoder is omitted because the base model already surpassed it in the preceding table. Catasta reports that the repl-tuned model substantially outperforms Code Llama on nearly every displayed language and reaches 36% HumanEval Pass@1.

That result recalls Catasta’s earlier work on PaLM-Coder. The PaLM paper reports 36.0% zero-shot HumanEval Pass@1 with greedy decoding for both PaLM-Coder and Davinci Codex, identified as code-davinci-001. PaLM-Coder had 540 billion parameters—not tokens—making it about 164 times the size of the 3.3B Replit model; Catasta’s almost-200× description is loose rounding. Matching this historical score illustrates progress at a much smaller model size, but the talk does not establish identical decoding settings for Replit or equivalent general coding ability.

17:3017:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:30 · section reference included

Optimize both generation and capacity startup

The product objective is inference performance, not a benchmark position by itself. Catasta treats benchmark scores and qualitative checks as useful confirmation that the small model remains capable. He reports generation above 200 tokens per second on a single model with no batching. The talk does not specify serving hardware, precision, sequence lengths, or measurement boundaries, so the training cluster’s H100 specifications should not be treated as the conditions of this serving result.

The speed work spans the model and its runtime:

LayerChoicePurpose
VocabularySmaller, code-specific vocabularySupport efficient code representation and generation
Attention implementationFlashAttention with a Triton kernelAccelerate attention computation
Attention architectureGrouped-query attentionImprove inference efficiency
Serving stackTriton Inference Server and TensorRT-LLMOptimize execution on NVIDIA GPUs

The Triton kernel and Triton Inference Server occupy different layers: one implements computation, while the other serves inference requests. Catasta presents these choices as coordinated work to squeeze more speed from the model.

Fast generation is only one operational timescale. A traffic spike also requires new serving capacity, and the original Kubernetes deployment path included pod setup, dependency downloads, and build work. Catasta reports that Bradley reduced the interval from initiating deployment to a deployed model from 18 minutes to two minutes. He does not disclose the individual optimizations. The shorter interval lets the service respond much sooner when demand increases.

That deployment work connects directly to the opening announcement. Catasta says the team flipped the switch for code completion when Masad announced AI for All earlier in the talk. The model’s size, serving speed, and capacity startup time are therefore parts of the same access decision: assistance must be affordable to provide and responsive under the load created by making it a default feature.

19:5220:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:52 · section reference included

Let the community extend the model

Why release the model after investing in its training? V1 had already returned more than goodwill. Community members produced fine-tunes and instruction-tuned versions, ran the small model locally with GGML on Apple silicon, and built custom prompts for Copilot-like tools. These are deployment paths and adaptations beyond Replit’s own editor. Catasta expects similar work around V1.5, which he says is already available on Hugging Face during the talk. The README is still being completed, and he directs people with production-running questions to Madhava at the booth.

The next planned extension is an instruction-fine-tuned version developed with Glaive AI, using synthetic coding instructions. Catasta gives the dataset size as 210,000 coding instructions. The team has encouraging preliminary results but is still checking them before release; this is forthcoming work, not another completed benchmark announcement.

22:1122:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:11 · section reference included

Give completion the surrounding syntax

Replit is also working with Morph Labs as a design partner on FIST, or Fill In the Syntax Tree. Catasta introduces it through Fill In the Middle: when a developer edits inside a file, the model receives both the prefix above the cursor and the suffix below it. The requested output is the missing code between those two pieces, rather than a continuation of the prefix alone.

For a concrete Python example, the following record keeps the existing prefix and suffix separate from a proposed insertion:

python

prefix = "def total_price(prices):\n"
suffix = "\n\nassert total_price([2, 3]) == 5\n"
proposed_middle = "    return sum(prices)"

proposed_source = prefix + proposed_middle + suffix
print(proposed_source)

The suffix supplies a constraint on the missing function body, while the prefix establishes where it belongs. This example assembles a candidate; it does not run the assertion. FIST adds awareness of the underlying abstract syntax tree, so the completion format can use the source’s syntactic structure as well as surrounding text. Catasta reports promising early results and points to Jesse’s later session for details, with release expected in days or weeks.

Slide titled “Collaboration: Morph Labs” describing FIST as a syntax-aware extension of FIM and reporting preliminary improvements in code generation.Open full source frame
Morph Labs collaboration on fill-in-the-syntax-tree completion.

The closing invitation returns to trying the model directly. Catasta says Replit V1.5 is expected to appear on Perplexity AI Labs by that evening, offering fast hosted interaction for a qualitative check. Alongside the released weights, community adaptations, and planned instruction and syntax-aware work, that gives developers several ways to explore what a small, intensively trained code model can do.

23:5024:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:50 · section reference included

Resources

From the talk

  • The launch announcement explaining default AI assistance, free and Pro access, and the retirement of Ghostwriter branding.

  • Replit's overview of the model's training data, architecture, evaluation, and code-completion usage.

Transcript download
Full transcript

Loading transcript…