AI Engineer Summit 2023

Move Fast Break Nothing

Dedy Kredo· Chief Product Officer, CodiumAI13:01

Loading slides…

Read the talk

Move Fast, Break Nothing: Giving Generated Code a Critic

Dedy Kredo’s AutoScraper demonstration follows a separate code critic from behavior mapping and test generation through automated repair, a failed regression check, and PR review.

From a talk by Dedy Kredo

Before you start: Basic familiarity with Python, automated tests, and Git pull requests will help you follow the demonstration.

A critic for generated code

What checks whether generated code actually does what its developer intended? Dedy Kredo opens with generative adversarial networks, or GANs: two networks working together, one generating and the other criticizing its output. In his telling, transformers shifted attention toward generation and became the preferred architecture for many tasks. Code, however, needs more than a plausible next output. It needs scrutiny of the behaviors that output creates.

The proposed architecture is GAN-like at the system level, not an adversarially trained neural network. One component generates code; a separate code integrity component reviews it, analyzes its behavior, and searches for edge cases. The target is not simply code that looks correct, but code that works according to the developer’s intent.

Slide connecting Code Gen and Code Integrity with paired arrows beneath the heading Code that works as intended.Open full source frame
Code generation and code integrity form a feedback loop aimed at code that works as intended.

CodiumAI focuses on that critic. Kredo argues that behavior coverage is more useful than code coverage: the organizing question is which behaviors have been identified and tested, rather than only which code has executed. The product connects behavior discovery to test generation, code enhancement, and review. At the time of the demonstration, those capabilities appeared in JetBrains and VS Code extensions and a Git plugin; APIs for embedding them in agents were described as forthcoming.

0:180:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:18 · section reference included

Starting with an untested scraping class

The live demonstration begins in VS Code with the CodiumAI extension installed. Kredo reports around 200,000 installs across JetBrains and VS Code. The project is AutoScraper, a Python scraper that generates extraction rules from examples of desired information. Kredo describes the demo project as having more than 5,000 GitHub stars but no tests. That absence is the practical obstacle: changing the scraper leaves the developer without a suite to catch unintended behavior. This describes his demo checkout; the current repository contains tests.

Kredo invokes CodiumAI on what he describes as a 600-line class. The extension can also operate on an individual method, but starting at class scope lets it analyze the larger unit first. It maps behaviors and then streams generated tests into the interface. Alongside the tests, it produces an explanation of the class, including example usage, components, and methods. The initial suite spans happy paths and edge cases.

3:003:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:00 · section reference included

Choose behaviors, then run and repair tests

The behavior map is hierarchical. Happy paths and edge cases can be expanded into sub-behaviors, and the developer can choose which deserve additional tests. This makes the map a place to exercise judgment: generation proposes a set of behaviors, while the developer decides where to investigate more deeply.

VS Code shows AutoScraper Python code beside CodiumAI’s Behaviors Coverage panel, with Happy Path and Edge Cases sections and Go to test or Generate test controls.Open full source frame
AutoScraper behavior coverage lists happy paths and edge cases alongside the class source.

For a scraper, a small behavior-focused test can hold the page content fixed and state the expected extraction directly. For example, this Python test uses a local HTML string to express the happy path of learning a rule from a desired value:

python

from autoscraper import AutoScraper


def test_build_finds_requested_product_name():
    html = "<html><body><h2>Travel Mug</h2></body></html>"
    scraper = AutoScraper()

    results = scraper.build(
        html=html,
        wanted_list=["Travel Mug"],
    )

    assert "Travel Mug" in results

The useful unit of coverage here is the behavior expressed by the assertion. Kredo’s next action is to select additional edge cases in the map and request tests for them. During that expansion, he reports nine tests in the interface.

Next comes Run & Autofix. The demonstrated sequence is:

  1. Execute the generated tests; some pass immediately.
  2. Inspect a failing test’s execution result.
  3. Have the model analyze the failure and generate a repair.
  4. Run the repaired test again.

The demonstrated failing test passes on its second attempt. Kredo describes this as a reflection process, using the language of chain of thought. The observable mechanism is the execution feedback loop: a failure supplies information for a revision, which is then checked by another run.

Kredo then opens eight tests as a file and saves them into the project. The recording does not explain the change from nine tests during generation to eight at this point. Saving the suite turns the generated output into a project artifact that can be rerun when the implementation changes.

4:564:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:56 · section reference included

An enhancement still has to survive the tests

With a saved suite, Kredo moves to code enhancement. He selects AutoScraper’s build method because it contains much of the class’s central functionality, then opens CodiumAI’s code suggestions. These suggestions aim beyond linting: performance, security, best practices, and readability are all in scope.

He chooses a proposed performance improvement that replaces hashing through Python’s hashlib with BLAKE3. The distinction matters: hashlib is a module exposing multiple hash algorithms, whereas BLAKE3 is an algorithm. The original algorithm is not identified in the narration. Kredo prepares the suggested changes, applies them, saves the code, and returns to the test suite.

The rerun does not produce a clean result. One test fails after the change, and Kredo explicitly sets it aside to continue the demonstration. The cause is not established, so the failure cannot be attributed specifically to BLAKE3. Nor is a speedup measured. At this point, the workflow has produced an applied enhancement and an unresolved regression signal—not a validated refactor. Generating a change and validating it remain separate steps.

6:547:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:54 · section reference included

From a local change to PR review

The next stage is preparing the change for review. In the CodiumAI PR Assistant, Kredo invokes /commit to generate a commit message and then commits the changes. Only after committing does he invoke /review. The assistant is expected to summarize the PR, assign a score, and look for issues the developer may have missed. Its initial assessment identifies tests as the main theme and describes the change as well structured. These are the historical IDE assistant commands shown in the recording.

8:549:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:54 · section reference included

The review finds an API key

The review then flags a potential security vulnerability. Kredo invokes /improve to request a fix and recognizes the issue: an API key has been left in the code. This extends the critic’s scope beyond functional test results. A change can have useful tests and still contain something that should not be committed.

VS Code with a pull request assistant showing review feedback about a hardcoded API key beside Python source code.Open full source frame
The review flags a hardcoded API key as a potential security vulnerability.

The response is delayed, so Kredo retries; an interjection points to the room’s poor network connection. Once the result arrives, clicking the finding navigates to the key’s location. Kredo identifies a displayed fix, but the demonstration does not establish its implementation or show it being applied and validated. The security result is therefore a located issue and a proposed repair, not a completed remediation.

The demo ends with a connected workflow: map behaviors, generate and execute tests, assess changes, and review the resulting contribution. Kredo also points to the Git plugin as a way to bring this work into GitHub. The critic is intended to operate throughout development, including the transition from an individual developer’s editor to a shared review process.

10:0610:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:06 · section reference included

A personal closing

Kredo closes by turning from the software to the people building it. CodiumAI is based in Israel, and he says Hamas attacked while the team was on the plane to the event. He condemns the attackers in dehumanizing terms and describes civilians being killed in their homes and abducted into Gaza.

He then introduces a photograph sent by his co-founder and CEO, Itamar. According to Kredo, Itamar had left his eight-month-pregnant wife at home to enter military reserve duty. The photograph shows a laptop and a rifle behind it; Kredo identifies the chart on the screen as increasing CodiumAI usage. The juxtaposition closes the talk on the circumstances in which his colleagues were continuing their work. His final declaration is: “We will prevail.”

Slide photograph of an open laptop on rough ground, a rifle behind it, and a circular portrait inset.Open full source frame
A closing slide shows a laptop with a rising chart and a rifle behind it.
11:4511:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:45 · section reference included

Resources

From the talk

  • Python scraper that learns extraction rules from examples, with installation instructions and build, reuse, and persistence examples.

  • Official BLAKE3 implementations, usage examples, test vectors, and benchmark context.

Updates since the talk

  • Community-maintained open-source PR review agent with CLI and GitHub Actions setup, distinct from Qodo's primary offering.

Transcript download
Full transcript

Loading transcript…