AI Engineer World's Fair 2026

While my guitar gently speaks

Loading slides…

Read the talk

While My Guitar Gently Speaks

A guitar-controlled voice starts with triggering a speech clip, then grows through word segmentation, pitch detection, a vocoder, local conversation and precomputed singing samples.

From a talk by Todd Fisher

Before you start: No audio-programming background is required; the article introduces the signal-processing concepts used in the demonstrations.

Making the audience wonder how it works

At a Slipknot concert Todd Fisher attended in high school, the drummer’s platform began to rise during a solo. Then it tilted, leaving the drummer playing upside down as the audience cheered. Years later, at Stranger Things: The First Shadow on Broadway, Fisher watched a performer fall backward in what appeared to be slow motion. An effect familiar from post-production was happening on a live stage. Both experiences left him with the same fascination: technology could make a performance do something the audience did not expect.

That is the starting point for making a guitar speak. Fisher loves the instrument, and he has the familiar engineer’s backlog of ideas waiting for spare time. AI assistance makes some of those projects easier to pursue—easier, he qualifies, rather than effortlessly easy. The opportunity is to take a personal curiosity far enough that it becomes something people can experience.

Black slide reading “Live performances rock” and “Leverage technology in a creative way,” with a lightbulb icon and the guitarist inset.Open full source frame
“Live performances rock” expands into a call to use technology creatively.
0:360:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:36 · section reference included

A guitar already has a history of transformations

Fisher frames the project as another step in the guitar’s technological evolution. Pickups and amplification made it louder. Effects pedals let musicians chain transformations together to create distinctive sounds. The talk box, illustrated by Peter Frampton, took a particularly physical approach: route the guitar’s sound through a hose into the player’s mouth, then use mouth shapes to articulate that sound into something resembling words.

Software moved many sound transformations into digital audio workstations such as Pro Tools, Logic and Fruity Loops. Whether software removes the need for physical pedals is a matter of preference; for this project, the useful consequence is that another transformation can live in the same software environment. What could AI add to the instrument next?

2:472:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:47 · section reference included

First, make notes spell a message

The practical setting was Halloween. Fisher began bringing his guitar and amplifier outside while handing out candy, turning it into a recurring performance. Later, dressing as Eddie Munson—the guitar player from Stranger Things—gave that performance a theme. After adding the music, he wanted something more of the show’s world in the experience.

He built an app to display the Stranger Things alphabet on his garage door. In the browser demonstration, he enables microphone access and plays notes to trigger the alphabet’s lights. A custom-message field lets him enter text such as Happy Halloween and spell it through the interaction. The guitar is already doing more than producing sound: it is driving a second medium.

Browser demo with three rows of painted letters, colored string lights, a bright light near I, and controls at the upper right.Open full source frame
The alphabet-wall demo shows a brightly illuminated light above the letters.

Once typed messages could become illuminated letters, the next question followed naturally: how hard would it be to make the guitar speak them?

4:034:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:03 · section reference included

Turn a guitar note into a speech trigger

The audio implementation starts with JUCE. A digital audio workstation, or DAW, plays a role analogous to an IDE for musicians and producers: it hosts the work and the plug-ins that process it. Fisher experiments with Piper and Apple’s built-in speech capabilities, then uses a deliberately small pipeline:

  1. Enter the text to be spoken.
  2. Convert that text into a speech audio clip.
  3. Play the clip when a guitar note triggers it.

In Logic Pro, Fisher inserts his plug-in into the effects chain like another audio effect. The first demonstration repeatedly plays the word “Developers,” becoming a rhythmic call for the audience to clap along. This establishes a working connection between the instrument and speech playback, but the unit being triggered is still an entire clip.

5:516:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:51 · section reference included

Speech does not arrive neatly divided into words

Repeating one word is entertaining, but a sentence needs finer control. Fisher’s next version slices speech so that guitar playing can advance through its fragments. The demonstration now says, “Look at me. I can speak.” That exposes a harder problem: deciding where one spoken word ends and the next begins.

The first heuristic is energy-gap segmentation: inspect the waveform for low-energy pauses and use them as candidate cut points. This works when a speaker leaves gaps, but connected speech can run words together without silence. The useful signal is low amplitude or energy, not a decibel reading near zero.

Fisher then adds a sonority-peak syllabifier, looking for vowel-related peaks as clues to syllables. The two heuristics supply different evidence:

MethodBoundary clueLimitation
Energy gapsPauses in the waveformAdjacent words may have no pause
Sonority peaksVowel-related syllable cuesSyllable cues do not directly give word boundaries

Combining them gives the slicer more information than silence alone. The slide’s sonority diagram, labeled with the letters in cats run fast, illustrates why the internal structure of speech matters when choosing cuts.

Slide with two bullets about phoneme alignment and timing, above a sonority diagram labeled with the letters in “cats run fast.”Open full source frame
“Syllables are tricky!” illustrates phoneme alignment and sonority peaks.

A longer demonstration thanks the audience for letting the guitar be there and jokes about getting out of its guitar case. It works well enough to show progress, but the automatic boundaries still need correction. Fisher adds draggable segment boundaries so he can repair the cuts manually. Automatic segmentation supplies a starting point; editable boundaries make the result usable for performance.

7:187:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:18 · section reference included

Give the voice the guitar’s pitch

The guitar can now control when speech happens. To move toward singing, it also needs to control pitch. A musical note contains multiple frequencies; the target is the fundamental frequency associated with the perceived note. Fisher uses the YIN pitch algorithm to estimate that fundamental from the guitar signal.

The detected pitch drives a synthesized sawtooth note. ADSR—attack, decay, sustain and release—controls shape the note’s amplitude envelope. Fisher then combines that pitched sound with the voice clip through a vocoder. The synthesized note supplies the pitched sound, while the speech supplies its vocal shaping. This returns to the talk-box idea, replacing the physical mouth-and-hose arrangement with audio processing.

To demonstrate it musically, Fisher first plays a backing chord part. He avoids explicitly performing the song suggested by the talk’s title because the recording will be posted online. With the backing part in place, he adds the guitar-controlled voice. Several attempts encounter technical difficulties and produce fragmented words, although a phrase identifying itself as a guitar that can speak comes through.

The result has remaining bugs, and the mix itself is part of the experiment. A clarity control blends the synthesized note with the original AI-generated voice. That gives the performer a way to retain more of the spoken voice while introducing the pitched component; it does not, by itself, resolve the fragmented articulation heard in the demonstration.

9:369:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:36 · section reference included

Ask the guitar a question

Once generated text can reach the guitar, conversation becomes another input path. Fisher adds a microphone and Whisper to turn spoken input into text. A local language model produces a response, and that response feeds the existing guitar speech pipeline. He does not require a particular local LLM: its role is to generate the next conversational turn.

An audience suggestion supplies the question: “What is reality?” Fisher speaks it into the microphone, waits for the model, then plays the response through the guitar. The voice acknowledges the scale of the question and asks how the listener’s music helps them understand reality. The output is choppy, as Fisher points out, but the full interaction works: a spoken question becomes a generated reply delivered through the instrument.

13:5014:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:50 · section reference included

Start with singing, then prepare it for playback

Conversation extends what the guitar can say, but Fisher still does not consider the result singing. His next approach changes the source material. Instead of starting with synthesized speech, he takes recordings of singers from VocalSet. He uses WORLD to process the recordings, including shifting their pitches so they can be mapped to guitar notes.

In Fisher’s implementation, preparing those samples is too computationally heavy to do live, so he performs the work ahead of time. The slide separates grain analysis from resynthesis at each target pitch. Precomputation moves the expensive transformation out of the performance path: the prepared audio is ready before the musician needs to play it. This is a constraint of his sample-preparation setup, not a claim that WORLD universally requires offline operation.

Slide titled “This is a heavy process,” with side-by-side diagrams for analyzing a grain and resynthesizing it at each target pitch.Open full source frame
WORLD processing separates grain analysis from pitch-specific resynthesis performed ahead of playback.

Because preparation takes time, Fisher begins with five vowel sounds. He demonstrates the prepared vocals on their own, then plays them over the backing chords. The sound is closer to singing, though he characterizes it as progress rather than an opera-quality result.

The performance mapping is now concrete: each guitar note or fret selects a corresponding sample whose pitch has already been prepared. The instrument controls playback from that bank rather than requiring a fresh pitch transformation for every note. Fisher wants to explore more AI-intensive approaches next, and invites ideas for making the guitar sing better.

That open question is part of the project’s appeal. A Halloween performance grew into an alphabet display, then speech playback, conversation and an experiment in singing. There is still work to do, but each version gives Fisher something he can play. His closing invitation is to choose an equally personal project and build it: AI assistance can reduce the development effort enough to bring an idea off the backlog and into the world.

15:3015:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:30 · section reference included

Resources

From the talk

  • C++ framework for building audio applications and plug-ins, with examples and build instructions.

  • Original local neural text-to-speech project, with a pointer to its successor.

  • OpenAI's speech recognition implementation for converting recorded speech into text.

  • Professional singing recordings covering five vowels, varied vocal techniques, scales, arpeggios, and excerpts.

  • Speech analysis and synthesis software exposing fundamental frequency, spectral envelope, and aperiodicity parameters.

  • Original research describing WORLD's speech synthesis architecture and evaluation for real-time applications.

Updates since the talk

Transcript download
Full transcript

Loading transcript…