AI Engineer World's Fair 2026
While my guitar gently speaks
Loading slides…
Read the talk
While My Guitar Gently Speaks
A guitar-controlled voice starts with triggering a speech clip, then grows through word segmentation, pitch detection, a vocoder, local conversation and precomputed singing samples.
From a talk by Todd Fisher
Before you start: No audio-programming background is required; the article introduces the signal-processing concepts used in the demonstrations.
Making the audience wonder how it works
At a Slipknot concert Todd Fisher attended in high school, the drummer’s platform began to rise during a solo. Then it tilted, leaving the drummer playing upside down as the audience cheered. Years later, at Stranger Things: The First Shadow on Broadway, Fisher watched a performer fall backward in what appeared to be slow motion. An effect familiar from post-production was happening on a live stage. Both experiences left him with the same fascination: technology could make a performance do something the audience did not expect.
That is the starting point for making a guitar speak. Fisher loves the instrument, and he has the familiar engineer’s backlog of ideas waiting for spare time. AI assistance makes some of those projects easier to pursue—easier, he qualifies, rather than effortlessly easy. The opportunity is to take a personal curiosity far enough that it becomes something people can experience.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A guitar already has a history of transformations
Fisher frames the project as another step in the guitar’s technological evolution. Pickups and amplification made it louder. Effects pedals let musicians chain transformations together to create distinctive sounds. The talk box, illustrated by Peter Frampton, took a particularly physical approach: route the guitar’s sound through a hose into the player’s mouth, then use mouth shapes to articulate that sound into something resembling words.
Software moved many sound transformations into digital audio workstations such as Pro Tools, Logic and Fruity Loops. Whether software removes the need for physical pedals is a matter of preference; for this project, the useful consequence is that another transformation can live in the same software environment. What could AI add to the instrument next?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
First, make notes spell a message
The practical setting was Halloween. Fisher began bringing his guitar and amplifier outside while handing out candy, turning it into a recurring performance. Later, dressing as Eddie Munson—the guitar player from Stranger Things—gave that performance a theme. After adding the music, he wanted something more of the show’s world in the experience.
He built an app to display the Stranger Things alphabet on his garage door. In the browser demonstration, he enables microphone access and plays notes to trigger the alphabet’s lights. A custom-message field lets him enter text such as Happy Halloween and spell it through the interaction. The guitar is already doing more than producing sound: it is driving a second medium.
Once typed messages could become illuminated letters, the next question followed naturally: how hard would it be to make the guitar speak them?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a guitar note into a speech trigger
The audio implementation starts with JUCE. A digital audio workstation, or DAW, plays a role analogous to an IDE for musicians and producers: it hosts the work and the plug-ins that process it. Fisher experiments with Piper and Apple’s built-in speech capabilities, then uses a deliberately small pipeline:
- Enter the text to be spoken.
- Convert that text into a speech audio clip.
- Play the clip when a guitar note triggers it.
In Logic Pro, Fisher inserts his plug-in into the effects chain like another audio effect. The first demonstration repeatedly plays the word “Developers,” becoming a rhythmic call for the audience to clap along. This establishes a working connection between the instrument and speech playback, but the unit being triggered is still an entire clip.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Speech does not arrive neatly divided into words
Repeating one word is entertaining, but a sentence needs finer control. Fisher’s next version slices speech so that guitar playing can advance through its fragments. The demonstration now says, “Look at me. I can speak.” That exposes a harder problem: deciding where one spoken word ends and the next begins.
The first heuristic is energy-gap segmentation: inspect the waveform for low-energy pauses and use them as candidate cut points. This works when a speaker leaves gaps, but connected speech can run words together without silence. The useful signal is low amplitude or energy, not a decibel reading near zero.
Fisher then adds a sonority-peak syllabifier, looking for vowel-related peaks as clues to syllables. The two heuristics supply different evidence:
| Method | Boundary clue | Limitation |
|---|---|---|
| Energy gaps | Pauses in the waveform | Adjacent words may have no pause |
| Sonority peaks | Vowel-related syllable cues | Syllable cues do not directly give word boundaries |
Combining them gives the slicer more information than silence alone. The slide’s sonority diagram, labeled with the letters in cats run fast, illustrates why the internal structure of speech matters when choosing cuts.
A longer demonstration thanks the audience for letting the guitar be there and jokes about getting out of its guitar case. It works well enough to show progress, but the automatic boundaries still need correction. Fisher adds draggable segment boundaries so he can repair the cuts manually. Automatic segmentation supplies a starting point; editable boundaries make the result usable for performance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the voice the guitar’s pitch
The guitar can now control when speech happens. To move toward singing, it also needs to control pitch. A musical note contains multiple frequencies; the target is the fundamental frequency associated with the perceived note. Fisher uses the YIN pitch algorithm to estimate that fundamental from the guitar signal.
The detected pitch drives a synthesized sawtooth note. ADSR—attack, decay, sustain and release—controls shape the note’s amplitude envelope. Fisher then combines that pitched sound with the voice clip through a vocoder. The synthesized note supplies the pitched sound, while the speech supplies its vocal shaping. This returns to the talk-box idea, replacing the physical mouth-and-hose arrangement with audio processing.
To demonstrate it musically, Fisher first plays a backing chord part. He avoids explicitly performing the song suggested by the talk’s title because the recording will be posted online. With the backing part in place, he adds the guitar-controlled voice. Several attempts encounter technical difficulties and produce fragmented words, although a phrase identifying itself as a guitar that can speak comes through.
The result has remaining bugs, and the mix itself is part of the experiment. A clarity control blends the synthesized note with the original AI-generated voice. That gives the performer a way to retain more of the spoken voice while introducing the pitched component; it does not, by itself, resolve the fragmented articulation heard in the demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask the guitar a question
Once generated text can reach the guitar, conversation becomes another input path. Fisher adds a microphone and Whisper to turn spoken input into text. A local language model produces a response, and that response feeds the existing guitar speech pipeline. He does not require a particular local LLM: its role is to generate the next conversational turn.
An audience suggestion supplies the question: “What is reality?” Fisher speaks it into the microphone, waits for the model, then plays the response through the guitar. The voice acknowledges the scale of the question and asks how the listener’s music helps them understand reality. The output is choppy, as Fisher points out, but the full interaction works: a spoken question becomes a generated reply delivered through the instrument.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with singing, then prepare it for playback
Conversation extends what the guitar can say, but Fisher still does not consider the result singing. His next approach changes the source material. Instead of starting with synthesized speech, he takes recordings of singers from VocalSet. He uses WORLD to process the recordings, including shifting their pitches so they can be mapped to guitar notes.
In Fisher’s implementation, preparing those samples is too computationally heavy to do live, so he performs the work ahead of time. The slide separates grain analysis from resynthesis at each target pitch. Precomputation moves the expensive transformation out of the performance path: the prepared audio is ready before the musician needs to play it. This is a constraint of his sample-preparation setup, not a claim that WORLD universally requires offline operation.
Because preparation takes time, Fisher begins with five vowel sounds. He demonstrates the prepared vocals on their own, then plays them over the backing chords. The sound is closer to singing, though he characterizes it as progress rather than an opera-quality result.
The performance mapping is now concrete: each guitar note or fret selects a corresponding sample whose pitch has already been prepared. The instrument controls playback from that bank rather than requiring a fresh pitch transformation for every note. Fisher wants to explore more AI-intensive approaches next, and invites ideas for making the guitar sing better.
That open question is part of the project’s appeal. A Halloween performance grew into an alphabet display, then speech playback, conversation and an experiment in singing. There is still work to do, but each version gives Fisher something he can play. His closing invitation is to choose an equally personal project and build it: AI assistance can reduce the development effort enough to bring an idea off the backlog and into the world.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
C++ framework for building audio applications and plug-ins, with examples and build instructions.
Original local neural text-to-speech project, with a pointer to its successor.
OpenAI's speech recognition implementation for converting recorded speech into text.
Professional singing recordings covering five vowels, varied vocal techniques, scales, arpeggios, and excerpts.
Speech analysis and synthesis software exposing fundamental frequency, spectral envelope, and aperiodicity parameters.
Further reading
Original research describing WORLD's speech synthesis architecture and evaluation for real-time applications.
Updates since the talk
Local speech synthesis engine with installation instructions, voices, and command-line and programming interfaces.
Full transcript
Loading transcript…