AI Engineer World's Fair 2024
AI Music Generation: From Prompt to Production
Loading slides…
Read the talk
AI Music Generation: From Prompt to Production
Follow Phlo Young from voice conversion and text-generated songs to personalized prompts, lyric preparation, and hands-on control of separated vocals and instruments.
From a talk by Phlo Young
Start with the listener’s taste
Have you used a music-generation model—and what would you want it to make? Phlo Young opens with those questions. About half the room has tried music generation. Boomy draws little recognition; Young remembers it as an earlier, less complex tool. Suno and Udio, the workshop’s main tools, are more familiar. Asked for a favorite artist, one listener names Miles Davis, and the room supplies the genre: jazz. That exchange establishes the practical problem: translate someone’s musical taste into something they can hear.
Young approaches the task as a self-taught songwriter, not a classically trained musician. He wrote his first song when young and welcomes corrections. QR-linked notes, live questions, and a short poll let participants steer the workshop toward their interests.
His experimental approach predates generative music. After learning Python for automation during the pandemic, he also began testing music distribution: release the same recording under different artist names, vary the cover art, marketing, and audience targeting, and observe how it performs. Young reports that one instrumental pen name accumulated about five million streams over the preceding couple of months. His own background spans R&B and hip-hop, with EDM and country-influenced experiments; generation adds another way to explore those musical directions. Distribution and artist income motivate him, although the workshop does not develop a release or monetization procedure.
The poll favors prompting and tweaking. A more advanced option—download generated music, separate stems, then mix and master—also attracts interest. Young points to Suno and Udio’s Discord communities, including an unnamed contributor whose unusual banjo and bagpipe prompts produce striking results. The useful premise is that effective musical descriptions may be less intuitive than simply naming an instrument.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate three meanings of AI music
The input tells you which part of music-making the model is doing. Young found “AI music” frustratingly ambiguous because it could describe a generated song, an expanded recording, or a familiar voice applied to someone else’s performance.
| Approach | Input | Transformation |
|---|---|---|
| Text to music | Description, optionally lyrics | Generate a sample or song |
| Audio to music | Existing sound | Transform or develop it into music |
| Voice conversion | Recorded voice | Change its vocal identity |
Young uses style transfer for the third category. In the examples that first attracted his attention, consumers often called voice conversion “AI music” without distinguishing it from generating the composition itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A human performance, a different vocal identity
The first video makes the distinction audible. Its creator, whom Young tentatively identifies as Roberto Nickson, begins with a Kanye-style voice and a converted excerpt of Day ’N Nite. He then finds a Kanye-style beat on YouTube, writes eight bars, and records them himself. The workstation view shows the human performance entering an ordinary audio-production workflow.
Next comes the conversion: the same recorded performance plays back with a Kanye-like vocal identity. The beat, writing, and reference performance were already supplied by a person. Voice conversion changes the voice without replacing the entire songwriting process. Seeing this sequence pushed Young into Discord servers and subreddits to learn how to reproduce it.
Ghostwriter’s viral Drake-and-The-Weeknd-style song illustrates the same distinction. In Young’s account, Ghostwriter wrote and produced the song and converted his own vocals; the workshop plays an excerpt rather than demonstrating its creation. Young recalls finding it below 1,000 views and estimates nine million streams in its first 24 hours. Those are his recollections, not verified platform totals. His attribution of the removal to the RIAA also differs from contemporaneous reporting, which identifies UMG in the YouTube copyright notice. The mechanism remains clear even where the viral history is uncertain: an authored recording became associated with recognizable voices through conversion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Voice conversion in the artist’s hands
Listening again to the earlier Kanye example, Young now hears conspicuous artifacts. What initially astonished him sounds less natural after subsequent improvements in voice models. He credits experimentation within communities that included teenage model trainers; his description of the tools becoming “10 times better” expresses that listening impression, not a measured quality improvement.
The Randy Travis example changes the purpose of the technique. Travis had lost his voice, and his team used conversion to make a new recording available in his vocal identity. Young plays an excerpt and describes unusually positive listener reactions, including a comment from an older fan moved to tears. Against the hostility he often encounters around AI music, this artist-directed use offers a different relationship between performer and technology: the recognizable voice helps the artist continue making music.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the prompt supplies the starting material
Text-to-music moves the starting point upstream. Young turns from converted recordings to full generated songs, beginning with BBL Drizzy in the context of the Drake–Kendrick dispute. The distinction matters more than the meme’s backstory: this example starts with words rather than a recorded singer. The taxonomy slide separates that route from transforming existing audio or converting a voice.
Young describes comedian King Willonius entering a prompt into Udio and, as he recalls less confidently, supplying lyrics. In his account, there was no studio vocal performance to replace: text went in and a song came out. He characterizes it as a one-shot generation and emphasizes its subsequent reuse in other songs. That is the workshop’s clearest contrast with Ghostwriter’s human-authored reference performance. Copyright questions arise here but remain unresolved.
A second example comes from ElevenLabs: a text-generated song about programming and GPUs. Its lyrics describe writing code and hoping to make it work, making the result a self-referential example of an AI system singing about AI development. Young reports that it circulated widely on Twitter and groups it with the meme-song examples.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beatboxing and desk percussion become arrangements
Audio-to-music keeps a performed sound at the beginning of the process. A Stable Audio example starts with beatboxing and transforms that input into music. Young remembers it as the first publicly available audio-input feature of this kind he encountered, while distinguishing a musical output from a necessarily complete song. A second clip, from the Suno team, starts with fingers drumming on a desk and produces rock music with guitar and other instruments. The attraction for a musician is immediate: a rhythm can be performed directly instead of translated entirely into descriptive text.
An audience member hears a Tom Petty resemblance in the preceding generation. That observation arrives alongside Young’s discussion of complaints alleging that Suno and Udio trained on copyrighted music. This is a historical workshop, held at the AI Engineer World’s Fair on June 26, 2024; the interface behavior, account options, and service integrations that follow belong to that demonstration. Young has no affiliation with the featured services. He points technically curious participants toward music-model papers in his notes, but follows the poll’s preference for practical work rather than explaining model architectures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give everyone a way to generate
To let participants work alongside him, Young distributes business-card codes. Entering a code through the green Gateway button reveals credentials for workshop accounts. The demonstrated login choices include Google, Discord, Twitter, and Apple for Udio, and Discord, Google, and Microsoft for Suno. He has paid for accounts to expand the room’s generation allowance, although his “Premier” and “unlimited” terminology should not be read as Udio’s documented plan names or limits: its original subscription announcement describes finite-credit Standard and Pro plans.
Apple’s two-factor authentication may obstruct some logins, but participants confirm that they can get in. The participation loop is straightforward: generate a song, copy its share link, and submit it through the workshop’s Google Form. Young proposes randomly selecting two participants to keep one account each. He has purchased a month of access rather than annual subscriptions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build Patrick’s birthday song
Patrick has a birthday that week, so the first live task becomes a personalized birthday song. Young initially includes a Randy Travis reference and anticipates artist-name filtering. An audience member suggests changing the spelling and adding the associated musical style; another supplies a more useful personal detail: Patrick is a patent attorney. Patrick names The Band as a favorite, and the room settles on rockabilly as a prompt description.
Young starts generation and carries the same prompt across services. The artist reference becomes a concrete failure case: a rejection names Randy Travis, after which he tries the altered spelling Tandy Ravis. This does not establish a reliable way around moderation. The useful prompt ingredients are already independent of that attempt: the occasion, the person, his occupation, and a musical direction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A songwriting partner and an in-house producer
Young thinks of Udio as a songwriting partner. He describes its default workflow as generating roughly 30 seconds of music at a time, with an experimental model available in the paid account shown. That short result becomes an object to revise rather than an answer to accept wholesale.
| Action | Purpose |
|---|---|
| Extend | Continue a promising idea |
| Remix | Explore a different version |
| Discard | Abandon an unhelpful result |
| Inpaint | Modify a selected section |
| Upload audio | Start from an existing recording |
For someone with original music, Young recommends bringing the recording, tempo, key, and lyrics into the process. He has tried this with his own songs and found the reinterpretations surprisingly compelling.
Suno, by contrast, feels to him like an in-house music producer: its output often has a polished, Top 40 character. His suggestion that this might come from a hidden system prompt or the training data is speculation, not knowledge of its internals. While onboarding friends and family, he noticed a more actionable issue in custom mode: lyric lines with poorly controlled syllable counts could fall badly out of alignment with the beat. He reports that his more recent generations handled timing and song structure better. The comparison is about his working experience—iterative development on one side, production polish on the other—not a controlled quality ranking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate an event song and listen to the birthday results
The next prompt is a song about the AI Engineer World’s Fair. Organizer Benjamin Dunphy’s taste suggests old-school hip-hop and boom bap. The demonstrated procedure is short:
- Sign in to Udio and open
Create. - Enter the song description and musical direction.
- Choose generated lyrics, supply custom lyrics, or request an instrumental.
- Choose an available model and start generation.
The free-account default shown is Udio 32; the longer experimental option belongs to the paid experience in the demonstration. A compact prompt expressing the spoken request is:
A song about the AI Engineer World's Fair.
Old-school hip-hop, boom bap.
Young then uses the same prompt in the other service. Before starting, he refreshes the page because he has encountered a state problem: repeated generation could unexpectedly extend a previous song instead of starting a new one.
Returning to Patrick’s songs, Young first encounters another moderation error associated with the Randy Travis reference. Other results do play: the vocals name Patrick, celebrate his birthday, and refer to rock and roll. Patrick calls one result the “Theme song to my sitcom.” On Suno, clicking the song name exposes its lyrics. A later alternate playback seems to repeat the same song, and Young leaves open whether the service generated it twice. The session produces enjoyable personalized music, but neither the birthday playback nor the event-song experiment becomes a systematic comparison of the services.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prepare and edit lyrics before generating audio
With time running short, Young turns to the workshop notes, where he also plans to publish the recording and transcript. A GPT-4 prompt template takes a Song Description and asks for structured lyrics. The displayed notes include style tags, an intro, verses, a chorus, an instrumental break, and an outro. Young flags a GitHub rendering problem around brackets, but the workflow is still clear: have ChatGPT draft the lyrics, edit them yourself, then give the music model the revised text. Separating lyric preparation from audio generation creates an explicit revision step before the musical performance is generated.
For Patrick’s example, that preparation step can be expressed as a reusable instruction:
Song Description:
A rockabilly birthday song for Patrick, a patent attorney
who loves rock and roll.
Draft lyrics with an intro, verses, chorus,
instrumental break, and outro.
Keep the lines singable and the chorus easy to repeat.
Return the lyrics so I can edit them before music generation.
The song description supplies the subject and direction; the separate lyric draft gives the user a chance to change phrasing before spending another generation on audio. Young also points to official Suno and Udio tips and returns briefly to the community “Prompt Master,” without completing that deeper walkthrough.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generic voices, emotional delivery, and descriptive labels
Philip asks whether conversion can produce a generic rapper’s voice instead of a recognizable performer such as Eminem. Young proposes two approaches:
- Blend voice sources: combine material from several speakers rather than targeting one identity.
- Use a synthetic source: obtain a TTS voice and use that output as training material.
Young suggests that some unspecified tools can begin with six seconds of voice, illustrating a blend with equal short samples from different people. He names no model, training conditions, or quality criterion for that minimum. Neither blending nor synthetic input by itself establishes permission to use the source material or clears rights in the result.
A responsible-use question gets a non-lawyer disclaimer rather than substantive ethical guidance. Another question asks about preserving emotional tone, citing the changing delivery in Kendrick Lamar’s Euphoria. Young recommends RVC as his preferred voice-conversion tool at the time. He emphasizes that cloud execution is an alternative to running it locally, where hardware requirements may be an obstacle. He also confirms that participants can have the slides.
After an audience member reports that the shared Google document requires an access request, Young acknowledges the permissions mistake. His final concrete prompting technique is to search Rate Your Music by artist, genre, or desired sound, collect the descriptive labels attached to relevant music, and use those labels in Suno or Udio prompts. He tentatively extends the suggestion to Stable Audio. The practical move is to replace a vague taste reference with musical vocabulary. His further claim that the models were trained on Rate Your Music labels comes from unnamed reverse engineering and is not established training-data provenance.
Young acknowledges the distinction between ethics and law but defers that discussion for lack of time. He reports twelve song submissions and proposes a later drawing, possibly streamed, followed by changing the shared credentials before handing accounts to the winners. That handoff remains a plan. The apparent closing then plays the World’s Fair song, whose lyrics include copper wires, coders, blueprints, and robots. Young reacts positively and wonders what Udio produced, without completing the comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the song into parts you can control
A late question brings the workshop back to production: how do you pull out stems and edit them? Young returns to Patrick’s birthday song and demonstrates the historical Suno-to-DAW route. Starting from the song’s link, he changes the domain from suno.com to sunodaw.com, retaining the song path. He explicitly corrects his initial wording: add daw to the domain, not to the end of the whole URL. After signing in, the service imports the song for separation. Young estimates 30–90 seconds for import and separation, sometimes longer for more complicated songs.
A digital audio workstation, or DAW, gives the generated song a different form of editability. The imported mix becomes source-separated audio for vocals and instrumental parts, with individual level controls. These are estimates extracted from the mixed recording, not recovered original recording tracks or guaranteed isolation of every instrument. Young first describes keeping music he likes while reducing unwanted vocals, then plays vocals alone and identifies the bass. The task has changed from asking for another whole song to listening to, and adjusting, particular components of this one.
Young identifies the service as WavTool and tentatively describes its accounts as free at the time. For a local alternative, he recommends Ultimate Vocal Remover, or UVR5, an open-source application with a more involved setup and hardware considerations. Its input does not have to be AI-generated: a conventional mixed song can also be separated. The demonstration stops short of a complete mix or master, but it reaches an important production boundary. Patrick’s birthday song is no longer only an output to accept or reject; its vocal and instrumental parts can now be auditioned and balanced separately.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and setup resources for training and using retrieval-based voice-conversion models.
A local application for separating vocals and other sources from mixed audio, with installers and model documentation.
Further reading
Historical documentation of editing selected audio regions, extending clips and Udio's initial paid generation allowances.
- Stable Audio 2.0Article
Explains audio-to-audio generation and the diffusion-transformer architecture behind Stable Audio's 2024 release.
Full transcript
Loading transcript…