AI Engineer World's Fair 2024

AI Music Generation: From Prompt to Production

Phlo Young54:34

Loading slides…

Read the talk

AI Music Generation: From Prompt to Production

Follow Phlo Young from voice conversion and text-generated songs to personalized prompts, lyric preparation, and hands-on control of separated vocals and instruments.

From a talk by Phlo Young

Start with the listener’s taste

Have you used a music-generation model—and what would you want it to make? Phlo Young opens with those questions. About half the room has tried music generation. Boomy draws little recognition; Young remembers it as an earlier, less complex tool. Suno and Udio, the workshop’s main tools, are more familiar. Asked for a favorite artist, one listener names Miles Davis, and the room supplies the genre: jazz. That exchange establishes the practical problem: translate someone’s musical taste into something they can hear.

Young approaches the task as a self-taught songwriter, not a classically trained musician. He wrote his first song when young and welcomes corrections. QR-linked notes, live questions, and a short poll let participants steer the workshop toward their interests.

His experimental approach predates generative music. After learning Python for automation during the pandemic, he also began testing music distribution: release the same recording under different artist names, vary the cover art, marketing, and audience targeting, and observe how it performs. Young reports that one instrumental pen name accumulated about five million streams over the preceding couple of months. His own background spans R&B and hip-hop, with EDM and country-influenced experiments; generation adds another way to explore those musical directions. Distribution and artist income motivate him, although the workshop does not develop a release or monetization procedure.

The poll favors prompting and tweaking. A more advanced option—download generated music, separate stems, then mix and master—also attracts interest. Young points to Suno and Udio’s Discord communities, including an unnamed contributor whose unusual banjo and bagpipe prompts produce striking results. The useful premise is that effective musical descriptions may be less intuitive than simply naming an instrument.

1:361:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:36 · section reference included

Separate three meanings of AI music

The input tells you which part of music-making the model is doing. Young found “AI music” frustratingly ambiguous because it could describe a generated song, an expanded recording, or a familiar voice applied to someone else’s performance.

ApproachInputTransformation
Text to musicDescription, optionally lyricsGenerate a sample or song
Audio to musicExisting soundTransform or develop it into music
Voice conversionRecorded voiceChange its vocal identity

Young uses style transfer for the third category. In the examples that first attracted his attention, consumers often called voice conversion “AI music” without distinguishing it from generating the composition itself.

10:0010:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:00 · section reference included

A human performance, a different vocal identity

The first video makes the distinction audible. Its creator, whom Young tentatively identifies as Roberto Nickson, begins with a Kanye-style voice and a converted excerpt of Day ’N Nite. He then finds a Kanye-style beat on YouTube, writes eight bars, and records them himself. The workstation view shows the human performance entering an ordinary audio-production workflow.

Audio workstation with waveform tracks and an inset of a man performing into a microphone.Open full source frame
Recording vocals alongside an audio-track timeline.

Next comes the conversion: the same recorded performance plays back with a Kanye-like vocal identity. The beat, writing, and reference performance were already supplied by a person. Voice conversion changes the voice without replacing the entire songwriting process. Seeing this sequence pushed Young into Discord servers and subreddits to learn how to reproduce it.

Ghostwriter’s viral Drake-and-The-Weeknd-style song illustrates the same distinction. In Young’s account, Ghostwriter wrote and produced the song and converted his own vocals; the workshop plays an excerpt rather than demonstrating its creation. Young recalls finding it below 1,000 views and estimates nine million streams in its first 24 hours. Those are his recollections, not verified platform totals. His attribution of the removal to the RIAA also differs from contemporaneous reporting, which identifies UMG in the YouTube copyright notice. The mechanism remains clear even where the viral history is uncertain: an authored recording became associated with recognizable voices through conversion.

11:5812:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:58 · section reference included

Voice conversion in the artist’s hands

Listening again to the earlier Kanye example, Young now hears conspicuous artifacts. What initially astonished him sounds less natural after subsequent improvements in voice models. He credits experimentation within communities that included teenage model trainers; his description of the tools becoming “10 times better” expresses that listening impression, not a measured quality improvement.

The Randy Travis example changes the purpose of the technique. Travis had lost his voice, and his team used conversion to make a new recording available in his vocal identity. Young plays an excerpt and describes unusually positive listener reactions, including a comment from an older fan moved to tears. Against the hostility he often encounters around AI music, this artist-directed use offers a different relationship between performer and technology: the recognizable voice helps the artist continue making music.

16:4216:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:42 · section reference included

When the prompt supplies the starting material

Text-to-music moves the starting point upstream. Young turns from converted recordings to full generated songs, beginning with BBL Drizzy in the context of the Drake–Kendrick dispute. The distinction matters more than the meme’s backstory: this example starts with words rather than a recorded singer. The taxonomy slide separates that route from transforming existing audio or converting a voice.

Slide titled “What is AI Music?” listing text-based generation, transformation of existing sounds, and audio filtering or voice conversion.Open full source frame
Three approaches: text to music, audio to music, and style transfer.

Young describes comedian King Willonius entering a prompt into Udio and, as he recalls less confidently, supplying lyrics. In his account, there was no studio vocal performance to replace: text went in and a song came out. He characterizes it as a one-shot generation and emphasizes its subsequent reuse in other songs. That is the workshop’s clearest contrast with Ghostwriter’s human-authored reference performance. Copyright questions arise here but remain unresolved.

A second example comes from ElevenLabs: a text-generated song about programming and GPUs. Its lyrics describe writing code and hoping to make it work, making the result a self-referential example of an AI system singing about AI development. Young reports that it circulated widely on Twitter and groups it with the meme-song examples.

20:1020:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:10 · section reference included

Beatboxing and desk percussion become arrangements

Audio-to-music keeps a performed sound at the beginning of the process. A Stable Audio example starts with beatboxing and transforms that input into music. Young remembers it as the first publicly available audio-input feature of this kind he encountered, while distinguishing a musical output from a necessarily complete song. A second clip, from the Suno team, starts with fingers drumming on a desk and produces rock music with guitar and other instruments. The attraction for a musician is immediate: a rhythm can be performed directly instead of translated entirely into descriptive text.

An audience member hears a Tom Petty resemblance in the preceding generation. That observation arrives alongside Young’s discussion of complaints alleging that Suno and Udio trained on copyrighted music. This is a historical workshop, held at the AI Engineer World’s Fair on June 26, 2024; the interface behavior, account options, and service integrations that follow belong to that demonstration. Young has no affiliation with the featured services. He points technically curious participants toward music-model papers in his notes, but follows the poll’s preference for practical work rather than explaining model architectures.

24:5325:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:53 · section reference included

Give everyone a way to generate

To let participants work alongside him, Young distributes business-card codes. Entering a code through the green Gateway button reveals credentials for workshop accounts. The demonstrated login choices include Google, Discord, Twitter, and Apple for Udio, and Discord, Google, and Microsoft for Suno. He has paid for accounts to expand the room’s generation allowance, although his “Premier” and “unlimited” terminology should not be read as Udio’s documented plan names or limits: its original subscription announcement describes finite-credit Standard and Pro plans.

Apple’s two-factor authentication may obstruct some logins, but participants confirm that they can get in. The participation loop is straightforward: generate a song, copy its share link, and submit it through the workshop’s Google Form. Young proposes randomly selecting two participants to keep one account each. He has purchased a month of access rather than annual subscriptions.

29:0329:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

29:03 · section reference included

Build Patrick’s birthday song

Patrick has a birthday that week, so the first live task becomes a personalized birthday song. Young initially includes a Randy Travis reference and anticipates artist-name filtering. An audience member suggests changing the spelling and adding the associated musical style; another supplies a more useful personal detail: Patrick is a patent attorney. Patrick names The Band as a favorite, and the room settles on rockabilly as a prompt description.

Young starts generation and carries the same prompt across services. The artist reference becomes a concrete failure case: a rejection names Randy Travis, after which he tries the altered spelling Tandy Ravis. This does not establish a reliable way around moderation. The useful prompt ingredients are already independent of that attempt: the occasion, the person, his occupation, and a musical direction.

32:1232:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:12 · section reference included

A songwriting partner and an in-house producer

Young thinks of Udio as a songwriting partner. He describes its default workflow as generating roughly 30 seconds of music at a time, with an experimental model available in the paid account shown. That short result becomes an object to revise rather than an answer to accept wholesale.

ActionPurpose
ExtendContinue a promising idea
RemixExplore a different version
DiscardAbandon an unhelpful result
InpaintModify a selected section
Upload audioStart from an existing recording

For someone with original music, Young recommends bringing the recording, tempo, key, and lyrics into the process. He has tried this with his own songs and found the reinterpretations surprisingly compelling.

Suno, by contrast, feels to him like an in-house music producer: its output often has a polished, Top 40 character. His suggestion that this might come from a hidden system prompt or the training data is speculation, not knowledge of its internals. While onboarding friends and family, he noticed a more actionable issue in custom mode: lyric lines with poorly controlled syllable counts could fall badly out of alignment with the beat. He reports that his more recent generations handled timing and song structure better. The comparison is about his working experience—iterative development on one side, production polish on the other—not a controlled quality ranking.

35:2935:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

35:29 · section reference included

Generate an event song and listen to the birthday results

The next prompt is a song about the AI Engineer World’s Fair. Organizer Benjamin Dunphy’s taste suggests old-school hip-hop and boom bap. The demonstrated procedure is short:

  1. Sign in to Udio and open Create.
  2. Enter the song description and musical direction.
  3. Choose generated lyrics, supply custom lyrics, or request an instrumental.
  4. Choose an available model and start generation.

The free-account default shown is Udio 32; the longer experimental option belongs to the paid experience in the demonstration. A compact prompt expressing the spoken request is:

A song about the AI Engineer World's Fair.
Old-school hip-hop, boom bap.

Young then uses the same prompt in the other service. Before starting, he refreshes the page because he has encountered a state problem: repeated generation could unexpectedly extend a previous song instead of starting a new one.

Returning to Patrick’s songs, Young first encounters another moderation error associated with the Randy Travis reference. Other results do play: the vocals name Patrick, celebrate his birthday, and refer to rock and roll. Patrick calls one result the “Theme song to my sitcom.” On Suno, clicking the song name exposes its lyrics. A later alternate playback seems to repeat the same song, and Young leaves open whether the service generated it twice. The session produces enjoyable personalized music, but neither the birthday playback nor the event-song experiment becomes a systematic comparison of the services.

39:1639:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

39:16 · section reference included

Prepare and edit lyrics before generating audio

With time running short, Young turns to the workshop notes, where he also plans to publish the recording and transcript. A GPT-4 prompt template takes a Song Description and asks for structured lyrics. The displayed notes include style tags, an intro, verses, a chorus, an instrumental break, and an outro. Young flags a GitHub rendering problem around brackets, but the workflow is still clear: have ChatGPT draft the lyrics, edit them yourself, then give the music model the revised text. Separating lyric preparation from audio generation creates an explicit revision step before the musical performance is generated.

GitHub notes showing a GPT prompt with song description, style tags, intro, verses, chorus, instrumental break, and outro, followed by music prompt examples.Open full source frame
Workshop notes provide a structured lyric prompt.

For Patrick’s example, that preparation step can be expressed as a reusable instruction:

Song Description:
A rockabilly birthday song for Patrick, a patent attorney
who loves rock and roll.

Draft lyrics with an intro, verses, chorus,
instrumental break, and outro.
Keep the lines singable and the chorus easy to repeat.
Return the lyrics so I can edit them before music generation.

The song description supplies the subject and direction; the separate lyric draft gives the user a chance to change phrasing before spending another generation on audio. Young also points to official Suno and Udio tips and returns briefly to the community “Prompt Master,” without completing that deeper walkthrough.

44:0244:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

44:02 · section reference included

Generic voices, emotional delivery, and descriptive labels

Philip asks whether conversion can produce a generic rapper’s voice instead of a recognizable performer such as Eminem. Young proposes two approaches:

  • Blend voice sources: combine material from several speakers rather than targeting one identity.
  • Use a synthetic source: obtain a TTS voice and use that output as training material.

Young suggests that some unspecified tools can begin with six seconds of voice, illustrating a blend with equal short samples from different people. He names no model, training conditions, or quality criterion for that minimum. Neither blending nor synthetic input by itself establishes permission to use the source material or clears rights in the result.

A responsible-use question gets a non-lawyer disclaimer rather than substantive ethical guidance. Another question asks about preserving emotional tone, citing the changing delivery in Kendrick Lamar’s Euphoria. Young recommends RVC as his preferred voice-conversion tool at the time. He emphasizes that cloud execution is an alternative to running it locally, where hardware requirements may be an obstacle. He also confirms that participants can have the slides.

After an audience member reports that the shared Google document requires an access request, Young acknowledges the permissions mistake. His final concrete prompting technique is to search Rate Your Music by artist, genre, or desired sound, collect the descriptive labels attached to relevant music, and use those labels in Suno or Udio prompts. He tentatively extends the suggestion to Stable Audio. The practical move is to replace a vague taste reference with musical vocabulary. His further claim that the models were trained on Rate Your Music labels comes from unnamed reverse engineering and is not established training-data provenance.

Young acknowledges the distinction between ethics and law but defers that discussion for lack of time. He reports twelve song submissions and proposes a later drawing, possibly streamed, followed by changing the shared credentials before handing accounts to the winners. That handoff remains a plan. The apparent closing then plays the World’s Fair song, whose lyrics include copper wires, coders, blueprints, and robots. Young reacts positively and wonders what Udio produced, without completing the comparison.

45:5346:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

45:53 · section reference included

Separate the song into parts you can control

A late question brings the workshop back to production: how do you pull out stems and edit them? Young returns to Patrick’s birthday song and demonstrates the historical Suno-to-DAW route. Starting from the song’s link, he changes the domain from suno.com to sunodaw.com, retaining the song path. He explicitly corrects his initial wording: add daw to the domain, not to the end of the whole URL. After signing in, the service imports the song for separation. Young estimates 30–90 seconds for import and separation, sometimes longer for more complicated songs.

A digital audio workstation, or DAW, gives the generated song a different form of editability. The imported mix becomes source-separated audio for vocals and instrumental parts, with individual level controls. These are estimates extracted from the mixed recording, not recovered original recording tracks or guaranteed isolation of every instrument. Young first describes keeping music he likes while reducing unwanted vocals, then plays vocals alone and identifies the bass. The task has changed from asking for another whole song to listening to, and adjusting, particular components of this one.

WavTool timeline with stacked audio tracks, a blue selected region, a vertical playhead, and two waveform channels below.Open full source frame
WavTool displays multiple audio tracks and an expanded waveform.

Young identifies the service as WavTool and tentatively describes its accounts as free at the time. For a local alternative, he recommends Ultimate Vocal Remover, or UVR5, an open-source application with a more involved setup and hardware considerations. Its input does not have to be AI-generated: a conventional mixed song can also be separated. The demonstration stops short of a complete mix or master, but it reaches an important production boundary. Patrick’s birthday song is no longer only an output to accept or reject; its vocal and instrumental parts can now be auditioned and balanced separately.

50:3750:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

50:37 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…