How to run Udio UA2: the new best local AI music generator

AI music tools are evolving fast, but most of the best models are locked behind cloud services or paywalls. UA2 changes that. It’s a powerful open-source music generator you can run locally on your own machine, with support for vocals, multiple languages, and even cover songs – all with as little as 4GB of VRAM.

What is UA2?

UA2 is an open-source AI music generation model designed to run locally. It can create full songs with vocals from text prompts and lyrics, and it can also transform existing audio into new versions or covers in different styles.

According to benchmarks shared by its creators, UA2 outperforms other open-source music models like MiniAX Music and AEP 1.5, and even beats some closed-source systems such as Suno v5.5 and Suno v6 on a song quality index. It still trails slightly behind Suno v5 in those tests, but for a local, open-source model, its quality is impressive.

Key features and capabilities

UA2 is more than a simple text-to-music toy. It’s built around a structured, editable representation of music, which makes it unusually flexible compared to many other local models.

Text-to-music with vocals

UA2 can generate complete songs from a style prompt plus lyrics. You describe the genre, mood, and instrumentation (for example, “smoky jazz ballad with piano and upright bass” or “energetic future bass, modern and inspiring”), then provide the lyrics. The model outputs a full track with vocals and backing instrumentation.

It can handle a wide range of genres, including:

• Jazz and lounge-style tracks
• Boogie-woogie and vintage 1930s styles
• Flamenco, folktronica, emo, and more

Multilingual singing

UA2 supports singing in multiple languages. In the demos, it successfully generated tracks in Spanish (flamenco-style), Russian (folktronica), and Korean (emo). As long as you provide lyrics in the target language, UA2 can attempt to sing them in that language while following your style prompt.

Cover songs and reference-based generation

One of UA2’s standout features is its ability to use an existing audio clip as a reference. You can feed it a song and then:

• Preserve the melody but change the style (for example, turning “Auld Lang Syne” into a groovy jazz-funk track)
• Transform classical pieces like Beethoven into theatrical hard rock with completely new lyrics
• Convert familiar tunes such as “Jingle Bells” into a darker, minor-key version

This makes UA2 currently one of the only open-source tools that can produce decent-sounding AI covers locally.

Instrumental-only generation

If you don’t want vocals, UA2 can generate instrumental tracks as well. You still need to provide something in the lyrics field, but you can simply describe the desired instrumental feel in brackets (for example, “[steady buildup of staccato strings and ethnic drums]”) and specify “instrumental only” in the style prompt. The model will then focus on orchestration rather than singing.

Agent-driven music editing

Because UA2 can be controlled entirely via text prompts, you can connect it to AI agents like GPT-based systems or GLM. An agent can:

• Generate an initial song from your prompt
• Iteratively edit it based on your feedback (for example, “keep the melody and lyrics but reharmonize it to sound more jazzy”)
• Adjust instrumentation (“remove the guitar,” “add more piano,” etc.)

This turns UA2 into a flexible backend that agents can use for autonomous or semi-autonomous music creation and refinement.

How UA2 actually works

Most text-to-music systems go directly from prompt to audio. UA2 takes a more structured approach: it first generates an editable musical score, then turns that into audio.

Step 1: Generating an editable score

When you run UA2, it first creates a symbolic representation of the song – essentially basic sheet music. This includes:

• Vocal melody notes
• Instrumental notes
• Key, tempo, and structure

This intermediate format is often referred to as ABC notation. You can preview it as a score before the final audio is generated.

Step 2: Turning the score into audio

Once the notation is ready, UA2 uses that as the backbone to synthesize the full audio track. Because the structure is explicit, it’s easier to:

• Change lyrics while keeping the melody
• Modify the key (for example, from major to minor)
• Generate covers that follow an existing melody but use a different style

This two-step design is a big part of what makes UA2 so editable and flexible compared to other local models.

Performance and hardware requirements

UA2 is optimized to run locally, even on modest GPUs.

• VRAM requirements: There’s a full BF16 model (~7.8 GB) and a quantized version (~3.96 GB). The quantized version is designed for GPUs with less than 4GB VRAM.
• Decode options: If your GPU has 12GB+ VRAM, you can use the faster standard decode mode. With less VRAM, you should use the tiled decode option, which is slower but much more memory-efficient.
• Speed: On a laptop GPU like an RTX 5000 (16GB VRAM), generating a multi-minute track took just over 2 minutes in the example.

Running UA2 inside ComfyUI

While UA2 can be run directly via Python, the easiest way for most people is to use ComfyUI – a popular node-based interface for running local image, video, and audio models. If you’ve already experimented with local image generators like Ernie or ACE, this will feel familiar. (If you’re curious about other local setups, check out how to run Ernie Image as a local image generator or ACE 1.5 XL for music.)

1. Update ComfyUI

First, make sure ComfyUI is up to date:

• Open your ComfyUI root folder.
• Go into the update folder.
• Run update_comfy.bat (or the equivalent script on your OS).
• Wait for the update to complete, then close the window and start a fresh ComfyUI session.

2. Load the UA2 workflow

The easiest way to use UA2 is with a pre-built ComfyUI workflow:

• Download the UA2 workflow JSON file from the project’s page.
• Save it somewhere convenient (for example, your ComfyUI root folder).
• Drag and drop the JSON file onto the ComfyUI interface.
• The full UA2 workflow will load automatically, so you don’t need to build it from scratch.

3. Download and place the model files

When you first load the workflow, you’ll likely see error messages about missing models. To fix this, you need two components from the UA2 model page:

• Audio encoder (~1.4 GB):
• Download the audio encoder file.
• Place it in ComfyUI/models/audio_encoders.
• Checkpoint (main model):
• Choose either the full BF16 model (~7.8 GB) or the smaller quantized model (~3.96 GB).
• Place the file in ComfyUI/models/checkpoints.

Back in ComfyUI, press R to refresh the model list. In the UA2 workflow, set the “Load Checkpoint” node to the model you just downloaded, and set the “Load Audio Encoder” node to the audio encoder file.

Using the text-to-music workflow

The UA2 ComfyUI workflow is split into two main sections. The top section handles pure text-to-music generation (style + lyrics), while the bottom section handles reference audio and covers.

Style prompt and lyrics

In the text-to-music section, you’ll see two key input areas:

• Style prompt: Describe genre, mood, pacing, and instruments (for example, “future bass, modern, energetic, inspiring, wide synths, punchy drums”).
• Lyrics: Paste your lyrics. UA2 understands meta tags like [verse 1], [chorus], [bridge], [intro], and [outro] to structure the song.

Generating and previewing the score

The lyrics and style prompt feed into a “Generate ABC” node, which creates the underlying notation. A preview node lets you inspect this notation as a kind of simplified sheet music, showing vocal and instrumental lines plus tempo and key.

Song length and generation settings

Before you run the full generation, you can tweak several important parameters:

• Max duration (seconds): Sets the upper limit for song length. For example, 360 seconds = 6 minutes. If your lyrics are shorter, the song will usually end sooner.
• Seed: A numeric ID that controls randomness. Using the same seed with the same settings will reproduce the same song. Change it to get a different variation with identical prompts.
• Steps: The number of diffusion steps. More steps generally mean higher quality but slower generation. The default of 32 is a good starting point.
• CFG (classifier-free guidance): Controls how strictly the model follows your prompts. If the style or lyrics feel off, increase CFG slightly to make it obey the prompt more literally.
• Sampler and scheduler: These are the underlying generation algorithms. The defaults work well for most cases.

Finally, choose the appropriate decode node based on your VRAM (standard vs tiled), connect it to the output, and run the workflow. ComfyUI will generate and save the resulting audio file.

Using reference audio and creating covers

The bottom section of the UA2 workflow adds reference audio support for cover-style generation and style transfer.

Switching from text-only to reference mode

To use reference audio:

• Bypass the top “Generate ABC” node and its preview node (select them and press Ctrl+B).
• Un-bypass (enable) the nodes in the reference audio section (select and press Ctrl+B again if needed).
• Connect the output of the reference-based ABC node into the main “Generate Music” node.

Uploading and configuring your reference

In the reference section, you’ll find an audio loader node where you can upload a clip from an existing song. You then have two options for how UA2 interprets it:

• Melody only: Focuses on the melodic line and ignores most arrangement details. This is usually best for covers and melody transfers.
• Full song: Uses more information from the original track. The difference can be subtle, but melody-only tends to give more flexible, style-shifted results.

Next, you provide a new style prompt and lyrics. You can:

• Keep the original lyrics and just change the style (for example, turning a pop song into a jazz ballad with piano, sax, double bass, and brushed drums).
• Swap in completely new lyrics while preserving the original melody.
• Translate the lyrics into another language and have UA2 sing the new version over the same melodic structure.

Once configured, run the workflow and UA2 will generate a new track that follows the reference melody while honoring your new style and lyrics.

Advanced tips and creative use cases

Because UA2 exposes an editable score and works well with agents, there are plenty of creative workflows you can explore:

• Iterative arrangement: Start with a simple pop version of a song, then iteratively adjust prompts to explore jazz, rock, orchestral, or electronic arrangements while preserving the core melody.
• Key and mood changes: Use the score representation to flip songs from major to minor, or vice versa, and regenerate them with a new emotional tone.
• Language covers: Take an English track, keep the melody, and rewrite the lyrics in another language to create multilingual versions.
• Agent-driven workflows: Let an AI agent handle tasks like lyric writing, style selection, and iterative feedback, turning UA2 into a backend engine for autonomous music creation.

Licensing and commercial use

UA2’s licensing is split into two parts:

• Code and documentation: Released under the Apache 2.0 license, which is very permissive and allows commercial use.
• Model weights: Released under a Creative Commons Non-Commercial license. This explicitly restricts use to scenarios that are not primarily intended for commercial advantage or monetary compensation.

In practice, this means you should treat UA2’s outputs as non-commercial unless you obtain additional permissions. Posting tracks on platforms like Spotify and monetizing them may not be allowed under the current weight license, so be cautious and review the exact terms if you plan to use it for anything revenue-generating.

Why UA2 matters for local AI music

UA2 represents a major step forward for local, open-source music generation. It offers:

• High-quality vocal and instrumental output
• Multilingual singing
• Cover and reference-based generation
• Deep editability through its score-first design
• Reasonable hardware requirements, with a 4GB-friendly quantized model

If you’re serious about experimenting with AI music locally, UA2 is one of the most capable options available right now, especially when combined with ComfyUI and agent-based workflows.

Whether you’re a content creator, hobbyist producer, or developer building AI music tools, UA2 is well worth installing and exploring.