MusicLM

MusicLM: Generating Music From Text

MusicLM is an AI model developed by Google Research that generates high-fidelity music from text descriptions. It approaches conditional music generation as a hierarchical sequence-to-sequence modeling task. This allows it to generate music at 24 kHz that maintains consistency over several minutes.

What is MusicLM?

MusicLM is a cutting-edge AI model designed to create music from textual descriptions. Unlike previous systems, MusicLM excels in both audio quality and its ability to adhere strictly to the given text description. For example, it can generate "a calming violin melody backed by a distorted guitar riff" based solely on that text.

How does MusicLM work?

MusicLM works by framing music generation as a hierarchical sequence-to-sequence modeling problem. This approach enables the model to generate long, coherent musical pieces at a high-fidelity 24 kHz. The model can also be conditioned on both text and melody, allowing it to transform whistled or hummed melodies into various styles described in a text caption.

Key Features and Capabilities

• High-Fidelity Music Generation: Generates music at 24 kHz, ensuring high audio quality.
• Text-to-Music Conversion: Creates music from text descriptions, such as specific instrument combinations or genres.
• Melody Conditioning: Transforms hummed or whistled melodies into different styles based on text captions.
• Long Generation: Maintains music consistency over several minutes.

Use Cases

• Soundtrack Creation: Generating soundtracks for games, videos, or other media based on textual descriptions.
• Music Composition: Assisting musicians and composers in creating new musical pieces.
• Personalized Music Generation: Creating music tailored to individual preferences described in text.
• Creative Exploration: Exploring different musical styles and combinations through text prompts.

Examples of Audio Generation From Rich Captions

• Arcade Game Soundtrack: Generates a fast-paced, upbeat track with catchy electric guitar riffs, repetitive melodies, and unexpected cymbal crashes and drum rolls.
• Spacey Reggaeton Fusion: Creates a fusion of reggaeton and electronic dance music with an otherworldly sound, evoking a sense of wonder and danceability.
• Soothing Synth Buildup: Produces a track with rising synth arpeggios, pads, sub bass lines, and soft drums, creating a soothing and adventurous atmosphere suitable for festivals.
• Relaxed Reggae Song: Generates a slow tempo, bass-and-drums-led reggae song with sustained electric guitar, high-pitched bongos, and relaxed, expressive vocals.

Story Mode

MusicLM can generate music based on a sequence of text prompts, influencing how the model continues the semantic tokens derived from the previous caption. For example, it can create a musical story with different sections:

• Time to meditate (0:00-0:15): A calm and peaceful introduction.
• Time to wake up (0:15-0:30): A more energetic and uplifting segment.
• Time to run (0:30-0:45): A fast-paced and rhythmic section.
• Time to give 100% (0:45-0:60): An intense and motivational conclusion.

Text and Melody Conditioning

MusicLM can generate music that respects a given text prompt while following a provided melody. Examples include transforming a hummed or whistled melody into different styles such as a cappella chorus, electronic synth lead, guitar solo, jazz with saxophone, and more.

Painting Caption Conditioning

MusicLM can generate music inspired by painting descriptions, creating soundscapes that reflect the visual and emotional content of the artwork. Examples include:

• The Persistence of Memory - Salvador Dalí: Generates music that captures the surreal and dreamlike atmosphere of the painting.
• Napoleon Crossing the Alps - Jacques-Louis David: Creates a majestic and heroic musical piece.
• Dance - Henri Matisse: Produces a joyful and rhythmic composition.
• The Scream - Edvard Munch: Generates a disturbing and unsettling soundscape.

Datasets

To support future research, the MusicLM team publicly released MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts.

Who is MusicLM for?

MusicLM is designed for:

• Musicians and composers seeking new tools for creating music.
• Game developers and filmmakers needing custom soundtracks.
• AI researchers exploring text-to-music generation.
• Anyone interested in exploring the intersection of AI and music.

Why choose MusicLM?

MusicLM stands out due to its:

• High-fidelity audio generation.
• Ability to adhere to detailed text descriptions.
• Capacity to transform melodies into various styles.
• Support for long and consistent musical pieces.

MusicLM is a powerful tool for generating high-quality music from text descriptions, offering a wide range of creative possibilities for musicians, developers, and researchers alike.