EDGE

EDGE: Editable AI Dance Generation from Music

EDGE is a cutting-edge AI tool that allows users to generate and edit dance choreographies from music. It leverages a transformer-based diffusion model combined with Jukebox, a powerful music feature extractor, to create realistic and physically plausible dances. This technology opens up new possibilities for dancers, choreographers, game developers, and anyone interested in exploring the intersection of AI and dance.

What is EDGE?

EDGE is a method for editable dance generation capable of creating realistic, physically-plausible dances while remaining faithful to arbitrary input music. It is built upon a transformer-based diffusion model and utilizes Jukebox for music feature extraction. This combination enables powerful editing capabilities, making it well-suited for dance applications.

How does EDGE work?

1. Music Embedding: EDGE uses a pre-trained Jukebox model to encode input music into embeddings, capturing the nuances and characteristics of the music.
2. Conditional Diffusion Model: A conditional diffusion model learns to map the music embedding into a series of 5-second dance clips. This model is trained on a dataset of dance motions, allowing it to generate realistic and diverse dance sequences.
3. Temporal Constraints: At inference time, temporal constraints are applied to batches of multiple clips to enforce temporal consistency. This ensures that the generated dance flows smoothly and naturally over time.
4. Stitching: The temporally consistent clips are then stitched together to create a full video of arbitrary length. This allows users to generate dances that match the duration of their input music.

Key Features and Capabilities

• Joint-Wise Conditioning: EDGE allows users to control the motion of specific body joints, enabling targeted edits and customizations. For example, users can generate the lower body motion based on the upper body movement, or vice versa.
• Motion In-Betweening: This feature allows users to specify the starting and ending motions of a dance, and EDGE will automatically generate the intermediate motions to create a seamless transition.
• Dance Continuation: Users can provide an initial dance sequence, and EDGE will generate a continuation that seamlessly extends the original motion.
• Arbitrarily Long Dances: EDGE can generate dances of any length by enforcing temporal continuity between batches of multiple sequences.
• Physical Plausibility: EDGE is trained with physical realism in mind, avoiding unintentional foot sliding and ensuring that the generated dances adhere to realistic biomechanical principles. The Contact Consistency Loss ensures that feet should and shouldn't slide, which significantly improves physical realism while keeping sliding intact.

Why choose EDGE?

EDGE outperforms existing methods in terms of realism, physical plausibility, and editability. Human raters strongly prefer dances generated by EDGE over those of previous work like Bailando and FACT.

Who is EDGE for?

• Dancers and Choreographers: EDGE can be used as a tool for exploring new dance ideas, generating variations on existing choreography, or creating custom dances for specific musical pieces.
• Game Developers: EDGE can be used to generate realistic and diverse dance animations for video game characters.
• AI researchers: EDGE serves as a research platform for exploring the intersection of AI and dance.
• Anyone interested in AI and dance: EDGE provides an accessible and engaging way to explore the creative potential of AI in the realm of dance.

How to use EDGE?

1. Upload or select a music track.
2. Use Joint-Wise Constraint to generate lower body from upper body, or vice versa.
3. Use Motion In-Betweening to create dances that start and end with prespecified motions.
4. Use Dance Continuation to create dances that start with a prespecified motion.
5. Download the generated dance video.

EDGE represents a significant advancement in the field of AI-driven dance generation, offering a powerful and versatile tool for creating realistic and editable dance choreographies. Its ability to generate physically plausible motions, combined with its flexible editing capabilities, makes it an invaluable resource for dancers, choreographers, game developers, and anyone interested in exploring the intersection of AI and dance.