SIGNATURE WORK
CONFERENCE & EXHIBITION 2022

Dance choreography has often been viewed as a creative manifest rather than a scientific problem. Yet we recognize that dances are permutations and combinations of fundamental body movements, and the process of choreography is planning and arranging these movement units based on perception of music, performing experiences and spontaneous creativity. In this team-based signature work, we explored the possibility of using deep learning to automatically generate dance movements based on music input. We created a high-quality Motion Capture dataset for ballet and classical Chinese dance. Dance motions are captured with 55 body markers, including finger-tracking-a critical feature for AI to characterize the fluidity of dance that is lacking in most of the existing works. Our inclusion of this feature makes it a unique contribution to the open-source community at the intersection of Artificial Intelligence and Art. As dancers for over a decade, we incorporated dancer-inspired innovations and implemented a transformer-based multimodal seq2seq model with an attention mechanism that encompasses the human choreographic process, which includes (i) identifying music patterns, (ii) predicting key movements, and (iii) joining movements with smooth transitions. Experiments have demonstrated that the model, trained on our high-quality dataset, can produce more diverse, on-beat, style-consistent, and natural movements. The outcome of our work is not only insightful and instructive from the research aspect but also has great market potential. It could be adopted for empowering dancers to develop dance performances, enhancing virtual gaming, and facilitating avatar interactions in the Metaverse.