The Paper was written by Doga Cavdir, Chris Clarke, Patrick Chiu, Laurent Denoue and Don Kimber and published at the NIME conference in 2021. It’s about a real-time movement sonification framework for physical activity practices and learnings with Reactive Video.
Reactive Video (RV) adjusts video playback based on user movement, offering real-time visual and summary feedback by mirroring the instructor’s movements. The newly implemented auditory feedback is designed to assist users in aligning their poses with the instructor’s movements, focusing on pose error and play speed metrics.
Personally, I really like the idea and concept of auditory feedback in Reactive Videos for learning and practicing physical activities. The ability for users to perform movements at their own pace without the video continuing in its original tempo in the background, making it difficult for users to keep up with the trainer, is very valuable. This way, users don’t lose motivation due to failures and can focus more or less on executing the movements correctly. Thanks to Movement Sonification, they also receive real-time feedback and can adjust their movements to perform exercises better and correctly.
This framework has many potential applications. Although Tai Chi was used for development because of its slow movements, I can imagine that this could also be applied to other sports such as yoga, CrossFit, or table tennis.
While I am enthusiastic about the concept, I also have some concerns. In the example videos provided, I noticed that more atmospheric sounds were used. This isn’t necessarily a bad choice as it keeps users immersed in the video or exercise. However, I wonder if these sounds are effective or guiding enough for users. Personally, I think more distinct sounds could work better, but this would need to be tested with users. It’s also possible that I’m completely off-base and that the chosen sounds are already optimal for this use case. I can’t judge this as I’m not knowledgeable in sound design.
The Paper: https://www.nime.org/proc/nime21_14/index.html
Video excerpts:
Both audio clues: https://youtu.be/WvZAZ5vywyk
Audio clues for pose alignment: https://youtu.be/CfzI8a6bw0g
Audio clues for tempo alignment: https://www.youtu.be/Df7Q0Fqgg0