In this bachelor's thesis, I conceived a program that combines my two parallel careers: electrical engineering and dance. The basic idea originated from the question of how movement could be transformed into sound. Whenever a dancer moves in silence, they create their own music in their mind. The goal was to develop a prototype program in which music follows movement rather than the other way around.
The program had to combine motion capture technologies and sound synthesis. For the development of the program, I used the Python programming language because of the wide availability of libraries that enable the integration of both fields. I used the OpenCV and MediaPipe libraries for computer vision and body pose estimation, and the NumPy and SoundDevice libraries for sound synthesis and playback.
The body positions captured by the camera are stored as arrays of numbers, on which mathematical operations can be performed and which can be transformed into sound waves. The frequencies of a chord are linked to the position of the head, with the frequency ratios remaining constant while their absolute pitch changes according to the height of the head. The distance between the hands affects the dampening of the chord, while the speed of hand movement determines the volume of the highest notes.
The sound produced by this mapping of movement was not sufficiently dynamic, so I also incorporated triggers linked to the positions of the hands and feet into the program. For this purpose, I required a function that emulates the sound of a piano. When the feet rise above hip level, a random lower note from the pentatonic C major scale with a longer sound decay is triggered, whereas when the hands rise above head level, a random higher note with a shorter sound decay is triggered. These events contribute to greater expressiveness and musical diversity.
The results show that the developed prototype enables a meaningful transformation of movement into sound that satisfactorily reflects the characteristics of the movement. In this way, the primary objective of the bachelor's thesis is achieved, whereby the body becomes an instrument for creating music and serves as an example of integrating computer vision, sound synthesis, and artistic expression, while opening possibilities for further development.
|