CS seminar (CSC 781) September 10: Synchronizing Visual Images with the Sounds They Interpret

This talk will explore the challenges of synchronizing spectral analysis of a piece of music with an image representing the frequency changes of the music as it plays in real-time.  In particular, a Julia fractal is computed, and its color map is altered every n milliseconds, corresponding to the frequency components in n milliseconds of music.  The computational challenge lies in synchronizing the Fourier analysis of the section of music with the the computationally complex recomputation of the fractal in the same amount of time.  The goal is to have the colors in the fractal reflect the changing frequencies. Doing this off-line and creating multiple images that later can be composed into a video is not difficult.  Doing the computation in real-time is the challenge.  Analyzing the computational complexity of the components of the problem sheds light on how sound analysis is performed over time. The need for synchronized threads adds to the programming difficulty. And zooming in on the Julia fractal with changing colors makes the problem visually interesting.