SCORE! From video to audio through deep latent representations
SCORE! is a cooperation between the VU University Amsterdam and the national institute Sound and Vision. Its aim is to use simple unsupervised analysis of audio and video, to create meaningful mappings between the two, through latent-space representations. We focus on using these mappings as a creative tool: for instance to automatically generate a soundtrack to a piece of video. However, the methods we use have much broader applicability, for instance in information retrieval, e-humanities or general multimedia manipulation. Examples Some preliminary examples of video-to-audio mapping can be found here.
All the code is available under an open source license on GitHub here.