For a class assignment on synesthesia, I decided to build something to turn a video input into sound. This was in some ways a culmination of a several interactions I created with the clear purpose of being counter-intuitive. In this case, the video camera provides the material to be translated into sound, but it’s set up such that only moving sections of video will result in nonzero samples: the video output being scanned is seen on the television in the background. At the same time, a wii remote is used to control the rate that the pixels are being scanned, which adds some gross control of pitch. The two simultaneous interactions are sometimes at odds with each other.
I set this up for my classmates and let them try it out, which added the element of physical performance. Afterwards, my professor asked me, “Where is the piece?” or, in other words, where should we be looking? At the screen or at the person gesticulating wildly? I didn’t have an answer: later I realized that I was pleased with the ambiguity.
Here is the Max Patch
and here is my Osculator Conf