Motionscapes is a installation which allows children with profound learning needs to interact creatively with the world around them in a way not previously possible. It utilizes computer-vision techniques to allow students' simple physical gestures and motions to control and transform a series of playful audio-visual landscapes. The aim is to give children with special needs an intuitive and fluid means of self-discovery and expression.

Utilizing an ordinary security camera, the system transforms simple motions and gestures into meaningful responses. The software uses computer vision techniques with the aim of understanding what kind of motion is occurring across the frame. Since the focus of analysis is on motion, and not simply presence, the piece allows the child to interact independently if the care giver remains still.

Motionscapes was created with the support of an educational grant from NESTA, together with Land Design Studio, and working closely with technical and media partners, Production Science, Simple Productions, and ARS Electronica Futurelab.

video: examples - silent (12mb) (this film requires latest quicktime)

Twirl presents a twirling ball which move from left to right across the screen. As the ball moves, it circles around a central axis. There is a simple looping sound associated to the ball, so as the ball moves across, the sound pans. As students move more rapidly, the ball increases in size and the pitch of the note rises. With little or no motion, the system quickly settles back down.

In fluid, motion is transformed into paint in an abstract fluid world. Upward motion is deposited in red, while downward motion paints in green. In addition, when students move, they deposit small particles which swim through this fluid. Large, emphatic gestures activate the fluid and make the scene move in certain directions. Over time, with little or no input, the fluid settles down. The sound uses a direct mapping of the particle positions to control pan, volume, frequency and timbral brightness of an FM granular synthesizer.

In loops, results of the motion extraction are presented back to the students though a system which presents a time delay. There are, on screen, X many panels. The top right displays the current frame, while subsequent panels, moving from right to left and down the screen, display consecutively older frames. When a motion is made, it starts in the upper right corner, and works its way down the screen. It takes approximately 2 seconds for the motion to completely leave the screen. This sets up an internal rhythm - since as the motions you make last for a set amount of time, it is easy to begin making patterned movement based on this timescale. When there is no motion, starting in the upper right corner and working its way down, the screen clears. The sound is a simple sound which is tightly based on the amount, position and distribution of incoming motion.

Stripes presents a series of vertical bands of color, which are activated through motion. As a child moves from left to right, they activate the colors, which remain on screen for certain duration of time before fading out. The angle of motion creates different colors - for example, by raising your arms up, you can get a series of deep red bands, while moving from left to right paints the bands a sky blue. The angle of motion is mapped cyclically to the rainbow. With little or no motion, the bands fade out and scene is black. The sonification is tightly coupled to the energy of the bands of color, as well as the position and angle of the motion occurring in the scene.

Bounce uses motion which is occurring across the frame to control a collection of bouncing balls. When there is no motion, the balls settle into a neat pile on the screen. When there is motion, it sends the balls flying around the screen, bouncing into each other and the walls. The motion can also be used to control the balls, scooting them from side to side, or arranging them in piles. The sound is tightly related to the horizontal positions on screen of each individual ball, as well its velocity and height. When the balls settle, there is no sound. After they are activated, the ball's sounds come to life and all of the collisions and fast movements can be heard as well as seen.