Generative Study – Generating sound using materiality

CONCEPT – Converting texture of three-dimensional forms into sound

My idea is to create a ‘instrumental’ system to generate sound using data from physical materials. To me, instruments are generative systems, while composed music are outcomes. Intrigued by our material perception, I would like to map the patterns of stones (primary medium of interest) and convert the visual data into sound waves/ audio signals in real-time.  I am interested in exploring the perceptual commonalities of our senses of sight, touch and hearing. When we touch a ‘rough’ material and we hear a ‘rough’ sound, how do we cognitively recognise or associate a tactile quality of ‘roughness’ to what we sense? My project is an attempt to connect material textures (through sight and touch) to sound textures.

Perimeters/ Structure of system:

Connecting textural qualities (visual) to auditory qualities, the sound quality will vary according to the type of stone/ texture used. I would define a space when the stones can be placed in the range of the camera lens (kinect). When the stone is stationary, the three dimensional data captured by the camera will be mapped as two-dimensional lines to be converted into sound waves (Need for a software that connects visual data to audio). I will also experiment with other forms of data (numerical based on volume/ texture density) that can be converted into audio data. 

“Oh that sounds like a rough rock!” “That sounds like a smooth stone!”

Interaction design:

Each stone when placed under camera will have a sound loop related to their material qualities. The audience can move the stone to change the sound, using movement feedback as a ‘synthesizer’. Stones of different textures can be switched and combined, to ‘play’ the instrument to generate music in real time. 


  • Ambient Techno
    Connecting atmospheric textures of ambient music with the melodic and rhythmic elements of techno and electro.
  • “Play the Barcoder” by ELECTRONICOS FANTASTICOS, Ei Wada– Converting patterns into sounds
  • “The sound of the earth” Yuri Suzuki
    A record player uses textures/ grooves (sound waves directed into the diaphragm) on a record to create vibrations when in contact with a needle. The sound is played through an amplifier. This is an analogue method of generating sound through textures on a “predetermined” record, similar to a conventional artwork.  My projects deals with a digital/ computational system to generate sound as an instrument, as a generative system.
  • “Years” Bartholomäus Traubeck

– Kinect and processing/ Touchdesigner
– VCV Rack (software modular synthesizer)
– Research into visual-feedback audio software

Leave a Reply