Month: October 2020

Posted on

Reading Assignment 3 – “Framework for Understanding Generative Art”

Dorin, Alan, Jonathan McCabe, Jon McCormack, Gordon Monro, and Mitchell Whitelaw. “A framework for understanding generative art.” Digital Creativity 23, no. 3-4 (2012): 239-259.

Aim of the Paper 

In the process of analysing and categorising generative artworks, the critical  structures of traditional art do not seem to be applicable to “process based works”. The authors of the paper devised a new framework to deconstruct and classify generative systems by their components and their characteristics. By breaking down the generative processes into defining components – ‘entities (initialisation, termination)’, ‘processes’, ‘environmental interaction’ and ‘sensory outcomes’ – we are able to critically characterise and compare generative artworks which underlying generative processes, as compared to outcomes, hold points of similarity.

The paper first looked at different attempts at and previous approaches of classifying generative art. By highlighting the ‘process taxonomies’ of different disciplines which adopt the “perspective of processes”, the authors used a ‘reductive approach’ to direct their own framework for the field of generative art in particular. Generative perspectives and paradigms began to emerge in the various seemingly unrelated disciplines, such as biology, kinetic art and time-based arts and computer science, adopting algorithmic processes or parametric strategies to generate actions or outcomes. Previous studies explored specific criterions of emerging generative systems by “employing a hierarchy, … simultaneously facilitate high-level and low-level descriptions, thereby allowing for recognition of abstract similarities and differentiation between a variety of specific patterns” (p. 6, para.4). In developing the critical framework for generative art, the authors took into consideration of the “natural ontology” of the work, selecting a level of description that is appropriate for the nature. Adopting “natural language descriptions and definitions”, the framework aims to serve as a way to systematically organise and describe a range of creative works based on their generativity.

Characteristics of ‘Generative Art System’ 

Generative art systems can be broken down into four
(seemingly) chronological components – Entities, Processes, Environmental Interaction and Sensory Outcomes. As generative art are not characterised by the mediums of their outcomes, the structures of comparison lie in the approach and construction of the system.

All generative systems contain independent actors ‘Entities’ whose behaviour is mostly dependent on the mechanism of change ‘Processes’ designed by the artist. The behaviours of the entities, in digital or physical forms, may be autonomous to a certain extent decided by the artist and determined by their own properties. For example, Sandbox (2009)  by artist couple Erwin Driessens and Maria Verstappen, is a diorama of a terrain of sand is continuously manipulating by a software system that controls the wind. The paper highlight how each grain of sand can be considered the primary entities in this generative system and how the system behaves as a whole is dependent on the physical properties of the material itself. The choice of entity would have an effect on the system, such as in this particular work where the properties of sand (position, velocity, mass and friction) would have an effect on the behaviour of the system. I think the nature of the chosen entities of a system is important factor, especially when it comes to generative artworks that use physical materials.

The entities and algorithms of change upon them also exist within a “wider environment from which they may draw information or input upon which to act” (p. 10, para. 3). The information flow between the generation processes and their operating environment can be classified as ‘Environmental Interaction’, where incoming information from external factors (human interaction or artist manipulation) can set or change the parameters during execution which leads to different sets of outcomes. These interactions can be characterised by “their frequency (how often they occur), range (the range of possible interactions or amount of information conveyed) and significance (the impact of the information acquired from the interaction of the generative system)”. The framework also classifies interactions as “higher-order” when it involves the artist or designer in the work, where he can manipulate the results of the system through the intermediate generative process or adjusting the parameters or the system itself in real-time, “based on ongoing observation and evaluation of its outputs”. The higher order interactions are made based on feedback of the generated results, which hold similarities to machine learning techniques or self-informing systems. This process results in changes to its entities, interactions and outcomes and can be characterised as “filtering”. This high order interaction is prevalent in the activity of live-coding, relevant to audio generation softwares which supports live coding such as SuperCollider, Sonic Pi, etc, where performance/ outcome tweaking is the main creative input.

The last component of generative art systems is the ‘Sensory outcomes’ and they can be evaluated based on “their relationship to perception, process and entities.” The generated outcomes could be perceived sensorially or interpreted cognitively as they are produced in different static or time-based forms (visual, sonic, musical, literary, sculptural etc). When the outcomes seems unclassifiable, they can be made sense of through a process of mapping where the artist decides on how the entities and processes of the system can be transformed into “perceptible outcomes”. “A natural mapping is one where the structure of entities, process and outcome are closely aligned.”

Case studies of generative artworks

The Great Learning, Paragraph 7 – Cornelius Cardew (1971)

Paragraph 7 is a self-organising choral work performed using a written “score” of instructions. The “agent-based, distributed model of self-organisation” produces musically varying outcomes within the same recognisable system, while it is dependent on human entities and there is room for interpretation/ error in the instructions, similarities with Reynolds’ flocking system can be observed.

Tree Drawings – Tim Knowles (2005)

Using natural phenomena and materials of ‘nature’, the movement of wind-blown branches to create drawings on canvas. The found process of natural wind to be used as the generator of movement of the entities highlights the point of the effect of physical properties of chosen materials and of the environment. “The resilience of the timber, the weight and other physical properties of the branch have significant effect on the drawings produced. Different species of tree produce visually discernible drawings.”

The element of surprise is included in the work, where the system is highly autonomous, where the artist involvement includes the choice of location and trees as well as the duration. It brings to mind the concept of “agency” in art, is agency still relevant in producing outcomes in generative art systems? Or is there a shift in the role of the artist when it comes to generative art?

 

Tree Drawings, Tim Knowles

The Framework on my Generative Artwork ‘SOUNDS OF STONE’ 

Point-cloud visualisation system for “Stone”

Visual system:

Work Details
‘Sounds of Stone’ (2020) – Generative visual and audio system

Entities
Visual: Stones, Points
Audio: Stones, Data-points, Virtual synthesizers

Initialisation// Termination:
Initialisation and termination determined by human interaction (by placing and removing a stone within the boundary of the system)

Processes
Visual and sound states change through placement and movement of stones
Each ‘stone’ entity performs a sound, where each sound corresponds to its visual texture (Artist-defined process)
Combination of outcomes depending on the number of entities is in the system
“Live” where artist or performer or audience can manipulate the outcome after listening/ observing the generated sound and visuals.

Environmental Interaction
Room acoustics
Human interaction, behaviours of the participants
Lighting

Sensory Outcomes
Real-time/ live generation of sound and visuals
Audience-defined mapping

As the work is still in progress, I cannot evaluate the sensory outcomes of the work at this point. According to the classic features of generative systems used to evaluate Paragraph 7 (performative instructional piece) such as “emergent phenomena, self-organisation, attractor states and stochastic variation in their performances”, I predict that the sound compositions of “Sound of Stones” will go from being self-organised to chaotic as the participants spend more time within the system. Existing as a generative tool or instrumental system, I predict that there will be time-based familiarity with the audio generation with audience interaction. With ‘higher order’ interactions, the audience will intuitively be able to generate ‘musical’ outcomes, converting noise into perceptible rhythms and combinations of sounds.

Posted on

Semester Project – ‘STONE’ update 27/10/20

logistics

date: 10 Nov, Week 13
location: Truss room/ LT (Mounting of the kinect/ camera system)

Week 11 – Process and Developing instructions
Week 12 – Set up and construction
Week 13 – Presentation

PROCESS

Human body position
Aim – To find out which positions the body should stay in and the duration to induce a sense of stiffness / being compressed and weighted
Instructions:
i. Use your body to imitate the form of the stone

Generative art – Sonify textures/ each stone

ii. Experiment with different body positions for different time periods
– Different sitting and crouching positions
– Duration of 2, 5 and 10 minutes

Sketch of possible body positions
more sketches
Instruction for ‘One Minute Sculpture’, Erwin Wurm
Instruction for Psycho I, One minute sculpture, Erwin Wurm

2. Philosophy of zen gardens + the making

Zen Buddhism
japanese rock gardens – “were intended to imitate the intimate essence of nature, not its actual appearance, and to serve as an aid to meditation about the true meaning of existence.”

Selection and Arrangement of rocks

the most important part is the selection and placement of rocks.
The Sakuteiki “Records of Garden Making” – manual on “setting stones” (ishi wo tateru koto). “In Japanese gardening, rocks are classified as either tall vertical, low vertical, arching, reclining, or flat. For creating “mountains”, usually igneous volcanic rocks, rugged mountain rocks with sharp edges, are used. Smooth, rounded sedimentary rocks are used for the borders of gravel “rivers” or “seashores.”

In Japanese gardens, individual rocks rarely play the starring role; the emphasis is upon the harmony of the composition.”

– Arranging rocks according to the Sakuteiki

Make sure that all the stones, right down to the front of the arrangement, are placed with their best sides showing. If a stone has an ugly-looking top you should place it so as to give prominence to its side. Even if this means it has to lean at a considerable angle, no one will notice. There should always be more horizontal than vertical stones. If there are “running away” stones there must be “chasing” stones. If there are “leaning” stones, there must be “supporting” stones.- Sand and gravel – The act of raking the gravel into a pattern recalling waves or rippling water

Zen priests practice this raking also to help their concentration
> might omit this aspect in the project
> alternatively, I can generate “white noise” as background for the audio

MAIN EXPLORATION: Arrangement of OBJECTS vs HUMANS

Playing around with the landscaping of stones

Arrangement of humans in current times

Social distancing at the Hajj
Social distancing concert in the UK

 

3. Textures – Wearable as embodiment/ ergonomics/ texture of form

Technical Apparel
Materials to endure nature/ harsh conditions

Cambre Jacket — Made from a durable performance denim with quick drying properties.

Ergonomic 3-Dimensional Patterning. Working with E3D in CLO3D has allowed our designers to experiment at a level that wouldn’t be possible in a more traditional environment, and maintain complete ownership through the ideation to manufacturing process.
  • Another layer to explore – embodiment with apparel to create visual texture
    Volcanic rocks vs. smooth rocks (no apparel)
    –  Gorotex
    – Ponchos

SET UP – FIRST DRAFT

PROGRESS
– Working on sonification system (generative art) currently
– Experiment with stone positions and wearable textures
– Aim to develop instructions for performance by the end of the week (OSS)

//

Malin Kundang folklore about an ungrateful son turning into stone
Posted on

FYP 20/21 Week 10 Updates and direction for Week 14

Week 10:

SENSORY EXPERIMENTS – Perceptual box

I built the perceptual box 🙂 I can now conduct my visual-tactile experiments.

Some Visual Experiments:

I experimented with two visual qualities of light that can be experienced visually in my perceptual box.

Some tests:

 

A. RHYTHM

Playing with the ‘blinking’ LED to create rhythm with light
> Varying the interval delay – Create different rhythms

B. MOTION
Using a servo motor to create a rotating wheel to vary the light source.

‘Sensory Modality Transition’ Experiments – Visual-Tactile  (Week 12)

Aim: Associate tactile qualities with Visual Qualities of light

CONSONANCE and DISSONANCE

  • Combine textures with the perceived/ disconnected tactile quality of light
    eg. wet texture to some form of light
  • Using the experimental framework: three perceptual mode conditions
    Visual expectation (V), Touch alone (T) and Touch following a visual expectation (VT)
  • To evaluate:
    Perception of disconfirmation – difference between VT and V
    Expectation effect – difference between VT and T
    Perceptual incongruence – different between V and T

 

Light Textures Classification (Week 11-12)

Referencing ‘Sound Textures Classification’ http://alumni.media.mit.edu/~nsa/SoundTextures/

Human Perception of Sound Textures
– Perception experiments which show that people can compare sound textures, although they lack the vocabulary to express formally their perception

  • the subjects share some groupings of sounds textures
  • 2 major ways used to discriminate and group textures: The characteristics of the sound (periodic, random, smooth) and the assumed source of the sound (voices, water, machine)

Texture or not?

Perceived Characteristics and Parameters:
Come up with possible characteristics and qualifiers of sound textures, and properties that determine if a sound can be called a texture.

CAN I DO THE SAME FOR HUMAN PERCEPTION FOR LIGHT TEXTURES?
> Classify light textures
> Perceived Characteristics and Parameters – Quantifiers and Qualifiers

ISEA DATA gloves workshop – Hugo Escalpalo 

Input gloves
– Measures specific finger anatomic and gestural positions as input
– Haptic feedback: Not as output. The design of the glove has in-built mechanism to incorporate proprietary feedback using Slider Sensors and Springs. When pulled forward, the gloves use resistance to simulate the feeling of grabbing an object.
– Intended to include tracking components to track position in VR

https://www.youtube.com/watch?v=MP9aJ5ThwK0

https://www.youtube.com/watch?v=Be3NTYqN0vA

Week 12-13:

  • Come up with a lo-fi prototype for gloves that does not require tech, Using springs to trigger a sense of haptic touch based on gestural and action-based movements (articulation)
  • Weight based simulation – Using liquid and hand position to simulate a sense
  • Output gloves – Vibration (fingers and palms), Pressure (pull-back action using servo-motors), Weight, Proprietary position or feedback

Direction for Week 14:

  1. LIGHT TEXTURES CLASSIFICATION – create framework for perceptual experience and understanding
  2. GLOVE (low-tech/ haptic prototype) – design to impart a sense of touch of solid objects
  3. SENSORY DISSOCIATION/ ASSOCIATION EXPERIMENTS with light textures
Posted on

Generative Artwork – Approaches for sonic synthesis from visual data

Using the sound generation system I used in my generative sketch:

MediaMob – Heliostat Field

RECAP for Generative Artwork:

Aiming to extract a sound from the visual textures of stone involves two aspects:
1. Subtracting three-dimensional forms of the material into visual data that can be converted into audio/ sounds
2. Designing a model that generates sounds that the audience would perceive/ associate the tactile qualities of the specific material.

POSSIBLE APPROACH

  1. Obtaining visual data from camera (kinect)
    – Depth value, x, y
    – 3D scanning – line/ wave data
    – Breaking 3D forms into grains
  2. Visual textures to Sound – WaveTable Synthesis
    – Ableton Live
  3. Granular Synthesis – Complex sonic textures
  4. Real-time generation of sound – OSC SuperCollider
    i. When the depth value hits a certain value, the wavetable will be triggered.
    ii. Granular synthesis using spatial data of the object
    iii. Using data points in 3D space, when it is triggered, a certain sound/ virtual synthesizer is played.

DIRECTION

  • Studying ‘Classifications of sound textures’ to design sounds that relate to tactile qualities
  • Granular Synthesis tutorial
  • Feedback from guest lecture
Posted on

Semester Project – ‘STONE’ research and update

Semester Project Proposal

https://docs.google.com/presentation/d/1tElhHiD0AllnlLwou-esg3JFUDa_oYsuHohpyuGDFb4/edit?usp=sharing

Update:

Sonic Development (Generative Art):

Generative Artwork – Stone

Materiality and Space (Performance and Interaction):

‘STONE’ AND ‘SPACE’
1. Arrangement of the human body

How it relates to an experience – eg. staying in a certain body position for a duration of time and what it feels like

Ergonomics 

Human condition 

Physical, Spiritual, Psychological and the Political

“whole entity of a human being: the physical, the spiritual, the psychological and the political” Erwin Wurm in The Artist Who Swallowed the World

Erwin Wurm ‘Disruption’ 2015

2. Arranging of things/ bodies in space

MAIN EXPLORATION: Arrangement of OBJECTS vs HUMANS

Zen Garden 

How are Zen Gardens made and their philosophy
The arrangement of elements and space – and their effect on experience

Human Geography/ Humanistic geography 

Yi-Fu Tian

Human geography – is the branch of geography that deals with humans and their communities, cultures, economies and interactions with the environment by studying their relations with and across locations. It analyses patterns of human social interaction, their interactions with the environment and their spatial interdependencies by application of qualitative and quantitative research methods.

‘Space and Place’ – The Perspective of Experience

Project: Exploring the Arrangement of Humans as Objects (stone)

Posted on

FYP 20/21 Week 8 Updates

SENSORY EXPERIMENTS

Perceptual box

I am still in the midst of designing my “perceptual box” and fine tuning the details of my sensory experiments.

Referencing:

Yanagisawa, H., & Takatsuji, K. (2015). Effects of visual expectation on perceived tactile perception: An evaluation method of surface texture with expectation effect. International Journal of Design, 9(1).

Surface Texture

  • People perceive and/or predict a surface’s characteristics corresponding to each physical attribute through sensory information, a process that we call perceived features (eg. Surface roughness perceived through touch).
  • Using a combination of perceived characteristics of surface texture, people perceive a tactile quality, such as “nice to touch”.

Sensory modalities

  • During such sensory modality transitions, we expect or predict the perpetual experience that we might have through a subsequent sensory modality by first using a prior modality, such as in the case of expecting a particular tactile perception by first looking at a surface texture.
  • On the other hand, prior expectation also affect posterior perceptual experiences – a phenomenon known as the expectation effect.
  • 3 aspects of experiment:
  • Participants were asked to evaluate the tactile quality of a target object under three perceptual mode conditions:
    Visual expectation (V), Touch alone (T) and Touch following a visual expectation (VT)
  • To evaluate:
    • Perception of disconfirmation – difference between VT and V
    • Expectation effect – difference between VT and T
    • Perceptual incongruence – different between V and T

Measurement:
Evaluate Tactile feeling using four opposite adjective pairs (“nice to touch-unpleasant to touch”, “smooth-rough”, “hard-soft” and “sticky-slippery”) – Between each adjective pair was a scale comprised of five ranges. Participants responded to each adjective scale by marking the rating on a questionnaire sheet, which employed a semantic differential (SD) scale.

SENSORY EXPERIMENTS 

PART 1
Things I want to explore when I finish constructing the perceptual box. I went back to look at my hypothesis for my project, and I would like to center my experiments through connecting the visual properties of light to the visual qualities of material, and see if a tactile quality can be associated.

  1. Playing with Rhythm
    Light quality: Rhythm?
    Visual quality: PatternTrying to connect how rhythmic/ flashing lights to a tactile effect

  2. Ryoji Ikeda – Test Pattern (2013)
    https://www.youtube.com/watch?v=XwjlYpJCBgk
  3. Playing with Colour
    Light quality: Frequency?
    Visual quality: ColourConnecting a texture (frequency) to colour

    Using touch sensors
  4. Playing with Colour
    Light quality: Intensity/ Diffusion
    Visual quality: Opacity/ TranslucencyTrying to connect force with sharpness of light

 

 

Slide potentiometer

PART 2

MATERIAL PERCEPTION – Can we perceive/ infer material based on visual results of reflected light?

  • Playing around with material’s interaction with light, if light can be given visual forms/ textures
  • Recreate the textures/ find tactile similarities with physical materials

Build a hologram in my perceptual box:

https://mashable.com/2016/10/24/holovect-3d-projections-star-wars/

 

Interesting Light phenomenon/ visual textures I noticed:

Ryoji Ikeda – Spectra (2014)

 

Reminds me of:

Interpretive flare display of unthought thoughts (2020) neugerriemschneider, Berlin Photo: Jens Ziehe

https://olafureliasson.net/archive/artwork/WEK110941/interpretive-flare-display-of-unthought-thoughts

 

‘Touch’ in Art – What would trigger touch?

Last week, it was mentioned that I could look into works that trigger touch, but I missed out on the example that was given.

Vocab? – To find consensus between descriptions of how light is subjectively perceived

ISEA 2020 – Why Sentience?

DATA GLOVES Workshop 17-18 October

The “Data Gloves” were developed with the purpose of interacting with the VR environment “Human After”, a piece by Anni Garza Lau. Under the problem of the high cost of a set of commercial gloves, we realized that we had the ability to manufacture a pair of gloves with a very detailed ability to acquire information about the position of the fingers for a fraction of the price.

https://www.radiancevr.co/artists/anni-garza-lau/

I am not sure how useful the gloves would be in terms of simulating haptic qualities, instead as a glove to obtain data but we will see how it goes.

Posted on

Creative Industry Report – House of Light by James Turrell

The House of Light, by James Turrell, is located in a hillside forest in Tokamachi, Niigata, Japan. House of Light exists as an experimental work of art that also serves as a guesthouse, where visitors can experience the light with their entire bodies during their stay. Using light as the main medium, which is key in his perceptual works, Turrell combined the intimate light of a traditional Japanese house with his works of light. Both time-specific and site-specific, “House of Light” serves as an experience of living in a Turrell work overnight.

“Outside In” by James Turrell
Image from: https://www.gadventures.com/blog/japan-house-of-light-james-turrell/

For Turrell who has been searching for and exploring the “perception of light”, House of Light is his attempt to contrast and merge day and night, East and West, Tradition and Modernity. One of the two works that can be experienced in the House of Light is “Outside In”, which is a tatami room with a ceiling that can be exposed to the sky using a sliding panel on the roof. As of Turrell’s Skyspaces, which are constructed chambers with an aperture exposing the sky, the work “Outside In” intended for the viewers to be able to “live in the light”.

“Light Bath” by James Turrell

Another work that can be experienced in House of Light is “Light bath”, which explores light in an indoor space, experienced by those who stay overnight at the House of Light. Using optical fibres than run through the bathroom, while being immersed in the tub, the viewer can experience the interplay of light and shadow against the water.

“House of Light” by James Turrell

The House of Light exists in itself as a work of art and also as an accommodation facility where you can interact with the art spaces. Turrell established the terms of the experience where he wanted the guests to interact and discuss their thoughts overnight as they share the world of light and shadow. As a fully functioning guesthouse, it charges 4000 yen (per person, plus a facility charge of 20000 yen for the building, which is divided evenly between guests.

I think it is intriguing that a work of art can function as a business establishment that sells the time-based experience of art. It alters the conventional way of interacting with an artwork, going beyond a short-term immersion in space. Relating to the concept of ‘experience’, the article “Welcome to the Experience economy” explores the emerging fourth economic offering of experience. As consumers unquestionably desire experiences, more and more businesses are responding by explicitly designing and promoting them. 

House of Light serves as an example of this direct relationship, where guests pay for the ephemeral experience. Considering a more philosophical aspect of the experiential perspective, which involves sensation, perception and cognition, how can I apply the novelty of sensory and perceptual experiences to the real world? The idea of using ‘place’ and giving it an experiential perspective seems to have a viable market, but it should be created for a targeted group that understands the work and would like to experience the art as it is intended. 

References

Pine, B. J., & Gilmore, J. H. (1998). Welcome to the experience economy. Harvard business review, 76, 97-105.

Tuan, Y. F. (1977). Space and place: The perspective of experience. U of Minnesota Press.

 

Posted on

Generative Artwork – Stone

CONCEPT 

How do materials sound like? While sound is subjectively perceived, there is a collective or general sentiment on the descriptive qualities of a sound/ soundscape. We occasionally use tactile qualities (eg. rough, soft) to describe sounds, to assign textures to what we hear. In music, texture is the way the tempo, melodic and harmonic materials are combined in a composition, thus determining the overall quality of the sound in a piece. However, most sounds that we hear are more complex than simple harmonies, there is a more complicated process behind how we can perceive and cognitively recognised textures in sound. My project explores how visual textures of physical materials (stones) can be translated to the auditory, creating an interactive system to draw out the direct link between sound and textures.

CASE STUDY – Classification of Sound Textures

Saint-Arnaud, N. (1995). Classification of sound textures (Doctoral dissertation, Massachusetts Institute of Technology).

http://alumni.media.mit.edu/~nsa/SoundTextures/

“The sound of rain, or crowds, or even copy machines all have a distinct temporal pattern which is best described as a sound texture.” (Saint-Arnaud, 1995). Sounds with such constant long term characteristics are classified as Sound Textures. The thesis investigates the aspects of the human perception and machine processing of sound textures.

“Between 1993 and 1995, I studied sound textures at the MIT Media Lab. First I explored sound textures from a human perception point of view, performing similarity and grouping experiments, and looking at the features used by subjects to compare sound textures.

Second, from a machine point of view, I developed a restricted model of sound textures as two-level phenomena: simple sound elements called atoms form the low level, and the distribution and arrangement of atoms form the high level. After extracting the sound atoms from a texture, my system used a cluster-based probability model to characterize the high level of sound textures. The model is then used to classify and resynthesize textures. Finally, I draw parallels with the perceptual features of sound textures explored earlier, and with visual textures.”

His approach to sound textures from both the perspectives of subjective human perception and technical synthesis and processing of sound textures is especially relevant to my project. The difference is that the thesis explores the perceptual and machine constructs of existing sound textures, while I am trying to generate sounds that could be perceived/ associated with the actual tactile qualities of a material. I could say that my process (associating texture to sound) works in the opposite direction from his (associating sound to texture).

Aiming to extract a sound from the visual textures of stone involves two aspects:
1. Subtracting three-dimensional forms of the material into visual data that can be converted into audio/ sounds
2. Designing a model that generates sounds that the audience would perceive/ associate the tactile qualities of the specific material.

PROCESS

While his project is conducted on a large scale with the abundance of time and resources (MIT lab), it was conducted a while back (1993-1995). With the introduction of many audio-generation software and sound visualisation/ analysis technologies, my project might be feasible in the short span of time that we have (6 more weeks to Week 13).

Visual textures to Sound – WaveTable Synthesis 

I previously explored wavetable synthesis on SuperCollider (one wavetable to another), but I was not able to visualise a combined three-dimensional model of the periodic waveforms with the sound produced (only singular wave forms). For me to design a model for sound textures, I would need to be able to connect the sound I hear to 3D wavetable. I would have to experiment more with sound synthesis to decide on the type and range of sounds I would like to generate. Based on the above paper, I would look at perceptual features of sound textures to formulate how I could associate a tactile quality/ mood to what we hear.

Ableton Live 10: Sound Design with Wavetable

Obtaining visual data/ 2D forms from 3D material – 3D scanning and processing 

To be able to generate data in real-time based on the stone used and its movement, I would need a software to process the captured data (RGB data/ 3D scanning) by the kinect/ camera.

Possible software: 3D Scanner Windows 10 App, OnScale, CocaoKinect

Kinect 3D scanning in OSX with CocoaKinect

http://mskaysartworld.weebly.com/modeling-and-animation-basics.html
https://www.extremetech.com/extreme/236311-researchers-discover-how-to-shape-sound-in-3d

Researchers discover how to shape sound in 3D

I would have to obtain the 2D line vectors/ waves from a 3D scan and use them to compose a 3D wavetable to be played ideally in real-time.

 

Connecting Both – Open Sound Control (OSC)

I have previously worked with OSC to connect processing to SuperCollider, I will continue working on it. I would also like to find a way to design a more complex soundscape on SuperCollider by either inputting the designed wavetable synthesis or individually manipulating the way the sound is played for each sound (pitch, amp, freq, etc).

If the wavetable does not work out:

I would then use data values instead of visualised lines. Referencing Aire by Interspecifics, I would assign individual data line/points in a 3D space to specific designed synthesizers programmed on SuperCollider. Depending on the 3D forms thus there would be varying depth values on the data line, different sounds would be produced and played in consonance/ dissonance.

I would still have to study the soundscape design of Aire on their github to learn more about sound design and layering: https://github.com/interspecifics/Aire

SURPRISE

The concept of surprise is explored in my project through the dissonance between our perceptive modalities, where we usually generalise that they are mutually exclusive when we know they are not. Connecting the links between what we see, hear and touch, I am trying to touch upon the unnoticed relationships between our senses and try to play around with how our minds cognitively make sense of the world.

With sound as an intangible material, we don’t usually connect what we physically see to what we hear. When an instrument is played (eg. a key is pressed, a string is struck), the act serves as a trigger but it is not the only component to the sound we hear. Being human, we can’t fully understand how a sound is visually and technically synthesised beyond what we hear. With technology, we are able to work with the other end of the spectrum of generating sound (where we work from the visual/ components of sound).

Working with textures, I want to associate tactile quality to sound to enhance the experience with a material and its materiality. Stones are intriguing in themselves, even though seemingly fixed, their forms are intrinsically beautiful. Connecting their visual textures to sound seems like a way to explore their spirit.

TIMELINE

Week 9-10: Work on Sound/ wavetable synthesis + Obtaining visual data
Week 11-12: Design soundscape and interaction with stones
Week 13: Set-up

Posted on

MediaMob – Heliostat Field

Heliostat Field

Using the concept of a heliostat, my flashmob performance explores the idea of tracking the sun’s movement. A heliostat is a device that includes a mirror, which moves to constantly reflect sunlight towards a predetermined target, compensating for the sun’s apparent motions in the sky. The target can be a physical object, distant from the heliostat or a direction in space.

A heliostat field is a solar thermal power plant using computer controlled heliostat systems using data that tracks the sun position to focus the sun’s rays onto a special receiver at the top of the tower. The concentrated solar energy will be used to create steam that will produce electricity.

 

Heliostat SA

Initial Ideation

Building upon my InstructionArt project ‘Choreographic Light’, which uses artificial light to generate movement, I want to explore the movement and reflected position of natural light. My initial idea was to use the behaviours of reflected light (from the sun or a light source) to create ‘light’ drawings onto a surface, where the performers behave as ‘human’ heliostats by moving or flipping a mirrored surface to the scripted direction.

Initial idea

However, for the FlashMob project, I wanted to involve more of the body and bigger movements (ironically, as heliostats can’t move) to create a less static performance. Additionally, due to the restriction in the number of people and material, I was not sure if I could generate an obvious enough effect (of light).

Performance – Heliostat Field

Instead of generating light as an output, I decided on generating sound as a “predetermined target” according to movement and position of the human heliostats. This is also in line with my semester project of generating sound using visual textures.

Moving on to a more conceptual understanding of a heliostat, the performers would still be instructed to track the sun’s movement but in a more abstract manner. Using a website that provides real-time data on the sun’s direction and altitude specific to Singapore (https://www.timeanddate.com/sun/singapore/singapore?month=12), I would create a score for the participant’s movement according to the live changes in the sun’s movement.

Parameters of Performative Space

 

Sun-path chart, equidistant projection, generated by Sun-path Chart Software Courtesy: University of Oregon
This solargraph exposed over the course of a year shows the Sun’s paths of diurnal motion, as viewed from Budapest in 2014. Courtesy: Elekes Andor

Referencing the sun-path chart, the course of the sun movement relative to a location is in the form of a convex lines that varies outwards per time frame (a day). As heliostats, the performers would travel along the paths of the “sun” relative to the position of a specific “location”, which is the kinect connected to the computer to generate sound.

Using string and tape, I mapped out four curves of 10 points around the kinect. The performers would use the points of the web as parameters to navigate the space.

It’s interesting how the shape of the sun paths holds some similarity to the heliostat field.

Increased the sun map to four rows

Score for movement 

Real time data of sun direction and altitude

The participants will be instructed to stand in different points and directions in front of a kinect that is connected to a sound generating system that I have been working on on Supercollider and Processing. According to the changes in real-time data of the sun, the participants have to move according to the score of my designation.

Focusing on the sun direction data, a value and direction is given. Relative the original position (north) of the participant, he/ she has to rotate to face the arrow of the sun direction. The numerical data of both the sun direction and sun altitude changes. Whenever there is a change in values on the screen, the participant moves forward.

Given Instructions

  1. Stand at any point on the map, facing different directions
  2. Use your phone to access website (https://www.timeanddate.com/sun/singapore/singapore?month=10&year=2020)
  3. Rotate your body according the the arrow relative to where you are facing
  4. Move according to the score when the values change.

FlashMob Outcome

The kinect maps the depth positions of the ‘human heliostats’, and plays a sound  whenever a certain depth is reached, generating a  soundscape.
See more on my process of generating sound using real-life movement: (https://oss.adm.ntu.edu.sg/a170141/week-7-generative-sketch/)

The kinect, depending on its position, only tracks the depth and movement of a limited range. Referencing how the sun can only be seen in the day, the kinect does not capture the entire performative area, but the system only interacts when the performers move in front of its range. Whenever the performers enters the captured range of the kinect, a sound would be played according to the change in depth value.

To prevent the sound from cluttering (too many nodes), I reduced the skip value (originally 20) in processing to 60 so that the depth value is extracted for every 60 pixels instead of 20.

For sound generation: Whenever the b value (brightness/depth) obtained is more than a certain value (>140) meaning that forms are captured within the range of the kinect, a message is recorded in Processing. I connected processing with SuperCollider that I could design sounds using the (Synthdef function). Connecting processing to SuperCollider using an external NetAddress, the sound is played whenever the message is sent.

Further developments

In the future, I can consider designing more specific sound and using more data points from the kinect to generate a greater range of sound. I tried using x and y values from the kinect and mapping it to two different sounds but SuperCollider could not handle the overwhelming influx of information and the soundscape was very cluttered and laggy.

I could also vary the sound according the the depth, to map the sound to the individual performers. I would also like to expand the scale of the performative space in a public space and increase the number of participants.

FINAL PERFORMANCE

‘Heliostat Field (2020)
Interactive Performance by Alina Ling

Performed by Jake Tan and Nasya Goh (Thank you! :-))

More Documentation: