Rethinking Multimedia

It is often common for one to assume the meaning of certain words and phrases; especially if said word or phrase is common place in one’s daily life.

Navigating modern music libraries become a multimedia experience in and of themselves.

Such is the case with the term, multimedia. In this digital day and age, the word multimedia is thrown about rather flippantly. We have come to assume that all media is multimedia – no form of artistic expression really exists by itself anymore. Even something like assumably “single-media” like listening to music has become an exercise in multimedia consumption. The digital consumption of audio puts in a situation where we first interact with our digital “libraries” sorted by any number of categories from albums to playlists. We sift through the lists of words and images before arriving at our music. Even then we are likely presented with the album art of the track with all its subtext and suggestions as the music begins to play. The experience is never purely auditory; perhaps even a little distracted.

With the advent of video sharing services like YouTube, the act of music consumption if further skewed towards a multimedia experience with users likely engaging with music videos as opposed to a purely auditory soundtrack. These videos augment the experience of listening to a track by providing visuals that create a richer experience that would otherwise be unattainable with just one type of media. While some may embrace the expansion of the original intended experience of the media, others yet may lament the added distraction muddying the original intended experience.

Coldplay’s music video for “Hymn for The Weekend” illustrates the vibrant Holi festival in India, creating a stunning visual that is otherwise impossible to create using just the media of music.

No matter one’s stance, multimedia as a term then, could be defined as the totalizing sensory experience gained from combining more than one method of expression to create a sum that is arguably greater than its parts. This concept of totalizing is discussed in the essay as the Gesamtkuntswerk. Attributed to Richard Wagner, the Gesamkuntswerk or “Total Artwork” is Wagner’s idealized union of all the various aspects of creative expression; the combination and unification of all the arts to create an experience that could capture the essence of human experience.

This concept and definition was of particular interest to me as it reinforced a personal believe that I always maintained about the term multimedia. While it may have only be in prevalent use over the last 20 to 30 years and heavily tied with technology, I have always held onto the notion that multimedia was far from a recent concept. Like the caves of Lascaux, examples of multimedia experiences predate many of humankind’s technological advances. However, as the reading suggests, it is with the advent of technology (particularly the personal computer) that artists and technologist alike have been able to fully bring to fruition the Wagner’s vision. The concept of multimedia as a meta-media (a medium comprising other media) would not have been able to be as successful if not for the efforts of technologists like Vennevar Bush and Tim Berners Lee. Our modern understanding of multimedia is predicated on the works of these pioneers.

What was particularly striking to me as well was how many of the speculations made by artists and technologists in the essay have since come to be reality. For example, Weiner’s preoccupation with human-computer relationships in his study of cybernetics led him to the conclusion that “the quality of our communication with machines effects the quality of our inner lives”. Today, we have entire professions built around this notion – user experience design. The rise of social media and the significant social impact that our relationship with our devices have greatly changed how me consume media and perceive the world. These “machine-mediated” experience of social interactions and our experience of the world around us have the power to shape and affect how we think and perceive. Multimedia then, has become not just a tool for artists to create better modes of creative expressions, but it has also evolved to place greater power in the hands of creatives to influence audience reactions and the outcome of a work.

Another arresting prediction made by Scott Fischer was the vision of a immersive virtual space that would “give birth to a new form of participatory, interactive electronic theatre.” With recent advances in the field of Virtual and Augmented Reality, VR has fast become a technology that is getting more and more accessible to the general public. Platforms such as the HTC Vive have given birth to online shared spaces where users can engage in telematic interactions with others across vast distances. With full body tracking, voice chat and surround audio, people now have access to fully immersive environments where they are able to take on any likeness and interact with others as if in the same space.

VRChat and the Ugandan Knuckles meme phenomena is one such example of Fischer’s vision of a participatory electronic theatre coming to life. The platform was original envisioned as a space for users to interact with one another in an attempt to escape the mundane realities of regular life, similar to earlier concepts such as Second Life. However, the richer interactive experiences afforded to users soon saw players creating personas, characters, stories and lores in a machine-mediated multi media space. This types of ad-hoc, participatory theatre would be difficult to recreate without the aid of multimedia.

Multimedia over the years has come to represent more than just the nexus point of a host of various creative expressions. We have been able to create, richer, deeper and more all encompassing experiences that challenge both creators and audiences alike. The digital revolution has allowed for the genesis of type of media that truly allows for us to express ourselves the way our minds work. While some may argue that technology and multimedia have worked against us by making us more distracted individuals, perhaps, this was the intended result after all? As Ted Nelson philosophized, perhaps multimedia, like all human inventions, is an extension of ourselves. Perhaps the burden of responsible consumption sits of our shoulders after all.

“I See You” – An FYP Concept

The following is an elaboration of one of the perviously discussed FYP ideas that I am most interested in and confident of executing over the period of my final year in ADM.

1. Project Description

Overview

I See You is an interactive installation that aims to raise awareness about surveillance, spying, and stalking in a rapidly digitized world. With the rapid rise of ‘smart nations’ and connected cities world-wide, how much are we willing to give up in exchange for a more modern and convenient environment. Are we exchanging our privacy for a cause that is much greater than us, and how much of it is truly beneficial to society? This work aims to raise the dystopian question on whether the advancements of technologies and the convenience that comes with it are worth the trade-off in the civil liberties of the privacy of an individual. Do we, the Snapchat and Instagram generation, live life too lose and dangerously in an age where a simple username is capable of revealing whole identities to strangers both online and offline.

Concept

Visitors will be tracked using video analytics during their visit to the FYP show with hidden cameras. Using facial recognition algorithms to trawl through social media networks, visitors to the installation are then presented with their own social media feeds and online interaction. The installation would be in an enclosed space, so as not to ruin the element of surprise for the visitors. The audience will also be able to look through the publicly available social media interactions of other visitors that have passed through the FYP show’s exhibition space.

Installation Specifications

The installation would ideally be placed in an enclosed space such as the Truss Room. The three projection screens as well as the space in the middle would be ideal for recreating what I envision for the installation. The projections will be used to show the various live video feeds with the video analytics overlay showing what is happening in real-time. Some of these facial recognition algorithms running, surveillance looking UI resembling tropes from spy movies and games, and the results of the facial recognition will also be shown in these projections, creating a security console experience for the viewers.

The aesthetics of these projections would be black and white videos with markers and indicators that are overlaid to create an uncomfortable, dystopian appearance. Some inspirations that I drew from include the video game Watch_Dogs.

The main interaction console in the centre of the room will be a large touch sensitive overhead projection of LCD display. Through this console, users will be able to access the social media feed gathered from all the visitors that have made an appearance in the show. This aims to create a sense of discomfort in the visitors to the exhibit, showcasing how easy it could be for strangers to be able to access private and intimate aspects of someone else’s life. Users will also be able to interact with the projections on the screens from this console, choosing to display items and feeds on the projections.

2. Artist references

I Know Where Your Cat Lives

I Know Where Your Cat Lives by Owen Mundy and a team of creatives is an online artwork that seeks to raise awareness about data privacy. It uses publicly sourced images of cats from social networks like Instagram and Facebook to create a realtime data visualizations of where these said cats are located. The site uses the geolocation metadata that’s embedded in the photos to visualize the exact co-ordinate of the photo on a map. Users are able to pan and zoom through the map to look at the various cats that are catalogued on the visualization.

The satirical component to this work lies with the fact that cats are unable to take pictures of themselves. As such, these geo-location co-ordinates point straight at humans that are posting such pictures, indirectly to also allow users to know where you as an internet user lives. I think this work is pretty successfully in creating a sense of unease once you begin to realize what is happening behind the scenes. I would like to create a similar sense of unease that requires a certain degree of interaction from the viewers. In other words, I would not want the fact that I am exposing people’s personal information that they have willingly provided, to be apparent right from the beginning.

Border City

Border City is an installation by Natasha Jen and her team from Pentagram. It was made for the Mexican Pavilion for the London Design Biennale. The work features infographics and animations projected onto a panoramic wall that circles the entire exhibition space. It also features a central console in the centre of the room. While not interactive, I really love the layout and presentation of information, and the use of space within this installation. I want to create a similar all encompassing effect for my own work as well. This is a huge reason as to why I think the Truss is an ideal location for the work that I am proposing.

Face to Facebook & fbFaces

Face to Facebook stole one million Facebook photos and, using face recognition algorithms, posted these photos to a custom made dating site where users can be matched with a potential date. The social media platforms to which we post our data on aren’t the only ones that have access to our information. Depending on our privacy settings, our photos and intimate details can be equally as exposed—available for download to just about anyone.

In the same vein as the work above, fbFaces is a web-crawler that targets public profiles for profile pictures and copies the image, Facebook ID and name. It then proceeds to do the same thing for your friends’ accounts. The artists printed out the downloaded profiles they collected and created  a wallpaper covering every surface of a large room. Profile pictures are reduced to mere pixels, as if to make them insignificant in the simplicity of their theft. They no longer represent people but information that is easily downloaded and appropriated—mere pixels.

The importance of these two works shows that what I am trying to achieve with my work is rather well within the realm of feasibility. In fact, my preliminary research into Facebook scraping and facial recognition led to a few possible algorithm models that I could emulate to create my own version for this project. However, the research also seems to suggest that I may have to alter my processes slightly and rethink how all these various components will fit together in once large installation that seems cohesive and a part of one artwork.

3. PROJECT timeline

The proposed production schedule may be found here.

4. Updates

After doing more research and sitting on the idea a little longer, I have come to realize that I would in fact need to change up the way I conduct the facial recognition process and how I obtain the faces of the visitors. I was drawn towards the concept of doxxing, which the process of using publicly available data to dig around for someone’s personal information such as bank and medical records. While the practice of doxxing is technically illegal, presenting already publicly available data is not. As such, I have begun playing around with the idea of creating another section of the installation that would be a pre-cursor to the “main room”.

I came up with the concept of a Photobooth that visitors must go through before they enter the Truss or exhibition space. This photograph of the visitor will allow me to run facial recognition in a much more controlled and systematic manner as opposed to attempting a real-time scan on the live video feeds. This would then allow me to get much more accurate results. As visitors leave the exhibit, I would ideally be able to provide a print out of what the viewer has made publicly available and that I have been able to scrap while the viewer has been going through the exhibit.

Thinking About FYP

Where to Begin?

I began the process of thinking about my FYP by undertaking some creative thinking exercises to kick-start my thought process. The first thing I did was to write down some of my skillsets and arrange them in order of confidence in my skillsets as well as how interested I was in creating those kinds of works.

This process was helpful in determining the kinds of works that I was most interested in executing. The red-line represented skills that I was slightly less confident in or types of work that I was not as interested in compared to those above the line. This process helped me narrow down the possible execution routes I could consider when I started thinking about the topics and ideas that I wanted to possibly tackle for my FYP.

The next thing I did was to list down the topics and ideas that I was personally interested and invested in, as well as issues that are hot topics within the public consciousness. (My personal ideas are on the larger slips of paper). The final step was to combine different slips of paper to see how these execution methods and topics could lead to interesting lines of questioning or idea projects that I could continue to develop into an FYP. That being said, here are 3 possible initial ideas that I have come up with.

1. I See You

Topic: Digital Surveillance | Execution: Installation

2. The Sound of the Skies

Topic: Weather & Climate Change | Execution: Data driven Audio & Visualization

3. City Movement

Topic: Transportation | Execution: Data driven interactive visualization

I will continue to elaborate on each of the ideas in the following posts.

 

In the Pursuit of Happiness

While I have been down to TeamLab’s Future World exhibition many times before, this was the first time that I was there guided by someone actually from TeamLab. This really changed the experience of Future World for me and brought closure to the many unanswered questions that have been sitting idle in my mind from all the previous visits.

Continue reading “In the Pursuit of Happiness”

Smart Nation Intervention: Data-driven Public Art

One of the key missions of the Smart Nation Initiative is to get the private sector in Singapore to work alongside the public sector to create a nation that is informed, driven, and made better by technology. The initiative aims to use technology to provide citizens with better services and an all round better quality of life. Continue reading “Smart Nation Intervention: Data-driven Public Art”

Thoughtful Interaction Design

In this day an age, more than ever, thoughtful interaction design has become a pertinent concern to the everyday man. More and more, systems and services are being digitized and the careful construction of an interface around these products can determine if someone will accomplish their goals, or cry out in frustration. Continue reading “Thoughtful Interaction Design”

Works That Inspire Me

As a media artist and designer, I am heavily inspired and influenced by other creators in the field. I am particularly interested in generative art and particle systems. An artist that has been influencing me of late is Refik Anadol. I like his use of minimal, geometric motifs to create living, breathing organic landscapes that evolve and move with the space.

I am also very interested in using data as a form of creative expression. I have been experimenting and working with real-time data visualization and large format screen based art.