Final Hyperessay

As They Might Think
Envisioning Media Art In The Age of Machine Intelligence


When Richard Wagner put forth his ideas for the Gesamtkunstwerk, he was concerned with the creation of a total artwork that would unify the various disciplines of art through theatre. The opera was seen as the pinnacle of artistic expression during the mid-to-late 19th century and it provided the perfect platform for Wagner to pursue his ideals. Introducing the darkened space and controlling how and what audiences saw, Wagner was one of the first to successfully execute on the concept of the “suspension of disbelief.” As we progressed along, creatives and technologists remained dedicated to furthering the “totalitizing experience of art.” The works of creators like John Cage and Nam Jun Paik began to explore the behavioral relationships between artist and observer; bringing into view cybernetic visions within the field of art.

Performers’ movements in John Cage’s “Variations V” triggered sensors around them that controlled audio and video systems. An indeterminate interactive space that redefined art-making.

Roy Ascott commented in his essay that modern art needed to offer “a high degree of uncertainty” and permit “a great intensity of participation.” Experiments into full body digital immersion in the 80s and 90s answered that call, and more. With the rise of head-mounted displays, binaural audio and sensory input devices, systems such as the VIEW at NASA Ames allowed artists to create “doorways to other worlds” that existed beyond the physical limitations of our own. Seminal virtual-reality works like Char Davies’ Osmose brought such worlds to life and formalized new relationships between audience and artwork. In effect, interactivity and immersion had now become key considerations in the construction of a gesamtkunstwerk — going so far as to say that it was essential now for these artworks to exist in the form of media art.

Virtual Reality technology allowed Char Davies to imagine a new reality and explore new forms of interactive experiences.

With these developments in technology, one could be excused for thinking that a pinnacle has been reached. What more can be done, given that it is now possible to transcend both body and mind into an altered state of reality? Indeed, there is more to be achieved. In the 30 years since the development of the VIEW system, technology, society and the way we interface with the world around us have shifted significantly. The rise of the internet and ubiquitous computing has forced us to reconsider our place amongst technology as well as its impact on our lives. Society has been reconditioned on a level that has not been witnessed since the Industrial Revolution and digital data has become the all encompassing common denominator of human society. Consequently, the total artwork of the present and future should take into account these cognitive shifts in societal behavior and derive new definitions of what constitutes the gesamtkunstwerk of the 21st century. Refik Anadol is one such artist whose works explore our relationship with technology, the merging of physical and digital space, and how data can be exploited as a medium to create profound media art experiences.

Rethinking Digital Immersion

A contemporary media artist who works across large varieties of digital media, much of Anadol’s works focus on site-specificity — choosing to create digital public art that leverage and accentuate the surrounding architecture. Anadol uses projections (both indoors and outdoors)  as well as large scale screen installations to create immersive spaces that transport audiences into an imagined environment. As a media artist, designer and spatial thinker, Refik Anadol is also intrigued by the ways in which the transformation of contemporary art culture requires the rethinking of new aesthetics, techniques and most importantly, a dynamic perception of space. His works aim to create visually agnostic depictions that form a visual language capable of appealing to large groups of people; despite cultural backgrounds or aesthetic preferences.

“Virtual Depictions – San Francisco”

The importance of space in Anadol’s works is also seen in his interpretation of it through his artistic expressions. By embedding media art into architecture, he questions the possibility of a future within which all human-made surfaces are informed in some shape or form, by the digital footprint that mankind leaves behind. Spatiality also informs how Anadol designs his viewers’ interactive experiences. His works cater towards eliciting the reactions and interactions of passers-by  within unconventional spatial orientations, going against the established status quo of pursuing highly abstracted digital spaces to create immersion.  Anadol’s works then, suggest all spaces and facades have the potential to be utilized as the media artists’ canvas; seemingly determined to prove that highly immersive spaces can in fact be achieved without resorting to the use of “The Ultimate Display“.

Data As A Medium

Refik Andol’s works and creative processes are based on and deeply influenced by technology: specifically generative data. As modern members of society, we are profligate with the creation of new data. In 2013, it was noted that 90% of all the data output ever made occurred over the span of 2 years (2011-2013). The pursuit of smart cities and the vision of embedded sensors within our built environment have contributed towards this exponential growth of data-generation. We now occupy living, evolving, self-aware cities, bristling with input devices that continuously monitor and provide a myriad of data feedback. Human society also produces large quantities of data through modes of self-expression such as social media and other computer mediated technologies. Often times, all of this data rarely sees the light of day; relegated to internal uses within the social groups, companies and government bodies that process them.

“Wind of Boston – Data Painting”

However, what if as creatives, one was to exploit these information sources and utilize these large quantities of unprocessed data to create new forms of expression? How do we negotiate the ever-fading distinction between our personas within the digital space and our “real” lives within the physical world? What if we were to question how our experience of space is changing now that digital objects ranging from smart phones and embedded sensors to urban screens have colonized our everyday lives. How have computer mediated technologies changed our conceptualizations of our everyday spaces, and how has our built environment embraced these shifting conceptualizations to accommodate these changes?

Through his work, Anadol tackles these questions; not by simply integrating media art into built forms, but by translating the logic of media technologies into spatial design itself. Data becomes the medium and the subject; the cause and the effect of trying to establish the gesamtkunstwerk of the 21st century. That said, how does one go about defining a new visual vocabulary for these types of works without resorting to the established visual practices of traditional data visualization.

During his talk at the Emergent Visions symposium, Refik Anadol addressed this specific concern when discussing his works. He elaborated that for creatives working with data, it was not enough to simply stop at data visualization. “We are not data scientists,” he remarked. Authenticity and accurate representation of the dataset was not always required, or maybe even useful in developing a visual outcome. Rather, Anadol encouraged creatives to attempt what he termed as data dramatization; to create a sense of empathy and derive emotion out of the data as opposed to presenting a cold, immutable tabulation of scientific findings. He argued that for data driven media art, the meaning of the work should be held in the poetics of the representation and not be derived from the source itself. This philosophy rings similar to Roy Ascott’s thoughts on telematic art where he put forth that “meaning is the product of interaction between the observer and the system, the content of which is a state of flux.”

Melting Memories (2018) – Refik Anadol. It is not necessary for the viewer to know or even grasp every single datapoint that contributes towards the visualization of the work. The final outcome and the emotion it invokes is enough to create meaning between the work and its viewer.

The usage of data as a medium then introduces a new approach to thinking about media art. It calls into question the very paradigms that form the foundations guiding our understanding about the collection, storage and presentation of data in the creative practice. In this regard, Archive Dreaming by Refik Anadol serves as a potential candidate to further dissect and understand how we could go about building future “total artwork” experiences.

A New Reality for Old Knowledge

Archive Dreaming (2017) by Refik Anadol is a data visualization work installed at SALT Research in Istanbul, Turkey. Being a public art installation, the work was conceived as an alternative method for visitors to peruse and engage with the document collections of SALT Research. The facility comprises a specialized library and an archive of both physical and digital sources. Their collections include visual and textual sources on art history, the development of architecture and design in Turkey, as well as documents dealing with transformations in the region during the transition from the Ottoman Empire to the Turkish Republic. Due to the significant age and frailty of most of these documents, large parts of the facility’s collections are housed in specialized, environment controlled rooms; reducing accessibility to large parts of the physical collection.

Architectural diagram of ‘Archive Dreaming’

While other attempts at a such a project might have led to the creation of a photographic or micro-film database, Archive Dreaming takes the concept of an interactive archive much further. The most obvious and significant aspect of this work is in its use of physical space and architecture. The installation is designed to be a spatial experience. In the words of Anadol himself, the work is described as an effort to “deconstruct the framework of an illusory space [that transgresses] the normal boundaries of the viewing experience of a library”. The work was specifically conceptualized with the notion to significantly transform the experience of accessing a knowledge repository. The work was to be a “a three dimensional kinetic and architectonic space of an archive.”

The vision to create a new paradigm for archival access may seem bold and ambitious. However, much of these ideas can be traced back to the research and thought experiments conducted by artists and technologists during the 60s.  Ivan Sutherland’s The Ultimate Display famously envisions a futuristic display system which takes the form of a room “within which the computer can control the existence of matter.”

While we are still decades away from experiencing anything remotely close to his description, Sutherland’s essay made significant arguments towards how we could utilize immersive environments to augment our current assumptions of a particular subject matter. He claimed that by simulating other realities foreign to our current comprehension, “we can learn to know them as well as we know our own natural world.” Archive Dreaming in many regards, does precisely this. Data is no longer confined to a network terminal or pages of a book, instead transporting the viewer into the data-space of the archive itself. It reframes our established preconceptions of what a library should be and might perhaps even establish new methods for scholarly research. Much like what Osmose did for physical transcendence, Archive Dreaming could conceivable do the same for academia, as users physically engage with, negotiate and understand the data-space.

One of the browsing modes of “Archive Dreaming”, the data-relationship view offers users a summary of how every document in the archive is related to others.
Hypermediated Hyperthoughts

The other aspect of Archive Dreaming that sets is apart from a simple digital database is the flexibility that it provides users. The system allows visitors to browse all 1.7 million documents in the archive in a variety of modes. Each mode draws different associations between the documents in the archives, assisting viewers to form relationships in a way that is suitable for their individual needs. In this way, the installation “intertwines history with the contemporary, and challenges [the] immutable concept of the archive.” This changing perception of the relationship between documents will seem familiar to those that have read the works of Vannevar Bush.

When Bush penned his ideas for the Memex in his essay As We May Think he was operating in an environment that was vastly different to ours today. While he lamented that the traditional index based referencing was counter-intuitive to how we approached thinking, the recent data explosion has made it all but impossible for even hypermedia to serve as a viable solution. Sifting-through and organizing data is now a task best left to algorithms and machine intelligence. Standing in for human curators, these systems are capable of  independently creating associations and forming relationships between countless datasets. These machines operate as we may think, but at speeds much faster than humans.

A mosaic of all the documents present in the SALT Archives

This realization is not lost on Refik Anadol either. In this post-hypermedia age, Archive Dreaming utilizes  Machine Learning and Artificial Intelligence to look at, understand and organize all of the content found within SALT’s large set of documents. The system builds its own set of definitions and correlations to make sense of the data, autonomously without human intervention. This codified information is then easily controlled by visitors from a touch sensitive console at the center of the installation space.

This interface showcases the benefit of digitizing the analogue experience of accessing a library’s collection. Users are able to much more easily access the content, with the process made more tactile and immediate. This also has the benefit of externalizing human intelligence, allowing us to transcend the limitations of what we can achieve with our own minds. In this regard, one begins to question if the gesamtkunstwerk of the future should also concern itself with the processing of data, in addition to using it as its medium.

A user accessing the central console to interact with the archive.
Data Dreaming

While Machine Intelligence has proven to be useful in helping humans circumvent the limitations of our intellectual minds, it also holds great potential in augmenting our creative minds. In Archive Dreaming, machine intelligence takes center-stage for the final aspect of the installation. Getting its name from this process, the installation enters into a “dream mode” when no one is interacting with it. The system uses the knowledge gained from studying and classifying all of the 1.7 million documents to “hallucinate” new ones that could possibly exist in the archives. Although the results may sometimes appear fuzzy and underdeveloped, acknowledging that humans are no longer the only ones capable of creative execution is a sobering thought.

The installation hallucinates new possible documents based on the ones it has already seen.

The dreaming feature of the installation is powered by the emerging field of Generative Adversarial Networks. This technology allows computers within the system to learn and evolve completely unsupervised. This presents new challenges for creatives working with technology both as a medium and as a subject matter. Now that machines are capable developing their own sense of aesthetics, how do we as creatives navigate this new space in art-making? In the case of Refik Anadol, he has chosen to embrace this development to work in concert with the machines, furthering his study and practice of media art. One could postulate then, that the future of media art is shifting towards the eventual removal of the human from the creative process entirely and that Anadol’s works are the first steps in this direction.

Perhaps the future of the gesamtkunstwerk lies in the complete release of creative control, allowing machines to determine the optimal experience required for a total artwork. Perhaps the role of the human artist in the future would be to provide vague suggestions while the machine computed the final visual outcome; creating as they the machines might think, as opposed to how we think.


In 1966, engineer and friend of artists Billy Klüver gave a talk about the changing relationship between art and technology. In it, he mentioned that up until that moment, art had remained “a passive viewer of technology.” However, more and more creatives had begun to experiment with the inclusion of technology, challenging and redefining what art meant. Despite this progress, to Klüver, an engineer was only “raw material for the artist” the same way paint and charcoal would have been. This seemed to suggest that while important cross-pollination was happening, art and technology were still very much divided into their own camps, and a true blend of both disciplines had yet to happen.

But as Klüver so rightly pointed out, for Aristotle “Techne” meant both art and technology. And indeed, they should exist as the same thing. We have come a long way since Wagner’s first foray into actualizing the gesamtkunstwerk. Art and technology are now so deeply intertwined, each informing the other and erasing the hard boundaries that were still present slightly less than 60 years ago. The future of media art is dependent on this continued partnership. It is important for man to continue pushing the boundaries of the artistic practice, be it through the embrace of technology, or the resistance of it. The total artwork then, becomes a futile pursuit, forever changing as the technology available to us evolves. Maybe the pursuit should instead be of something grander and more ambiguous — a pursuit of honesty and artistic purity, choosing to create works on a medium that best represents the views and intentions of the creative.

Final Hyperessay: Key Work Selection | Archive Dreaming

Archive Dreaming (2017) by Refik Anadol is a data visualization work installed at SALT Research in Istanbul, Turkey. It was presented as part of The Uses of Art: Final Exhibition with the support of the Culture Programme of the European Union. Being a public art installation, the work was conceived as an alternative method for members of the public to peruse and engage with the document collections of SALT Research. The research facility comprises a specialized library and an archive of both physical and digital sources. Their collection includes visual and textual sources on art history, the development of architecture and design in Turkey, as well as documents dealing with the transformations in society and the region from the Ottoman Empire to the Turkish Republic. Due to the significant age of most of these documents, large parts of the collection are housed in specialized, pressure and temperature controlled rooms; causing most of the physical sources to be as easily accessible.

Architectural diagram of ‘Archive Dreaming’

While other attempts at a such a project might have led to the creation of a simple photographic database, Archive Dreaming takes the concept of an interactive archive much further. The most obvious and significant aspect of this work is in its use of physical space and architecture. The installation is designed to be a spatial experience. In the words of Anadol himself, the work is described as an effort to “deconstruct the framework of an illusory space [that] will transgress the normal boundaries of the viewing experience of a library”. The work was specifically conceptualized with the notion to significantly transform the experience of accessing a knowledge repository. The work was to be a “a three dimensional kinetic and architectonic space of an archive.”

One of the browsing modes of “Archive Dreaming”

The other aspect of Archive Dreaming that sets is apart from a simple digital database is the flexibility that it provides users of the system. The work allows users browse the 1.7 million documents in the archive in a number of possible ways; each drawing different associations between the documents through the various viewing modes. This assists the viewer to draw relationships with the documents in a way that is suitable for their individual needs. In this way, the installation “intertwines history with the contemporary, and challenges immutable concepts of the archive.”

Such a system is only conceivable with the aid of Machine Learning and AI that have the capabilities of looking at, understanding and organizing all of the content found within this large set of documents. All of this processed information is then easily controlled by visitors from a touch sensitive console at the center of the installation space.

A user accessing the central console to interact with the archive.

Machine Intelligence also takes center-stage for the final aspect of the installation. Getting its name from this process, the installation enters into a standby “dream mode” when no users are interacting with the archives. The system uses the knowledge gained from studying and classifying all of the 1.7 million documents in the archive to “hallucinate” new ones that might possibly exist in the archives. In a post-truth era where “facts” are constantly challenged and called by other names, this process calls into question about how much of our shared history is “real” and how much of it is to be trusted; a challenge that historians and documentarians constant face and grapple with. It also postulates the idea of a brave new world where creativity and creation might no longer be just in the domain of humans and perhaps, machines might play just a vital role in shaping the art histories of the future.


Osmose by Char Davies is an example of a media art work that uses virtual reality to craft a fully immersive experience for its audience. Going beyond establishing a visual outcome, the artwork employs a multi-sensory approach to successfully transport viewers into the artist’s imagined space. The unique interaction method centered around breathing, as well as the use of a head-mounted display create a sense of tangibility that  embeds the immersants deep into an otherwise foreign and unrealistic space that feels real.

In that regard, Osmose may be seen as pioneering attempt at realizing Ivan Sutherland’s vision that was penned in his essay ‘The Ultimate Display’ thirty years prior to the creation of this work. In it, Sutherland postulated that as technology got better, humans would be more capable of designing all encompassing sensory interfaces that would appear to simulate our reality to a point where  “a computer can control the existence of matter”. While highly ambitions and visionary, the statement does align with how technology has progressed over the last 50 years since it was written. As a society, we strive to create computer systems and interfaces that remove the intermediary layer with which we manipulate media, and have tried to place the user as close to the content as possible with regard to the user experience. The ultimate goal then, is to march towards “complete immersion”.

Sutherland also discusses the idea of familiarity and how that affects the pursuit and uses of the theoretical ultimate display. He posits that being able to recreate experiences that one is already familiar with in real life would help to increase the perceived immersion in this “ultimate display”. In his essay  ‘Virtual Environments’, Scott Fisher mirrors this line of thought by explaining that “for most people, duplicating reality is an assumed […] goal” for digital imaging technology. While the display might be useful in perfectly recreating the reality we are all familiar with, the display could also be used to help us discover new alternate realities with rules that we are not familiar with. While Sutherland uses the example of possibly exploring the physical properties of quantum mechanics to further human understanding, exploiting such displays for a more poetic and artistic outcome is also reasonable. Davies uses Osmose to allow her viewers to experience a new reality that would have been other difficult to reconcile with something as static as a painting.

Historically, art has always been communicated to others on a mediated platform. A painting is a summation of an artists internal experiences and believes that are then repackaged and visually represented for the consumption of another individual. The experience becomes largely external to the original thought and intent of the artist; it is consumed second-hand. As Fischer puts it, the “abstracted second-hand knowledge is often more generalized and concentrated” but lacks the “balance and completeness of experience.” First-hand experience then, becomes the defining factor as to how complete the encounter with a work becomes. Osmose brings this to the table by having the immersants experience first-hand, the imagined space just as the artist would have in her head during the conceptualizing of this work. The artist is provided the opportunity to externalize their thoughts in a more tangible fashion and viewers are able to step into the artist’s mind and form their own conclusions from that experience.

One could argue that the crucial element that ties all of this together is the way in which locomotion is controlled within the world of Osmose. By having her immersants’ breath patterns tie directly to how they move within the world, Davies is able to successfully marry a real-world physical experience (breathing and the tangible principles of buoyancy) to an imagined new space with it’s own rules in a way that is both familiar and awe-inspiring at the same time. This was the key to the unprecedented immersion that viewers of the work felt and perhaps, the way to continue approaching immersion in the field of media art.


Rethinking Multimedia

It is often common for one to assume the meaning of certain words and phrases; especially if said word or phrase is common place in one’s daily life.

Navigating modern music libraries become a multimedia experience in and of themselves.

Such is the case with the term, multimedia. In this digital day and age, the word multimedia is thrown about rather flippantly. We have come to assume that all media is multimedia – no form of artistic expression really exists by itself anymore. Even something like assumably “single-media” like listening to music has become an exercise in multimedia consumption. The digital consumption of audio puts in a situation where we first interact with our digital “libraries” sorted by any number of categories from albums to playlists. We sift through the lists of words and images before arriving at our music. Even then we are likely presented with the album art of the track with all its subtext and suggestions as the music begins to play. The experience is never purely auditory; perhaps even a little distracted.

With the advent of video sharing services like YouTube, the act of music consumption if further skewed towards a multimedia experience with users likely engaging with music videos as opposed to a purely auditory soundtrack. These videos augment the experience of listening to a track by providing visuals that create a richer experience that would otherwise be unattainable with just one type of media. While some may embrace the expansion of the original intended experience of the media, others yet may lament the added distraction muddying the original intended experience.

Coldplay’s music video for “Hymn for The Weekend” illustrates the vibrant Holi festival in India, creating a stunning visual that is otherwise impossible to create using just the media of music.

No matter one’s stance, multimedia as a term then, could be defined as the totalizing sensory experience gained from combining more than one method of expression to create a sum that is arguably greater than its parts. This concept of totalizing is discussed in the essay as the Gesamtkuntswerk. Attributed to Richard Wagner, the Gesamkuntswerk or “Total Artwork” is Wagner’s idealized union of all the various aspects of creative expression; the combination and unification of all the arts to create an experience that could capture the essence of human experience.

This concept and definition was of particular interest to me as it reinforced a personal believe that I always maintained about the term multimedia. While it may have only be in prevalent use over the last 20 to 30 years and heavily tied with technology, I have always held onto the notion that multimedia was far from a recent concept. Like the caves of Lascaux, examples of multimedia experiences predate many of humankind’s technological advances. However, as the reading suggests, it is with the advent of technology (particularly the personal computer) that artists and technologist alike have been able to fully bring to fruition the Wagner’s vision. The concept of multimedia as a meta-media (a medium comprising other media) would not have been able to be as successful if not for the efforts of technologists like Vennevar Bush and Tim Berners Lee. Our modern understanding of multimedia is predicated on the works of these pioneers.

What was particularly striking to me as well was how many of the speculations made by artists and technologists in the essay have since come to be reality. For example, Weiner’s preoccupation with human-computer relationships in his study of cybernetics led him to the conclusion that “the quality of our communication with machines effects the quality of our inner lives”. Today, we have entire professions built around this notion – user experience design. The rise of social media and the significant social impact that our relationship with our devices have greatly changed how me consume media and perceive the world. These “machine-mediated” experience of social interactions and our experience of the world around us have the power to shape and affect how we think and perceive. Multimedia then, has become not just a tool for artists to create better modes of creative expressions, but it has also evolved to place greater power in the hands of creatives to influence audience reactions and the outcome of a work.

Another arresting prediction made by Scott Fischer was the vision of a immersive virtual space that would “give birth to a new form of participatory, interactive electronic theatre.” With recent advances in the field of Virtual and Augmented Reality, VR has fast become a technology that is getting more and more accessible to the general public. Platforms such as the HTC Vive have given birth to online shared spaces where users can engage in telematic interactions with others across vast distances. With full body tracking, voice chat and surround audio, people now have access to fully immersive environments where they are able to take on any likeness and interact with others as if in the same space.

VRChat and the Ugandan Knuckles meme phenomena is one such example of Fischer’s vision of a participatory electronic theatre coming to life. The platform was original envisioned as a space for users to interact with one another in an attempt to escape the mundane realities of regular life, similar to earlier concepts such as Second Life. However, the richer interactive experiences afforded to users soon saw players creating personas, characters, stories and lores in a machine-mediated multi media space. This types of ad-hoc, participatory theatre would be difficult to recreate without the aid of multimedia.

Multimedia over the years has come to represent more than just the nexus point of a host of various creative expressions. We have been able to create, richer, deeper and more all encompassing experiences that challenge both creators and audiences alike. The digital revolution has allowed for the genesis of type of media that truly allows for us to express ourselves the way our minds work. While some may argue that technology and multimedia have worked against us by making us more distracted individuals, perhaps, this was the intended result after all? As Ted Nelson philosophized, perhaps multimedia, like all human inventions, is an extension of ourselves. Perhaps the burden of responsible consumption sits of our shoulders after all.

“I See You” – An FYP Concept

The following is an elaboration of one of the perviously discussed FYP ideas that I am most interested in and confident of executing over the period of my final year in ADM.

1. Project Description


I See You is an interactive installation that aims to raise awareness about surveillance, spying, and stalking in a rapidly digitized world. With the rapid rise of ‘smart nations’ and connected cities world-wide, how much are we willing to give up in exchange for a more modern and convenient environment. Are we exchanging our privacy for a cause that is much greater than us, and how much of it is truly beneficial to society? This work aims to raise the dystopian question on whether the advancements of technologies and the convenience that comes with it are worth the trade-off in the civil liberties of the privacy of an individual. Do we, the Snapchat and Instagram generation, live life too lose and dangerously in an age where a simple username is capable of revealing whole identities to strangers both online and offline.


Visitors will be tracked using video analytics during their visit to the FYP show with hidden cameras. Using facial recognition algorithms to trawl through social media networks, visitors to the installation are then presented with their own social media feeds and online interaction. The installation would be in an enclosed space, so as not to ruin the element of surprise for the visitors. The audience will also be able to look through the publicly available social media interactions of other visitors that have passed through the FYP show’s exhibition space.

Installation Specifications

The installation would ideally be placed in an enclosed space such as the Truss Room. The three projection screens as well as the space in the middle would be ideal for recreating what I envision for the installation. The projections will be used to show the various live video feeds with the video analytics overlay showing what is happening in real-time. Some of these facial recognition algorithms running, surveillance looking UI resembling tropes from spy movies and games, and the results of the facial recognition will also be shown in these projections, creating a security console experience for the viewers.

The aesthetics of these projections would be black and white videos with markers and indicators that are overlaid to create an uncomfortable, dystopian appearance. Some inspirations that I drew from include the video game Watch_Dogs.

The main interaction console in the centre of the room will be a large touch sensitive overhead projection of LCD display. Through this console, users will be able to access the social media feed gathered from all the visitors that have made an appearance in the show. This aims to create a sense of discomfort in the visitors to the exhibit, showcasing how easy it could be for strangers to be able to access private and intimate aspects of someone else’s life. Users will also be able to interact with the projections on the screens from this console, choosing to display items and feeds on the projections.

2. Artist references

I Know Where Your Cat Lives

I Know Where Your Cat Lives by Owen Mundy and a team of creatives is an online artwork that seeks to raise awareness about data privacy. It uses publicly sourced images of cats from social networks like Instagram and Facebook to create a realtime data visualizations of where these said cats are located. The site uses the geolocation metadata that’s embedded in the photos to visualize the exact co-ordinate of the photo on a map. Users are able to pan and zoom through the map to look at the various cats that are catalogued on the visualization.

The satirical component to this work lies with the fact that cats are unable to take pictures of themselves. As such, these geo-location co-ordinates point straight at humans that are posting such pictures, indirectly to also allow users to know where you as an internet user lives. I think this work is pretty successfully in creating a sense of unease once you begin to realize what is happening behind the scenes. I would like to create a similar sense of unease that requires a certain degree of interaction from the viewers. In other words, I would not want the fact that I am exposing people’s personal information that they have willingly provided, to be apparent right from the beginning.

Border City

Border City is an installation by Natasha Jen and her team from Pentagram. It was made for the Mexican Pavilion for the London Design Biennale. The work features infographics and animations projected onto a panoramic wall that circles the entire exhibition space. It also features a central console in the centre of the room. While not interactive, I really love the layout and presentation of information, and the use of space within this installation. I want to create a similar all encompassing effect for my own work as well. This is a huge reason as to why I think the Truss is an ideal location for the work that I am proposing.

Face to Facebook & fbFaces

Face to Facebook stole one million Facebook photos and, using face recognition algorithms, posted these photos to a custom made dating site where users can be matched with a potential date. The social media platforms to which we post our data on aren’t the only ones that have access to our information. Depending on our privacy settings, our photos and intimate details can be equally as exposed—available for download to just about anyone.

In the same vein as the work above, fbFaces is a web-crawler that targets public profiles for profile pictures and copies the image, Facebook ID and name. It then proceeds to do the same thing for your friends’ accounts. The artists printed out the downloaded profiles they collected and created  a wallpaper covering every surface of a large room. Profile pictures are reduced to mere pixels, as if to make them insignificant in the simplicity of their theft. They no longer represent people but information that is easily downloaded and appropriated—mere pixels.

The importance of these two works shows that what I am trying to achieve with my work is rather well within the realm of feasibility. In fact, my preliminary research into Facebook scraping and facial recognition led to a few possible algorithm models that I could emulate to create my own version for this project. However, the research also seems to suggest that I may have to alter my processes slightly and rethink how all these various components will fit together in once large installation that seems cohesive and a part of one artwork.

3. PROJECT timeline

The proposed production schedule may be found here.

4. Updates

After doing more research and sitting on the idea a little longer, I have come to realize that I would in fact need to change up the way I conduct the facial recognition process and how I obtain the faces of the visitors. I was drawn towards the concept of doxxing, which the process of using publicly available data to dig around for someone’s personal information such as bank and medical records. While the practice of doxxing is technically illegal, presenting already publicly available data is not. As such, I have begun playing around with the idea of creating another section of the installation that would be a pre-cursor to the “main room”.

I came up with the concept of a Photobooth that visitors must go through before they enter the Truss or exhibition space. This photograph of the visitor will allow me to run facial recognition in a much more controlled and systematic manner as opposed to attempting a real-time scan on the live video feeds. This would then allow me to get much more accurate results. As visitors leave the exhibit, I would ideally be able to provide a print out of what the viewer has made publicly available and that I have been able to scrap while the viewer has been going through the exhibit.

Thinking About FYP

Where to Begin?

I began the process of thinking about my FYP by undertaking some creative thinking exercises to kick-start my thought process. The first thing I did was to write down some of my skillsets and arrange them in order of confidence in my skillsets as well as how interested I was in creating those kinds of works.

This process was helpful in determining the kinds of works that I was most interested in executing. The red-line represented skills that I was slightly less confident in or types of work that I was not as interested in compared to those above the line. This process helped me narrow down the possible execution routes I could consider when I started thinking about the topics and ideas that I wanted to possibly tackle for my FYP.

The next thing I did was to list down the topics and ideas that I was personally interested and invested in, as well as issues that are hot topics within the public consciousness. (My personal ideas are on the larger slips of paper). The final step was to combine different slips of paper to see how these execution methods and topics could lead to interesting lines of questioning or idea projects that I could continue to develop into an FYP. That being said, here are 3 possible initial ideas that I have come up with.

1. I See You

Topic: Digital Surveillance | Execution: Installation

2. The Sound of the Skies

Topic: Weather & Climate Change | Execution: Data driven Audio & Visualization

3. City Movement

Topic: Transportation | Execution: Data driven interactive visualization

I will continue to elaborate on each of the ideas in the following posts.


Smart Nation Intervention: Data-driven Public Art

One of the key missions of the Smart Nation Initiative is to get the private sector in Singapore to work alongside the public sector to create a nation that is informed, driven, and made better by technology. The initiative aims to use technology to provide citizens with better services and an all round better quality of life. Continue reading “Smart Nation Intervention: Data-driven Public Art”