Lecture

Software Tools for Generative Art

An overview of coding environments and programming languages for generative experimentation, sketching, testing and complex project design.

Lecture Slides

INTRO
Programming tools are technically most versatile, and conceptually most appropriate creative means specific to digital culture because they provide the most precise access and the highest level of control of digital architectures (microprocessors, computers, etc.) and infrastructures (computer systems, communication networks, etc.). Charlie Gere gives a wonderful introduction to the context, history and key features of digital culture and digital paradigm in Digital Culture (2008, 2^nd edition). Programming tools are particularly effective when combined with other tools, techniques and technologies, and work best when integrated with broader artistic skills and experience in visual arts, film, music, performance, etc. They offer huge potential for artistic exploration and expression but require procedural fluency: algorithmic thinking and the ability to convert events and phenomena into sequential/linguistic structures.
There are around 700 programming languages, some of which are popular with new media artists. General popularity, applicability and hardware support of programming languages changes over time due to a number of factors. Coding environments which are popular among the artists include: game and expanded reality tools, programming languages with visual interface, programming libraries, and core programming languages. Scripting languages such as HTML or the application-embedded scripting languages (Maya Embedded Language MEL, After Effects ExtendScript, former Flash ActionScript, etc.) conceptually and functionally stand between visual programming languages and programming libraries. They are application-specific (not multi-purpose), so I do not cover them in this overview.
Here is an overview of popular multi-purpose coding environments and tools:

GAME DEVELOPMENT AND EXPANDED REALITY (VR/AR)
These tools can be used for creating a wide range of interactive and non-interactive forms and experiences.

Unreal Engine (written in C++, uses C++ for additional functionality/development) and Unity (written in C++, uses C# for additional functionality/development). Both are powerful, portable to many platforms, offer many assets and learning resources. Limitations of both is steep learning curve, and main usage scenarios being proprietary.
https://www.unrealengine.com/en-US/
https://unity.com/

Godot
Written in C++, uses: C#, GDScript, VisualScript and Python for additional functionality. It is powerful, portable to many platforms, and Open Source for any usage/application scenario. It is easier to learn than UE and Unity, has solid learning resources and being open source, it is transparent for programmable extensions, so potentially more powerful. Its limitations are the variety of assets and learning resources compared to UE/U. It has been gaining popularity since its introduction.
https://godotengine.org/

PROGRAMMING LANGUAGES WITH VISUAL INTERFACE (VPL)
Visual interface offers arguably higher level of abstraction, more intuitive learning and usage than programming libraries and core programming languages. Common limitations of VPL are that visual interface is usually not suitable for complex projects (this depends on the particular VPL), and the pragmatism of their interfaces obscures deeper understanding as your learning and skills progress. However, these are powerful tools for many purposes.

Max/MSP/Jitter
Designed for interactive programming of music and multimedia. It is powerful, versatile, has large user and developer base and good learning resources, but is proprietary.
https://cycling74.com/

Pure Data
Designed for interactive programming of music and multimedia. It is powerful, has large user and developer base, good learning resources and is Open Source. It is somewhat limited due to idiosyncratic concepts of some operators, functions and objects.
http://puredata.info/

vvvv
Also known as four Vs or fier fau, as it was initially developed by German speaking coders. It is written in Delphi, and plugins can be developed in .NET and C#. Designed for interactive programming of multimedia, especially motion graphics and A/V processing in large media environments with physical interfaces. It is powerful, fast, has strong graphics support (OpenGL), and is Open Source. It is limited by idiosyncratic concepts of some operators, functions and objects. Also, it has smaller user and developer base and less learning resources compared to Max/MSP/Jitter.
https://vvvv.org/

TouchDesigner
Initially based on Houdini, now Python. Designed for interactive programming of multimedia, motion graphics and A/V processing in large media environments with physical interfaces. It is powerful, fast, has strong graphics support (OpenGL), and good learning resources. Its limitations are that the UI is complex to navigate, learn and maintain understanding. Also, it is proprietary. The free non-commercial license has image resolution limited to 1280×1280, and it does not include a subset of operators for Shared Memory, C++ extensions, SDI, etc. Otherwise, non-commercial license leaves you with a powerful tool, and is useful for learning Python.
https://derivative.ca/

PROGRAMMING LIBRARIES
Programming libraries comprise additional sets of instructions, functions, constructors and objects which offer higher level of abstraction (more intuitive approach) and functionality than their core programming languages. Processing, p5.js and openFrameworks are programming libraries designed specifically for artistic purposes, and they can be learned independently of their core programming languages.

Processing is a Java library and has its own programming development environment (GUI interface) for writing, executing, testing, debugging code and adding additional functionality. It has a relatively smooth learning curve, many resources and examples. It also has ports to many platforms and spin-off projects for various hardware devices. Its main limitation is execution speed for more demanding media projects. Also, its main components, although designed to be easy on the beginner, actually conceal key programming paradigms such as classes and objects, and some of the manuals and tutorials are extremely confusing regarding this. Therefore, selecting good learning resources and examples is essential for straightforward and effective learning and development. As with all other popular coding and hardware tools, this selection can be difficult but is important because many learning resources are not very good. Finally, like with all other programming libraries, the creative coder’s specific and more advanced needs lead to the programming language on which the library is built. In case of Processing it is Java.
https://www.processing.org/
http://learningprocessing.com/

p5.js is a JavaScript library built on the Processing paradigm, so the structure and syntax are similar. It works in a web browser and has relatively smooth learning curve. Although it has been rapidly gaining popularity in the last years, the learning resources and examples are good and succinct. Its limitations are speed, opaqueness (see Processing), and that specific needs lead to JavaScript. However, it can be used effectively for building complex web-based projects.
https://p5js.org/

openFrameworks is a C++ library. It is powerful, fast and extensible. Its limitations are steeper learning curve, it requires knowledge of its core programming language (C++) more than Processing and p5.js, and its learning resources and examples are relatively modest.
https://openframeworks.cc/

FFmpeg is a command-line suite of libraries and programs for A/V processing, editing, compositing, effects and post-processing. It was primarily designed and is mainly used for quick execution in a console application (such as Command Prompt or PowerShell in Windows, or bash in OSX and Linux). However, elaborate FFmpeg projects can be developed and executed in a development environment (see Text Editors and IDE/PDEs below) for complex media processing and manipulation projects. It is powerful, fast, and can be integrated with other programming languages to leverage its media manipulation capabilities. Its limitations are that its syntax is less intuitive than common programming libraries, and therefore less suitable for larger project development. Also, its learning resources are tricky to navigate, for example the official FFmpeg documentation portal, although extensive, is written in terse style and lacks examples. However, with a little effort all usage scenarios can be learned and FFmpeg can be used for all but highly sophisticated compositing needs.
https://www.ffmpeg.org/
https://trac.ffmpeg.org/wiki/WikiStart

PROGRAMMING LANGUAGES

C++ is a hard-core programming language, and although not designed specifically for the artists, it is one of the most powerful and popular programming languages today, and many artists use it. It is powerful, fast, highly modular and extensible, has a huge number of libraries, special platforms and implementations. It also has (too) many learning resources and examples. Its limitations are its vastness and complexity (number of libraries, modes and implementation scenarios), and finding good learning resources and examples.
https://isocpp.org/

Python is one of the most powerful and popular programming languages today, and many artists use it. It has also become one of the popular programming languages in machine learning. It was not designed specifically for the artists, but it features more intuitive, language-like syntax compared with other programming languages, such as C++ or Java. It is powerful, fast, highly extensible through many libraries, has a lots of platforms and implementations, and (too) many learning resources and examples. Its main limitations are finding good learning resources and examples. The official version of the language is Python 3.x, and most resources refer to it. The Python 2 language was officially discontinued in 2020 but there are still many online resourcesreferring to it, so you need to pay attention to versioning: discard 2.x and go for 3.x. With good learning resources, Python is my recommended programming language for the artists.
https://www.python.org/

JavaScript has evolved from its modest beginnings into a powerful programming language, and today has extensive development and support. It is versatile, has many libraries, platforms and implementations, and can be used online or offline, with client-based execution in a web browser. It is also relatively easy to learn, and has (too) many learning resources. Its limitation is speed which relies on the host browser’s JavaScript engine speed (which is not stellar) in addition to the speed of hardware. Also, several of its key concepts and components have quirky syntax, which may be confusing. Finally, finding good learning resources and examples. However, it can be used very effectively for building complex web-based projects.
https://en.wikipedia.org/wiki/JavaScript

SuperCollider is a programming language for real-time audio synthesis, processing, algorithmic composition and interaction. Although it is primarily used to work with sound and music, it is popular with new media artists working with sound. It is powerful, fast, extensible, and supports live coding – modifying code while it is being executed. It has solid learning resources. Its limitations are relatively steep learning curve and that it requires musical knowledge.
http://www.audiosynth.com/

TEXT EDITORS, IDES AND PDES
These are the main tools for working with programming libraries and programming languages. Their ease of use and functionality are essential for smoothness and efficiency of both learning  and development. I do most of my work with these tools, for example for preparing many components of our Generative Art classes. There are many of these tools, and their popularity and support fluctuates similar to programming languages. I selected three of the most popular of these tools, which can be used for development in all programming libraries and languages in this overview.

Sublime Text 3 is proprietary, but can be used for free with no limitations. It is a text editor which, with extensions (packages), easily becomes a full programming development environment. It is fast, stable, highly functional and customizable, has large user and developer community.
https://www.sublimetext.com/

Notepad++ is an Open Source text editor which, with extensions (packages), easily becomes a full programming development environment. It is fast, stable, highly functional and customizable, has large user and developer community.
https://notepad-plus-plus.org/

Visual Studio Code is an Open Source, fully integrated development environment. It is fast, stable, highly functional and customizable, has large user and developer community, and excellent documentation. This is arguably the best Microsoft’s product to date.
https://code.visualstudio.com/

ONLINE TOOLS AND SERVICES
Many online tools and services can be useful for learning to code and for developing your projects. Other than the two I selected her, there are many more online tools and services to cater your specific needs and programming languages. Their landscape constantly changes as well.

Stack Overflow is a vast Q&A portal, mainly dedicated to programming and related disciplines. You can post a question and wait for the answers, but most often you just need to enter a search phrase for your question, and find it already answered there, usually with several answers, ranked by usefulness.
https://stackoverflow.com/

GitHub is a software development versioning system on the web, based on Git versioning tool. It hosts a wide range of programs, libraries, plugins, assets, learning resources, and other material you may need.
https://github.com/

Tips on Creative and Effective Approach to Coding

– Understand basic principles of programming, computer architectures and information technologies. You will be able to learn coding tools faster and will be more effective in any development scenario. This does require some math, but is very beneficial. Unfortunately, a majority of learning resources do not offer insights into the fundamentals of computing, or often do it superficially and sometimes incorrectly.

– Select best tool(s) – coding, software or hardware – for your specific context and needs. So do a thorough research on the topic you want to address, and prepare well beforehand.

– Learn several programming languages and if you can, learn them in parallel. This is time consuming and can be tricky but you will learn faster as you progress.

– Select one good manual and one video tutorial per programming language or other tool you want to learn. This is crucial because you do not want to waste time and get frustrated. So do your research well.

– Finally, as in all creative disciplines: learn continuously, cultivate your creative methodology and evolve your relationship toward broader contexts in which you work and communicate.

LOOK UP FURTHER

Jupyter
Language-agnostic notebook system (IDE). Good for sketching Python and several other programming languages.
https://jupyter.org/

Online Tools for Exploring Machine Learning
Require solid programming skills but are useful.

ml5.js
p5.js library built on TensorFlow.js library. Part of Processing/p5.js initiative, integrates well with p5.js, Open Source.
https://ml5js.org/
Daniel Shiffman’s video intro: A Beginner’s Guide to Machine Learning with ml5.js

RunwayML (cloud computing service – commercial)
https://runwayml.com/
Several projects that integrate it with: Unity, Touch Designer, Pure Data, Max/MSP, p5.js, Processing, OpenFrameworks, JavaScript: https://runwayml.com/integrations/.
Daniel Shiffman’s video intro: Introduction to Runway: Machine Learning for Creators

Examples

Evan Roth – 2010 – Graffiti Analysis
4D dynamics of graffiti writing/drawing converted into a sculpture.

Robert Seidel – 2010 – chiral
Video animation projection-mapped on sculpture (5.1×2.6×3.7 m) and projected conventionally on a screen (2.5×2 m), both in handmade Taiwanese paper.

Luke DuBois – 2016 – Acceptance 2016
Nomination speeches Clinton/Trump (70-80% identical, distributed differently).
Speech to text + semantic analysis + matching + real time sync.

Oscar Sharp & Ross Goodwin – 2016 – Sunspring
Short film screenplay generated by LSTM RNN trained on SciFi movie screenplays available online.

Branger_Briz – 2017 – Muse AI Supercuts
AI generates daily supercut music videos in which the words of the Muse’s song Dig Down (2017) are voiced by notable persons from the online videos.

Look Up Further

Roman Signer
Proto-scientific cognitive drive and joy of real-life experimentation. Personal stories behind most of his projects.

Work Study

Processing 4: Interaction.

processing tutorial 4 – Interaction_compressed

Download Link to Generative Design code base

Booking and Ordering

Hi all, please let me know (via email or text) if you intend to book or purchase anything for your projects. Below is a link for more information on this matter.

Teaching Material

Again, you can contact me using email or my phone:

bleow001@e.ntu.edu.sg

81813134