GOAL

What my team and I wanted to accomplish with this project was to create a visual and dynamic way to represent music.

HOW

Music

Firstly came the generation of music. We knew Adafruit’s circuit playground had sound capabilities. So it felt like a good place to start. We realized we could also make use of the capacitive touch capabilities to form notes by linking objects to the circuit playground. However, what we noticed from this was that the notes were quite robotic and short. So we decided that we had to link the output of each of the notes to Processing so that with output from each sensor, it would play a different piano note from Processing.

The Piano

For the piano, rather than simply connecting the wires to fruit, we decided to make piano keys out of agar agar jelly. We would also put fruits or vegetables in the agar agar with starting letters corresponding to those on the piano keys.

Agar Agar piano keys.This took a few tries of experimentation and a lot of random hitting of fruits to figure out how the capacitive touch worked. Also why sometimes only half of the circuit playground worked.

 

 

After figuring out what worked and what didn’t, Dinis and Jamie remade the Agar Agar to look more like piano keys an less like a child’s attempt to be healthy through jelly.

 

 

Thanks to Dom’s work with Corey, they managed to get the circuit playground to send readings through Arduino to Processing whenever the sensors were triggered.

Processing

Now came the hard part. Firstly, how do we “show” the notes? In what manner can we still generate a form of art that builds on itself to show the effect left behind by the music generated? As the main coder for the processing side of this, I was quite at a lost trying to think of an idea that both could show what I wanted but also was still possible within my amateur capabilities. I had after all, only just begun to understand the “alphabets” behind the coding language.

PROCESS

Attempt #1

Inspired by Casey Reas’s own generative art, I wanted the notes to leave behind shapes by not regenerating the background.

 

void setup(){

 size(600,600);

 background(0);

}

void draw(){

 if(mousePressed){

   background(0);

 } else{

   stroke(255);

 fill(mouseX/2,mouseY/2,80,90);

 ellipse(mouseX,mouseY,80,80);}

 

}

However the problem I encountered with this basic code was that i had no idea how to make the designs constantly generative and keep building on itself once the effect has been carried out, as seen in casey’s work, which continues drawing itself even if more are being generated elsewhere.

ATTEMPT #2

I tried to create something that seemed to have more autonomous movement rather than generating wherever the mouse was.

Rain seemed like a good place to start, I thought maybe I could link individual raindrops to each keypress so that the virtual piano would generate different coloured rain and result in a colourful artpiece.

However, what remained looked like barcode, probably due to the set number of areas in which the rain could fall, despite the difference in speeds.

ATTEMPT #3

Being a bit more ambitious, I tried looking into boids. They mainly operate on sets of rules which dictate how they move to a destination, what route they get there and by what means. The boids are also generated in an array with individual classes that dictate how the boid should appear, be it as circles or triangles.

The first source code I found allowed a set amount of boids to be generated. The destination was set as the position of the mouse and MousePressed would trigger more boids to be generated.

 

While this was an improvement, I would have preferred if individual boids could be generated with each keypress. Additionally, once the boids appeared, all they did was chase the mouse and had no other rules to dictate their behavior.

I found another source code by Craig Reynolds, to show how rules affected boids. The boids within that code were generated with by MousePress. Another interesting effect was the code that allowed the colour of boids to affect the colour of boids surrounding it. So after a while, only one dominant colour would remain.

So while playing with the source code, I updated it so that boids will be generated not at the mouse x y position, but rather from the middle of the screen. I also coded it so that it will generate at different keypresses. I also added the piano sounds to Processing’s sound library before linking each to a keypress. Thus the program is now able to play a note while generating a boid. Thus creating “life” with a music.

Running this with the preexisting code meant that the boids would flock together like a swarm of birds. The removal of the rule to avoid causes the boids to start to clump together and the placement of “Avoid” at the highest priority causes the boids to explode outwards beautifully.

FINAL CODE

Boid barry;
ArrayList<Boid> boids;
ArrayList<Avoid> avoids;

float globalScale = .91;
float eraseRadius = 20;
String tool = "boids";

// boid control
float maxSpeed;
float friendRadius;
float crowdRadius;
float avoidRadius;
float coheseRadius;

boolean option_friend = true;
boolean option_crowd = true;
boolean option_avoid = true;
boolean option_noise = true;
boolean option_cohese = true;

// gui crap
//int messageTimer = 0;
//String messageText = "";

import processing.serial.*;
Serial myPort;
String val;

import processing.sound.*;
SoundFile file;

void setup () {
size(1650, 1000);
String portName = Serial.list()[3];
println(portName);
myPort = new Serial(this, portName, 9600);

textSize(16);
recalculateConstants();
boids = new ArrayList<Boid>();
avoids = new ArrayList<Avoid>();
for (int x = 100; x < width - 100; x+= 100) {
for (int y = 100; y < height - 100; y+= 100) {
// boids.add(new Boid(x + random(3), y + random(3)));
// boids.add(new Boid(x + random(3), y + random(3)));
}
}

//setupWalls();
}

// haha
void recalculateConstants () {
maxSpeed = 2.1 * globalScale;
friendRadius = 60 * globalScale;
crowdRadius = (friendRadius / 1.3);
avoidRadius = 90 * globalScale;
coheseRadius = friendRadius;
}

void setupWalls() {
avoids = new ArrayList<Avoid>();
for (int x = 0; x < width; x+= 20) {
avoids.add(new Avoid(x, 10));
avoids.add(new Avoid(x, height - 10));
}
}

void setupCircle() {
avoids = new ArrayList<Avoid>();
for (int x = 0; x < 50; x+= 1) {
float dir = (x / 50.0) * TWO_PI;
avoids.add(new Avoid(width * 0.5 + cos(dir) * height*.4, height * 0.5 + sin(dir)*height*.4));
}
}

boolean data[] = new boolean[8];

void draw () {
noStroke();
colorMode(HSB);
fill(0, 100);
rect(0, 0, width, height);

if (tool == "erase") {
noFill();
stroke(0, 100, 260);
rect(mouseX - eraseRadius, mouseY - eraseRadius, eraseRadius * 2, eraseRadius *2);
if (mousePressed) {
erase();
}
} else if (tool == "avoids") {
noStroke();
fill(0, 200, 200);
ellipse(mouseX, mouseY, 15, 15);
}
for (int i = 0; i <boids.size(); i++) {
Boid current = boids.get(i);
current.go();
current.draw();
}
for (int i = 0; i <avoids.size(); i++) {
Avoid current = avoids.get(i);
current.go();
current.draw();
}
if ( myPort.available() > 0) { // If data is available
try {
val = myPort.readStringUntil(10); // read it and store it in val
if (val.length() > 10) {
println(val);
String[] s = split(val.trim(), " ");
for (int i = 0; i<s.length; i++) {
if (int(s[i]) > 500) data[i] = true;
else {
data[i] = false;
}
}
}
}
catch(Exception e) {
}
}

// for (int i = 0; i< 8;i++){
// if (data[i]) fill(255);
// else fill(0);
// rect(50+10*i, 50, 10, 100);
// }
// if (messageTimer > 0) {
// messageTimer -= 1;
}
// drawGUI();
//}

void keyPressed () {
if (key == 'q') {
tool = "boids";
message("Add boids");
} else if (key == 'w') {
tool = "avoids";
message("Place obstacles");
} else if (key == 'e') {
tool = "erase";
message("Eraser");
} else if (key == '-') {
message("Decreased scale");
globalScale *= 0.8;
} else if (key == '=') {
message("Increased Scale");
globalScale /= 0.8;
} else if (key == '1') {
option_friend = option_friend ? false : true;
message("Turned friend allignment " + on(option_friend));
} else if (key == '2') {
option_crowd = option_crowd ? false : true;
message("Turned crowding avoidance " + on(option_crowd));
} else if (key == '3') {
option_avoid = option_avoid ? false : true;
message("Turned obstacle avoidance " + on(option_avoid));
} else if (key == '4') {
option_cohese = option_cohese ? false : true;
message("Turned cohesion " + on(option_cohese));
} else if (key == '5') {
option_noise = option_noise ? false : true;
message("Turned noise " + on(option_noise));
} else if (key == ',') {
setupWalls();
} else if (key == '.') {
setupCircle();
} else if (key =='a'||data[0]) {
file = new SoundFile(this, "a.wav");
file.play();
switch (tool) {
case "boids":
boids.add(new Boid(width/2, height/2));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(width/2, height/2));
break;
}
} else if (key =='s'||data[1]) {
file = new SoundFile(this, "b.wav");
file.play();
switch (tool) {
case "boids":
boids.add(new Boid(width/2, height/2));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(width/2, height/2));
break;
}
} else if (key =='d') {
file = new SoundFile(this, "c.wav");
file.play();
switch (tool) {
case "boids":
boids.add(new Boid(width/2, height/2));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(width/2, height/2));
break;
}
} else if (key =='f') {
file = new SoundFile(this, "d.wav");
file.play();
switch (tool) {
case "boids":
boids.add(new Boid(width/2, height/2));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(width/2, height/2));
break;
}
} else if (key =='g') {
file = new SoundFile(this, "e.wav");
file.play();
switch (tool) {
case "boids":
boids.add(new Boid(width/2, height/2));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(width/2, height/2));
break;
}
} else if (key =='h') {
file = new SoundFile(this, "f.wav");
file.play();
switch (tool) {
case "boids":
boids.add(new Boid(width/2, height/2));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(width/2, height/2));
break;
}
} else if (key =='j') {
file = new SoundFile(this, "g.wav");
file.play();
switch (tool) {
case "boids":
boids.add(new Boid(width/2, height/2));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(width/2, height/2));
break;
}

recalculateConstants();
}

//void drawGUI() {
//if(messageTimer > 0) {
// fill((min(30, messageTimer) / 30.0) * 255.0);

// text(messageText, 10, height - 20);
//}
}

String s(int count) {
return (count != 1) ? "s" : "";
}

String on(boolean in) {
return in ? "on" : "off";
}

void mousePressed () {
switch (tool) {
case "boids":
boids.add(new Boid(mouseX, mouseY));
// message(boids.size() + " Total Boid" + s(boids.size()));
break;
case "avoids":
avoids.add(new Avoid(mouseX, mouseY));
break;
}
}

void erase () {
for (int i = boids.size()-1; i > -1; i--) {
Boid b = boids.get(i);
if (abs(b.pos.x - mouseX) < eraseRadius && abs(b.pos.y - mouseY) < eraseRadius) {
boids.remove(i);
}
}

for (int i = avoids.size()-1; i > -1; i--) {
Avoid b = avoids.get(i);
if (abs(b.pos.x - mouseX) < eraseRadius && abs(b.pos.y - mouseY) < eraseRadius) {
avoids.remove(i);
}
}
}

void drawText (String s, float x, float y) {
fill(0);
text(s, x, y);
fill(200);
text(s, x-1, y-1);
}

void message (String in) {
//messageText = in;
//messageTimer = (int) frameRate * 3;
}

–

 

class Avoid {
PVector pos;

Avoid (float xx, float yy) {
pos = new PVector(xx,yy);
}

void go () {

}

void draw () {
fill(0, 255, 200);
ellipse(pos.x, pos.y, 15, 15);
}
}

 

–

 

class Boid {
// main fields
PVector pos;
PVector move;
float shade;
ArrayList<Boid> friends;

// timers
int thinkTimer = 0;

Boid (float xx, float yy) {
move = new PVector(0, 0);
pos = new PVector(0, 0);
pos.x = xx;
pos.y = yy;
thinkTimer = int(random(10));
shade = random(255);
friends = new ArrayList<Boid>();
}

void go () {
increment();
wrap();

if (thinkTimer ==0 ) {
// update our friend array (lots of square roots)
getFriends();
}
flock();
pos.add(move);
}

void flock () {
PVector allign = getAverageDir();
PVector avoidDir = getAvoidDir();
PVector avoidObjects = getAvoidAvoids();
PVector noise = new PVector(random(2) - 1, random(2) -1);
PVector cohese = getCohesion();

allign.mult(1);
if (!option_friend) allign.mult(0);

avoidDir.mult(1);
if (!option_crowd) avoidDir.mult(0);

avoidObjects.mult(3);
if (!option_avoid) avoidObjects.mult(0);

noise.mult(0.1);
if (!option_noise) noise.mult(0);

cohese.mult(1);
if (!option_cohese) cohese.mult(0);

stroke(0, 255, 160);

move.add(allign);
move.add(avoidDir);
move.add(avoidObjects);
move.add(noise);
move.add(cohese);

move.limit(maxSpeed);

//shade += getAverageColor() * 0.03;
shade += (random(2) - 1) ;
shade = (shade + 255) % 255; //max(0, min(255, shade));
}

void getFriends () {
ArrayList<Boid> nearby = new ArrayList<Boid>();
for (int i =0; i < boids.size(); i++) {
Boid test = boids.get(i);
if (test == this) continue;
if (abs(test.pos.x - this.pos.x) < friendRadius &&
abs(test.pos.y - this.pos.y) < friendRadius) {
nearby.add(test);
}
}
friends = nearby;
}

// float getAverageColor () {
// float total = 0;
// float count = 0;
//for (Boid other : friends) {
// if (other.shade - shade < -128) {
// total += other.shade + 255 - shade;
//} else if (other.shade - shade > 128) {
//total += other.shade - 255 - shade;
//} else {
//total += other.shade - shade;
//}
//count++;
// }
// if (count == 0) return 0;
// return total / (float) count;
//}

PVector getAverageDir () {
PVector sum = new PVector(0, 0);
int count = 0;

for (Boid other : friends) {
float d = PVector.dist(pos, other.pos);
// If the distance is greater than 0 and less than an arbitrary amount (0 when you are yourself)
if ((d > 0) && (d < friendRadius)) {
PVector copy = other.move.copy();
copy.normalize();
copy.div(d);
sum.add(copy);
count++;
}
if (count > 0) {
//sum.div((float)count);
}
}
return sum;
}

PVector getAvoidDir() {
PVector steer = new PVector(0, 0);
int count = 0;

for (Boid other : friends) {
float d = PVector.dist(pos, other.pos);
// If the distance is greater than 0 and less than an arbitrary amount (0 when you are yourself)
if ((d > 0) && (d < crowdRadius)) {
// Calculate vector pointing away from neighbor
PVector diff = PVector.sub(pos, other.pos);
diff.normalize();
diff.div(d); // Weight by distance
steer.add(diff);
count++; // Keep track of how many
}
}
if (count > 0) {
//steer.div((float) count);
}
return steer;
}

PVector getAvoidAvoids() {
PVector steer = new PVector(0, 0);
int count = 0;

for (Avoid other : avoids) {
float d = PVector.dist(pos, other.pos);
// If the distance is greater than 0 and less than an arbitrary amount (0 when you are yourself)
if ((d > 0) && (d < avoidRadius)) {
// Calculate vector pointing away from neighbor
PVector diff = PVector.sub(pos, other.pos);
diff.normalize();
diff.div(d); // Weight by distance
steer.add(diff);
count++; // Keep track of how many
}
}
return steer;
}

PVector getCohesion () {
float neighbordist = 50;
PVector sum = new PVector(0, 0); // Start with empty vector to accumulate all locations
int count = 0;
for (Boid other : friends) {
float d = PVector.dist(pos, other.pos);
if ((d > 0) && (d < coheseRadius)) {
sum.add(other.pos); // Add location
count++;
}
}
if (count > 0) {
sum.div(count);

PVector desired = PVector.sub(sum, pos);
return desired.setMag(0.05);
}
else {
return new PVector(0, 0);
}
}

void draw () {
for ( int i = 0; i < friends.size(); i++) {
Boid f = friends.get(i);
stroke(90);
//line(this.pos.x, this.pos.y, f.pos.x, f.pos.y);
}
noStroke();
fill(shade, 90, 200);
pushMatrix();
translate(pos.x, pos.y);
rotate(move.heading());
beginShape();
vertex(15 * globalScale, 0);
vertex(-7* globalScale, 7* globalScale);
vertex(-7* globalScale, -7* globalScale);
endShape(CLOSE);
popMatrix();
}

// update all those timers!
void increment () {
thinkTimer = (thinkTimer + 1) % 5;
}

void wrap () {
pos.x = (pos.x + width) % width;
pos.y = (pos.y + height) % height;
}
}

 

– CONNECTING EVERYTHING TOGETHER – 

Theoretically, now everything is supposed to work. The circuit playground connected to the agar agar would trigger the capacitive touch capabilities. The arduino in the circuit playground would allow Processing to read the inputs from the sensors of the circuit playground and trigger the KeyPressed function on the virtual piano, thus generating sound and a boid.

However, what we failed to take into account was how the arduino hardware, while registering the touch on the sensor, does not work the same as Adafruit’s sensor on their MakeCode system. Instead of registering the touch on the agar agar jelly through the wires, the system only registered the “touch” of the crocodile clip. Thus constantly generating boids when connected.

CONCLUSION

Despite the ultimate failure in getting a completely working system, I finally feel like I’m beginning to have a better grasp on the ABCs of coding. Especially since I was in charge of all the coding on Processing, I had to watch a lot of random videos and tutorial videos in order to understand briefly the original source code I found.

The feeling I get about coding is not unlike learning a new language. Where one’s “writing” can only be improved by analyzing and breaking down already completed “literary works”.

It’s for this reason that I am really thankful for the amount of people who spent the time to upload videos to teach newcomers who like me, were undoubtedly lost in the sea of new jargon and phrasings. Since young, I never had a good brain for math and physics, but having to understand how the boids work and to manipulate them to act autonomous saw me refreshing the unused knowledge I was forced to learn as a kid.

I am also thankful to people like Daniel Schiffman and Craig Reynolds that openly share their base code so that noobs like me can slowly break it down and understand how everything works by tinkering around with it.

Overall, I still believe I still have a long way to go, but the fact that I was able to even understand what I was presenting and recode the base code to suit what my group needed gives me hope that maybe even I can code. The possibilities of code are endless and I hope I will be able to continue learning more.