Interactive Devices: Multimodal Sketch (Arduino)

Standard

Project done by: Emma, Wei Lin, Natalie, Wan Hui

Our multimodal project is called Feel My Message. This project aims to inform a person about the content of a received telegram message without being disruptive to his entourage.

 

CONTEXT:

Our device is ideally an unobtrusive small box that can be placed on the table or in one’s pocket. We envisioned it to be used when one is unable to check their phones or laptops for messages during a meeting. However, in our prototype, our device is bigger than what we thought due to hardware and financial constraints.

 

HOW THE DEVICE WORKS:

The device will first receive messages sent to the user from someone else via our telegram bot. Then, two separate dials within the device will rotate to indicate who the sender is and the content of their message. This information is depicted through tactile symbols that the user will be able to touch with a finger and recognize upon contact. To make this device universal, the symbols that represent the message content is changeable to better suit the needs of different users. Users can also assign their own contacts to the dial that represents the sender of the message.

 

PICTURES OF THE UNCOMBINED PROTOTYPE:

For the symbols, we had prior prototypes before we decided on these symbols. We used plastic poly pellets to make the symbols because it was more forgiving when it came to the experimental stage because we could easily remould it, as we were still trying out which symbols worked best for what kind of message.

Before we settled on using a mould, we tried to shape the poly pellets into the symbol itself, but we realised that the symbols didn’t feel very obvious because it was too smooth to the touch. By using the mould, the finish was rougher and that was more obvious, hence we settled with using a mould.

 

DEMO:

HARDWARE:

  • Arduino Uno Microcontroller
  • ESP32 wifi and bluetooth board
  • 2 x 360° continuous servo motors (MG90S + MG995)
  • Vibration motor
  • Breadboard

SOFTWARE:

  • Arduino IDE
  • Telegram Bot Maker

 

SETTING UP THE TELEGRAM BOT:

 

CHALLENGES:

In the beginning, we tried using an ESP8266 wifi module with Arduino Uno, with the software Blynk to Arduino IDE. However we could not get the ESP8266 wifi module to work despite numerous attempts. It would not connect to the wifi or work with Blynk.

Afterwards, we bought an ESP32 wifi and bluetooth board. We tried having two separate codes on the ESP32 and Arduino Uno. We used the Arduino Uno to control the 2 servo motors (5V) and ESP32 to run the main code. We had to do this because the ESP32 runs on 3.3V while the servos run on 5V, so the ESP32 board did not have enough power to run the servos and the vibration motor. But by doing so, we were unable to get the Arduino Uno to communicate with the ESP32 via serial communication, because they were running on different baud rate (9600 vs 115200) and if we uploaded the code via the Arduino Uno, it would be unable to access the ESP32 library.

We attempted to connect the servos to the 5V or 3.3V pin on the ESP32. We were able to receive the messages, but yielded a weird error on the serial port, and the servos were unable to move. We researched that we needed a 5V to 3.3V logic leveler. However, we did not have a logic leveler available to us, so we had to think of an alternative.

In the end, our solution was to combine both our telegram bot code and servo motor codes into one main code, upload onto the ESP32 and use it as our main micro controller unit, then power the servos with a separate power source. We utilised the Arduino Uno as the 5V power supply through a breadboard. For this to work, the ESP32, as well as our other hardware had to all be grounded on the same line.

(how the wires are connected)

Now, there was sufficient power for the servo motors, we then moved on to combine the codes, making sure that the servos were working properly based on the received messages. Here is our final combined code.

 

FINAL COMBINED CODE:

#include <ESP32Servo.h>

#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <UniversalTelegramBot.h>

static const int outerservoPin = 16;
static const int innerservoPin = 17;

Servo innerServo;
Servo outerServo;

int vMotorPin = 18;
bool ended = false;
String text, from_name;
int person = 0;
int msgsent = 0;


// Initialize Wifi connection to the router
char ssid[] = "XXXX"; // your network SSID (name)
char password[] = "XXXXXXXXXXXXXXXXXX"; // your network key

// Initialize Telegram BOT
#define BOTtoken "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

WiFiClientSecure client;
UniversalTelegramBot bot(BOTtoken, client);

int Bot_mtbs = 1000; //mean time between scan messages
long Bot_lasttime; //last time messages' scan has been done
bool Start = false;

void handleNewMessages(int numNewMessages) {
// Serial.println("handleNewMessages");
// Serial.println(String(numNewMessages));

for (int i=0; i<numNewMessages; i++) {
String chat_id = String(bot.messages[i].chat_id);
String text = bot.messages[i].text;

String from_name = bot.messages[i].from_name;

if (from_name == "Wan"){
person = 1;
}
if (from_name == "Emma"){
person = 2;
}
if (from_name == "Wei Lin"){
person = 3;
}

if (text == "/love") {
bot.sendChatAction(chat_id, "typing");
delay(100);
bot.sendMessage(chat_id, "Received love!");
Serial.println("lof from " + from_name);
msgsent = 1;
}

if (text == "/happy") {
bot.sendChatAction(chat_id, "typing");
delay(200);
bot.sendMessage(chat_id, "Smile! :D");
Serial.println("happy from " + from_name);
msgsent = 2;
}

if (text == "/angry") {
bot.sendChatAction(chat_id, "typing");
delay(200);
bot.sendMessage(chat_id, "Anger sent!! >:(");
Serial.println("angry from " + from_name);
msgsent = 3;
}

if (text == "/emergency") {
bot.sendChatAction(chat_id, "typing");
delay(200);
bot.sendMessage(chat_id, "Called for help!!");
Serial.println("emergency from " + from_name);
msgsent = 4;
}

if (text == "/home") {
bot.sendChatAction(chat_id, "typing");
delay(200);
bot.sendMessage(chat_id, "Welcome Home!!");
Serial.println("home from " + from_name);
msgsent = 5;
}

if (text == "/start") {
String welcome = "Welcome to Feel My Message, " + from_name + ".\n";
welcome += "Here is a list of commands for you to use :D\n\n";
welcome += "/love - send love 💕 \n/happy - send happiness ☺ \n/angry - send anger 😡 \n/emergency - send sos 🆘 \n/home - send I’m at home 🏠";
bot.sendMessage(chat_id, welcome);
}
}
}


void setup() {
pinMode(vMotorPin, OUTPUT);
innerServo.attach(innerservoPin);
outerServo.attach(outerservoPin);

Serial.begin(115200);

// Attempt to connect to Wifi network:
Serial.print("Connecting Wifi: ");
Serial.println(ssid);

// Set WiFi to station mode and disconnect from an AP if it was Previously
// connected
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) {
Serial.print(".");
delay(500);
}

Serial.println("");
Serial.println("WiFi connected");
Serial.print("IP address: ");
Serial.println(WiFi.localIP());

}

void stopturning(){
innerServo.write(93);
}

void vibrate(){
delay(50);
digitalWrite(vMotorPin, HIGH);
delay(500);
digitalWrite(vMotorPin, LOW);
}

void msg1(){
innerServo.write(70);
delay(640);
innerServo.write(93);
vibrate();
delay(5000);
innerServo.write(110);
delay(750);
stopturning();
}

void msg2(){
innerServo.write(70);
delay(1000);
innerServo.write(93);
vibrate();
delay(5000);
innerServo.write(110);
delay(1150);
stopturning();
}

void msg3(){
innerServo.write(70);
delay(1550);
innerServo.write(93);
vibrate();
delay(5000);
innerServo.write(110);
delay(1700);
stopturning();
}

void msg4(){
innerServo.write(110);
delay(1280);
innerServo.write(93);
vibrate();
delay(5000);
innerServo.write(70);
delay(1010);
stopturning();
}

void msg5(){
innerServo.write(110);
delay(750);
innerServo.write(93);
vibrate();
delay(5000);
innerServo.write(70);
delay(620);
stopturning();
}

void sendmsg(){
if(msgsent == 1){
if (ended == false){
Serial.println("msg1");
msg1();
msgsent = 0;
}
}

if(msgsent == 2){
if (ended == false){
Serial.println("msg2");
msg2();
msgsent = 0;
}
}

if(msgsent == 3){
if (ended == false){
Serial.println("msg3");
msg3();
msgsent = 0;
}
}

if(msgsent == 4){
if (ended == false){
Serial.println("msg4");
msg4();
msgsent = 0;
}
}

if(msgsent == 5){
if (ended == false){
Serial.println("msg5");
msg5();
msgsent = 0;
}
}
}

void per1(){
}

void per2(){
outerServo.write(180);
delay(270);
outerServo.write(93);
}

void per2back(){
outerServo.write(0);
delay(270);
outerServo.write(93);
}

void per3(){
outerServo.write(0);
delay(250);
outerServo.write(93);
}

void per3back(){
outerServo.write(180);
delay(260);
outerServo.write(93);
}


void loop() { 
if (millis() > Bot_lasttime + Bot_mtbs) {
int numNewMessages = bot.getUpdates(bot.last_message_received + 1);

while(numNewMessages) {
// Serial.println("got response");
handleNewMessages(numNewMessages);
numNewMessages = bot.getUpdates(bot.last_message_received + 1);
}

Bot_lasttime = millis();
}
if(person == 1){
if (ended == false){
Serial.println("per1");
per1();
sendmsg();
person = 99;
}
}

if(person == 2){
if (ended == false){
Serial.println("per2");
per2();
person = 99;
sendmsg();
per2back();
}
}

if(person == 3){
if (ended == false){
Serial.println("per3");
per3();
person = 99;
sendmsg();
per3back();
}
}
}

Download our code here.

Interactive Devices: LED Room Sketch (Arduino)

Standard

Project done by: Sylvia, Daryl, Wei Lin, Wan Hui

After coming up with the sketches and putting them in Processing, we were then tasked with connecting it to Arduino. We settled with 4 main functions with different triggers.

  • On/Off – Proximity Sensor
  • Brightness – Volume (Microphone level)
  • Colour Change – Compass + Gravity
  • Preset Colour Patterns – Touch 2D + Wekinator

Video Demo

On/Off

We wanted the on/off function to be the most straightforward way, which was for the LED lights to turn on when the phone is faced up (i.e. proximity sensor not covered, giving a value of “false”/”0”), and then for it to off when you place the phone faced down on a surface (i.e. proximity sensor covered, giving a value of “true”/”1”).

 

Brightness

One interesting feature we found on the iOS version of ZIGSIM was “Mic Level”, so we wanted to experiment with it. We figured volume could work well with brightness because it was quite intuitive for the LED to be more intense as the environment gets louder. Additionally, we tested this feature with music, and the results turned out surprisingly well!

 

Colour Change

Essentially, we wanted to use the compass to draw a circle, using the gestures to change the colours while “colouring” the circle. We made use of the Gravity, Quaternion and Compass sensors from ZigSim, linking it through Processing to Arduino. From Processing, the data received from ZigSim is communicated to Arduino and translated to colour codes and number patterning for the Adafruit LED Strip.

 

Preset Colour Patterns

We learnt how to use Wekinator in the previous lesson, so we thought that it would be a good chance for us to apply it in our project. We coded various LED light strip patterns in Arduino, and the trigger for each of the different patterns would be the different touch gestures that we trained in Wekinator. We had 3 gestures to trigger 3 different patterns.

 

Other variations that we tried and tested

  • Saturation – Gravity
  • Colour Wheel – Touch 2D
  • Pattern Change – Accelerometer (Shake)

Saturation

Saturation defines the brilliance and intensity of a color. White and black (grey) are added to a colour to reduce its saturation. Hence, we worked in greyscale (black to white), with the highest point being white and the lowest point being black.

 

Colour Wheel

Using the full dimensions of the screen to map to the colours of the RGB colour palette.

 

Pattern Change

Using accelerometer, if the value reads more than 8, or less than -8, then the device will register the motion as one shake (i.e. “shaketrue”) and the next pattern will play. The shake cycles through the preset patterns.

Click here for our codes.

Interactive Devices: Social Distancing Project (Analog)

Standard

“Face-to-face” done by Nasya and Wan Hui.

“Face-to-face” is an analog device that allows us to interact in a time of social distancing. Social distancing helps limit opportunities of healthy people coming into close contact with sick people, which reduces opportunities for disease transmission.

With “Face-to-Face”, it forces you and your friend to stay at least 1 metre apart, in order for both of you to be able to fully see each other’s face in the mirror. At the same time, this allows you to still keep the “closeness” of a face to face conversation as you will still be in the same physical space, while also minimising the spread of transmission due to not directly facing each other.

Sketches (Device and how it works):

Structure of the device:

Prototype: made with cardboard, masking tape, reflective paper, and wooden sticks

Video demonstration: Interactive Devices: Social Distancing Project (Video Demo)