by mert.altinoez -

16 | Arduino with TouchDesigner 🌱

Although I initially planned to delve deeper into various forms of data visualization in TouchDesigner, I decided to postpone that. Instead, I set out to connect a controller to the software. Why? Because I want to create an interactive experience, currently focusing on data visualization. For this, a user input method is necessary, which ultimately makes the experience interactive.

The Controller

I would have preferred to build the controller myself, but I simply don’t have the time for that anymore. This is, of course, my own fault, as I haven’t spent enough time on this project in the past few weeks. Nevertheless, I received a great prototype controller from Lucas, which we developed for another project, or rather he developed. The controller consists of an ESP8266 12-F board, an HW-040 Rotary Encoder, and a 3D-printed knob. The Rotary Encoder allows for changing values with the rotation function and also includes a button. With the help of ChatGPT, I wrote an Arduino script for the controller that reads the rotation speeds and the button state. I then integrated the controller with the DAT components in TouchDesigner and converted the numerical values using the Math functions. Finally, I connected the converted numerical values to the controls of the particle system, allowing the particle system to be manipulated with the controller.

This setup proved to be quite efficient. The integration process went smoothly, and it was fascinating to see how the physical interaction translated into changes within the digital particle system. It brought a tangible aspect to the data visualization, making the experience much more engaging and interactive.

Arduino Controller made of an ESP8266 12-F board, an HW-040 Rotary Encoder, and a 3D-printed knob.
Arduino Controller made of an ESP8266 12-F board, an HW-040 Rotary Encoder, and a 3D-printed knob.

Particle System

To test everything, I used the galaxy visualization and defined the following states. The neutral state is the visualization shown in the last blog post, which represents a galaxy with a moderate rotation speed. If the specified values are „bad“ / negative, the galaxy rotates faster and shrinks, representing the phase just before a galaxy’s death. With good values, the galaxy rotates slower and is significantly larger, representing a healthy galaxy. I also wanted to integrate the button function to reset the display, but after three hours of work, it didn’t function, so I left it out.

Even without the reset function, the current setup is quite versatile. The ability to manipulate the galaxy’s behavior based on input values adds a dynamic element to the visualization. It’s a compelling way to illustrate how different data sets can affect the same system in various ways.

TouchDesigner Visuals (left to right: good state, neutral state, bad state)
TouchDesigner Visuals (left to right: good state, neutral state, bad state)

What’s Next?

In the next step, I want to explore various forms of data visualization and find some examples. This will help me understand the different possibilities and how they can be integrated into my project. Expanding my knowledge in this area will enable me to create even more sophisticated and engaging visualizations. Cheers!

by ingrid.hjelm-hansen -

Findings from User Tests

I recently conducted user tests on my technical prototype. The prototype does not yet work exactly as I vision my final product, but I find it important to involve users at an early stage. I wanted to test the core idea and get valuable feedback for further development of the concept. 

I chose to conduct the user tests with people I interviewed in my research phase. They are women in the age range 23 to 26 and went to music lessons for 1-3 years as children. They have been playing different instruments such as piano, guitar, drums and the clarinet, but never for a long period of time. To learn more about their musical background, read my blog post from January: Key Findings from Interviews. 

User test 1

Successful melodies: 4

Observations

  • Wants to test all the buttons before starting the game
  • Tries to sing the melody outload right away after hearing it
  • Points with her fingers on the buttons
  • Uses time to think before trying
  • Improved skills after every try
  • Gets frustrated when failing, but always wants to try again
  • Missing a replay button, to hear the melody again without needing to play it

Playing when looking at the serial monitor (printed notes): much easier, higher level of success. Wants to retry every time she fails. 

General feedback

  • Thought it was very fun to play
  • Felt competitive, did not want to quit
  • Found it very annoying to fail
  • In the beginning, she did not understand that every melody was not necessarily containing all the tones.
  • Low quality speaker makes it hard to separate the tones. Suggests using Max 8 instead of the piezo buzzer. 
  • The “incorrect melody”-sound is similar to some of the melody tones. This could be distracting and confusing. 
  • After successfully playing a melody, the next one plays right away. This was too fast for her to prepare for listening again.
  • If I want to upgrade the product and make it even harder, it could be an idea to also implement different rhythms in the melody. Another option is to make the melodies longer (more than four tones). Nevertheless, she states that it was already complicated enough for her. 

User test 2

Successful melodies: 1

Observations

  • Wants to start right away, before I am finished with the explanation
  • States that she is terrible, does not know anything about music theory
  • Struggles, but learns quickly
  • Notices that she is playing the melody wrong, but struggles to point out what the problem is
  • Decides to give up after numerous attempts on the second melody

Playing when looking at the serial monitor (printed notes): easier, but still needs to think a lot and use multiple attempts to succeed. 

General feedback

  • Said it was very fun, would love to play with it all the time as a child. 
  • Very nice way to train your ear. 
  • Thought it would be easier with a higher quality speaker. 
  • Suggested implementing an orange light for better feedback. It would make it easier to understand the number of wrong notes.  
  • Wanted to try again every time she failed, but it was easy to get stuck on one melody and become annoyed. 
  • Hard to hear what is wrong. 
  • Missed a replay button. 
  • Suggested writing the name of the tones on the buttons. Thinks it would be easier to understand the connections. 
  • Suggested removing the resistors to get brighter LED lights.

Conclusion

In general, it was very helpful to test my concept in such an early stage. Their positive feedback motives me to develop the idea further in the future, and their critique made it clear what changes I should make. I already did some adjustments: 

  • Changed the pitch on some of the tones
  • Increased delay between feedback and new melodies
  • Removed resistors from the breadboard
  • Adjusted the printed messages to the serial monitor
by ingrid.hjelm-hansen -

Development of Technical Prototype

Over the past few weeks, I have been developing a technical prototype. I explored various options considering the available equipment and project timeline. After many iterations and debugging, I created a functional prototype ready for user testing. 

The prototype is a simplification of my concept. It can be used to illustrate the main functionalities, but does not work exactly how I vision my finished product. I chose to follow the advice of making a low effort prototype with high indented impact on learning outcome. 

Main components: 

  • Random melodies generated out of four tones (C, D, E, F)
  • Four buttons for user input (C, D, E, F)
  • Green LED light for positive feedback
  • Red LED light for negative feedback
  • Piezo buzzer playing melodies, user input and feedback sounds

How does it work?

Instead of using a microphone for user input, I decided to work with buttons. A piezo buzzer (speaker) plays a random melody composed of four tones, and the user can use buttons to replicate the melody. In my programmed prototype, I used the tones C3, D3, E3, and F3. Each melody consists of four tones, which is always a random combination of the above. The buttons represent one tone each:

Yellow button: C3
Green button: D3
Blue button: E3
Black button: F3

Operation flow

  1. Melody playback: The speaker plays a random melody consisting of four tones.

  2. User interaction: After the melody finishes, the user can replicate it by pressing the corresponding buttons.

  3. Feedback mechanism:

Correct replication: A short positive sound is played, and the green LED lights up, signaling success. A new random melody then follows.

Incorrect replication: A negative sound is played, and the red LED lights up. The same melody is repeated until the user successfully replicates it. 

Code written in Arduino IDE:

by ingrid.hjelm-hansen -

How To Build the Prototype

To be able to user test my concept, I want to create a prototype with a certain technical functionality. Based on ease of use and personal prior knowledge, I decided to use Arduino to test the desired functions. 

Required equipment

  • Arduino Uno
  • Battery pack
  • Breadboard
  • Buttons
  • LED lights
  • NeoPixel ring
  • Electret microphone
  • Piezo speaker
  • Wires
  • Resistors

To begin the prototyping, I watched this tutorial on YouTube: https://www.youtube.com/watch?v=bMs5J4bJOD0. It shows how to connect a microphone to LED lights for instant feedback in an Arduino setup. I thought this was a good way of starting the building of my prototype, because it contains some of the major functionalities of my concept. 

I used the online web-based tool on tinkercad.com to build my Arduino setup and write the code. This is how it looks like so far:  

In the next step, I will research how to add a speaker. This tutorial shows a setup with both a microphone and speaker: https://www.youtube.com/watch?v=nIDhkvomrcg. It will hopefully help me on the way. If I can access the required equipment, I will also attempt to build it in a physical format. 

Further, I need to figure out how to generate random melodies and play them on the Arduino speaker. This website shows examples on how the melodies could sound: https://random-music-generators.onrender.com/melody. I ideally want the tempo parameter to be around 70, and the number of notes to be 4. 

For visual purposes, I also want to build a non-technical prototype to demonstrate the indented design in the end. The two prototypes will complement each other, and optimally be merged together in the final prototype video.