by Mahtab Jafarzadehmiandehi -

Audio Programming & Mapping Process

So the audio programming process basically took the entire semester, and it continues to be a focus as I refine it to better align with the compositional idea and ensure the patches are as practical as possible.

I started by mapping the sensors in Max MSP. Each sensor can be mapped to seven parameters using MIDI numbers, the 7th parameter is a measure of total acceleration obtained directly from the SOMI-1 receiver. It’s calculated using a Pythagorean formula based on acceleration data, excluding gravitational force to improve accuracy.

Figure I. Movement parameters of SOMI-1 around the X, Y, Z axis

What I learned from the mapping process is that even though I tried to be specific with the movements, generally you cannot isolate just one of these parameters while moving. This is a key difference I need to consider when mapping movement sensors compared to other stable

MIDI controllers. To effectively map the sensors and keep in mind the application of movements, I divided the motions into the 7 parameters for each sensor:

  • Rotation X
  • Rotation Y
  • Rotation Z
  • Acceleration X
  • Acceleration Y
  • Acceleration Z
  • Total Acceleration

   Figure II. An overview of sensor mapping in Max MSP

After completing the initial movement mapping, I began creating patches for the interaction part using the aforementioned list. This process is ongoing and will likely continue until the project’s completion. Meanwhile, a crucial aspect I am keen to focus on this summer is implementing patches to the sensors and testing their compatibility both independently and with the violin.

Throughout my learning, I am aware that due to the violin, my movement mapping is limited, and I am still trying to figure out how I can wear the sensors either on my hands or elsewhere, but it is also clear that the majority of the mapping is for the right hand or for the bowing part. However, there is also the possibility to map the sensors not only to the usual gestures that occur while playing the violin but also to some unconventional movements to trigger certain parameters, which would require more precise mapping in this case. In general, there are a few possibilities for the mapping process that I need to consider and examine thoroughly.

There are several types of mapping strategies that can be employed, regardless of whether the relationship between the control input and the parameter output is linear or non-linear:

One-to-One Mapping: This is the simplest form of mapping where each control or sensor input directly corresponds to a specific parameter or output

Multi-Parameter Mapping: This approach allows a single sensor input to influence several aspects of the sound to control multiple parameters simultaneously or sequentially. There is also this possibility to change the works of the sensors via pedal to have the combination of different tasks for the sensors

What I also have in mind is to avoid counter-intuitive mapping, which involves controlling parameters in unexpected ways, as it adds an element of unpredictability to the performance. I believe this is unnecessary for the project. Instead, my goal for the mapping part is to ensure that my movements have a clear and meaningful relationship with the parameters they control.

by Mahtab Jafarzadehmiandehi -

Types of Interactions

After finalizing the compositional idea, I attempted to compile a list of all the signal processing types and audio effects that I intend to use in my piece. To practice with fixed elements, the aim was to control the parameters rather than randomize them, so that I can bring the project closer to what I expected and make it executable as well. In other words, I determined which aspects should be under my control and which should not.

Throughout the semester, I experimented with these different types of interactions to explore my sound possibilities and predict how they would affect the sounds of the violin.

  • Multiple Loops (evolving musical patterns) On/Off through monitor?
    On/Off via Sensors? It needs more precise and complicated patches, and the patches should track unnormal gesture and not the way I usually play with violin
  • Audio Signal Processing Pitch shifting
    Time stretching Amplitude/Dynamic Reverb/Delay Timbre
  • Microsoft Word – Exposé II.docx
  • Chorus: to create a thicker sound by duplicating the signal and slightly varying the pitch and timing
  • Sound Synthesis Variations Additive synthesis
    FM (frequency modulation) synthesis Granular synthesis
  • Granular Synthesis
    Grain size:     shorter grains noisier and more textural, longer grains are more faithful to the recorded sounds of the buffer Grain Density: higher density sounds thicker and more continuous, lower density scattered texture and more noticeable individual grains Grain shape (windowing function): the envelope applied to each grain to shape its amplitude over time, currently I am using the Hamming window Grain position (start time): the start pointing of each grain within the original audio sample Grain playback direction: forward or backward!
    Grain spatialization: just have an idea that the grains move around the listeners from everywhere like a rain!
    Grain sequencing: different order of the grains’ playback for more chaotic textures Randomize the parameters: Which is not my goal, but it’s also another possibility!
  • Spatialization The goal is to begin with headphones that track movements and initially use a binaural mix. Additionally, I plan to explore spatialization, utilizing various IEM plugins, including those designed for granular purposes.

by Mahtab Jafarzadehmiandehi -

Compositional Aspect

As I started with audio programming, it felt like I walked into an unknown world with many possibilities, and it wasn’t clear to me which direction I should take. I came across a conference proceeding on New Interfaces for Musical Expression by Perry Cook15, which helped clarify my initial steps. In the conference paper mentioned that, “Make a piece, not an instrument or controller.” So, I decided to come up with the idea of a piece or specific musical composition because I didn’t want to create interesting research questions without a real product or future direction.
During the semester, I came up with two compositional ideas, and I aim to work on both of them since the interaction parts between them are not significantly different.

Piece No. I
The first concept comes from my own experience as a classical violinist who transitioned into electronic music. The overall idea involves contrasting aspects of my experience, demonstrating how every action reflects on other elements, and showcasing the complexity of identity. This piece has three sections:

Section I
1. Introduction
2. Then it goes to looping and layering
3. Atonality
4. Dissonant Intervals
5. Full of aggressive bowing
6. Using violin extended techniques
• For the transition part from section I to section II, I will either put the violin aside or have it in my hands but not playing it, instead I will just control the recorded sound and manipulate it with the sensors.
• Then will control the amplitude and do a smooth decrescendo to be prepared for the second section.

Section II
1. After some choreography, build the contrast on top of distorted and fading out loop
2. Play tonal patterns
3. With clear sentences
4. Subtle movements
5. Everything in this section is about contrast
6. Probably no loops at all!

Section III
1. Back to the chaos but this time overlapping and layering consonant intervals and melodies
2. End the piece immediately!
So the concept for this piece is mostly based on multiple loops, evolving patterns, and layering them to create a some sort of polyphony. The goal is to alter these layers through various types of interaction to showcase the role of the sensors. I will also build a contrast between the manipulated sounds and the raw sound of the violin, primarily in section two, and then take ideas from each of the two sections to build the final section based on that, for the conclusion part.

Piece No. II
The second piece also evolves from the first one, but with a difference: we start with the violin just for the beginning, and then the focus is more on the transition from acoustic to electronic sounds.
1. Playing violin melodically and record it into the buffer and create loops
2. Put the violin aside
3. Wear the sensors
4. Start with different types of interactions step by step
5. The end consists of many synth loops blending into each other
by Anna Semmelrath -

Interactive Audio Drama – Progress so far

The work for my interactive audio drama project is running, but also hitting some problems at the moment.

The process of getting the Bluetooth beacons to work is a slow one. Some problems came up, like that the beacons turn themselves off after a while without using. Also, the open-source App which is for receiving Bluetooth data, which I thought I could use, is now not possible to use, as it is only available for windows devices and not android and iPhone. Luckily I can get some help with the coding, as this is not my area of good knowledge. In the best case we would write our own App which is capable of receiving on processing the input from the beacons. And also has an easy user interface to have small interactions via the App, a little bit of gamification, so that the player is able to now and then choose between answers or but in some information like a code word.

The draft of the story plot is ready. The story combines fiction with a bit of history. In short words, it is based in the Dr. Who universe. The player travels back in time, meets Johannes Kepler (who lived in Graz from 1554 to 1600) and has to fix the timeline, cause something is very wrong.

The intro to the story is written in script form already. I am getting help with the writing of the script, so I can focus more on the recording and the sound design.  

I recorded the first part of the intro, to test if the script works like it is and to come up with some sound design ideas.

The next steps are, to find solutions for the beacon problem, record the next track as a test track, and create the map.  

by Anna Semmelrath -

Interactive Audio Drama

How should it work:

The story is intended to be played/experienced interactively to a certain degree. There should be an app, and it will be played using one’s own phone and headphones. There should be about 6 locations (more or fewer if necessary) that are relevant to the story, such as the well, the marketplace, etc. These locations will be displayed on a map, and one can navigate to them. Bluetooth beacons will be placed at these locations to trigger audio when one is close enough. It should be located in Stadtpark and Schlossberg to avoid streets and traffic.

The app will have its own interface for maps. Using the phone’s GPS, one can navigate on the map. The whole story is intended to take place in Graz.

Based on clues from the story, the player will have options to decide whether to go to location A or location B next to hear the next part of the story or get new clues relevant to solving the task.

It should be possible for the audio to be triggered only when it fits the story’s progression. That is, if someone passes by a location too early, and it wouldn’t make sense to hear something yet, it won’t be triggered.

There could also be a time limit to hear something at a certain location. For example, the player needs information from Person X at the well and Person Y at the marketplace, but one of them is about to leave soon, and the other is badly injured (for example), so there is a time constraint, and the player must decide where to go. Based on the story so far, the player might know which person is more relevant.

At the end, if the player succeeds, the story is resolved, they travel back to their time and they have either fixed their time completely, just a little, or not at all.

by Hannah Albrecht -

“ontextC” – Technical Diary 4

What happened so far?

To know where to start modifying the Max phase vocoder, I drew comparisons between the same stretch factors in PaulXStretch and the phase vocoder. To keep the conditions as similar as possible, I changed the FFT size in PaulXStretch to 1024 and turned off all of the other parameters in the processing chain (harmonics, tonal vs. noise, frequency shift, pitch shift, ratios, spread, filter, free filter and compressor), with the expectation that the resulting sounds would just stretch the source sound (a ten second snippet from an acoustic multitrack recording) using the respective stretching algorithm. This would then allow me to hear differences.

When comparing the results, it quickly became evident that while the phase vocoder provided very transparent sounding stretches at lower stretch factors, the aesthetic quality of the Paulstretch algorithm and the smearing it introduces were a) very different sounding and b) more usable for the intended sound design purposes, where especially stretch factors over 10 become interesting and the original sound source becomes almost unrecognisable.

Note: I have now switched to working with the default phase vocoder that comes with Max as a resource example (Max 8 > Show package contents > Resources > Examples > fft-fun > phase-vocoder-example-folder). It has a lot of similar components.

Ongoing

Currently I am in the process of settling on EQ, reverb and pitch shifting modules to use for the prototype. Another more research-based aspect of the project is to figure out how the provided Python code from the old Paulstretch algorithm works, which will hopefully allow me to modify the phase vocoder towards a direction that suits the imagined aesthetic outcomes of ontextC. My supervisors are kindly is helping me with this, since I am not familiar with Python at all.

Results and Reflection

The results of the comparison are useful, because they define the differences that need to be overcome in order to reach the aesthetic results I am looking for with this plug-in. While some of the inner workings of the Paulstretch algorithm still remain unknown as of now, the Python code will hopefully help to figure out what is missing. Furthermore, being able to set the FFT size over 2048 to a value closer to a value along 4400 would be a next step to imitate the workflow that started this project better – the steps that follow will show whether that is a limitation in Max or not.

As a sidenote: The shortcuts CMD + Option + M to open a locked Max patch and CMD + 8 to remove the frame have been proven very helpful.

Objectives for Next Time

  • Prep draft of sound example through all parts of the signal chain -> How does it start, which sound do we want to get to?
  • Check out Phase vocoder template, start to modify parameters in project draft and experiment
  • Settle on other modules in the processing chain

Keep in Mind: Mapping parameters together will become relevant sooner rather than later – it makes sense to research this as well.

by Hannah Albrecht -

“ontextC” – Technical Diary 3

What happened so far?

A recent priority was the comparison of different phase vocoders that are available in Max. With the help of the Cycling74 resources, I tested whether the difference between the modules using polar vs. cartesian coordinates affected my sound sources in a (noticeable) way that would make me choose one over the other – ultimately cartesian coordinates seemed like the better option for my project, also in terms of CPU usage. For windowing, the Hanning window is currently in use.

Furthermore, to better understand the processes the signal goes through within the plug-in, I asked my supervisor about the meaning of phase coherence in this context, and was able to bit by bit (little terminology reference here) connect the theory and the practical application, which will help me a lot going forward.

Ongoing

The evaluation and development of EQ, pitch shifting and reverb modules for my project is ongoing. Fortunately, there are a lot of libraries and resources especially for filtering and spatial effects, so the main challenge here is to find what works best to achieve the sound results I am aiming for, while also being functional and relatively simple to integrate. By studying existing Max patches, even though they might not be 100% what I am looking for, I am learning more not just about the Max environment, but also about best practices and how I could translate certain organisational aspects (comments are so helpful for external people looking at a patch to know what is going on!) and connections into my own project patch. My main resources for this are free patches that I download from the Max for Live library patch page and explore.

Results and Reflection

While it is good to know that there is a phase vocoder that can help me to realise my vision for this project, now it is time to start thinking about how to best integrate it, and define which modifications need to be made in order to make it sound the way I want it to in the context of my project. To do so, I will draw comparisons between PaulXStretch and the Max phase vocoders, to determine limitations, potential areas of improvement and differences in sound quality at different stretch factors.

Objectives for Next Time

  • Prepare and document sound examples to compare between the phase vocoder and PaulxStretch
  • Continue development of other modules
by Hannah Albrecht -

“ontextC” – Technical Diary 2

What happened so far?

Aside from a crude mockup in Max MSP, a diagram helps envision the signal flow and processing points of the plug-in now. The diagram is also quite a handy tool to identify challenges, as it lays the main idea out in a layout that is simplified, but representative of the core idea. Parameters have been defined and narrowed down further.

I have also been provided with copies of all three volumes of Electronic Music and Sound Design – Theory and Practice with Max 8, which I am using as a reference and also a learning opportunity to further familiarise myself with the Max environment.

The objective at this is to research and further refine the direction of the project. At this point, the audio signal chain has the potential to work, but the time stretch unit does not work by integrating PaulXStretch into the patch as an external VST, since the audio needs to be manually imported and exported in the application.

Top Objects

In the mockup, the bangbang object proved very useful to initiate the loading of a list of parameters in a umenu – to experiment, this was done with a list of parameters from Valhalla Supermassive, but the same procedure could be useful later down the line for menus that should operate similarly.

Results and Reflection

The biggest challenge at the moment is the PaulXStretch implementation. The lack of documentation of the application makes it difficult to decipher which algorithms make the parameters work, and since it is at the top of the signal chain it blocks the audio signal from coming through to the next stages of processing. More research on the Paulstretch algorithm will be necessary. Furthermore, the commercial nature of my ideal reverb for this project makes it more difficult to implement, meaning that now is a good point to look into alternatives and emulations.

Objectives for Next Week

  • Research reverb properties, documentation, and open source emulations/alternatives
  • Research publications on the Paulstretch algorithm
  • Find a good tool for pitch-shifting and EQ

Research Resources for Next Week

Timbral effects the Paulstretch audio time-stretching algorithm (Colin Malloy)

An approach for implementing time-stretching as a live realtime audio effect (Colin Malloy)

Max 8 Handbooks (Volume 1-3) – Alessandro Cipriani, Maurizio Giri

Valhalla Lear Resources (Plug-In Design)

by Anna Semmelrath -

Dromos/Autos The Autistic Ontology as Performance

This presentation, researched and performed by Matthew Rogerson, took place during the IRCAM Forum Workshops in Paris in March 2024. Rogerson took an approach to showing the possible ‘insides’ of an autistic brain through neurofeedback, visuals and sound. Through an EEG, he picked up electric brain signals that were sent through Max which triggered those sounds and visuals. The project included aspects of generative electronic music, psychoacoustics, audio-reactive visuals and performance. He kind of was the human interface, creating this performance through his brain waves.

Flashing images, strong light flashes in a darker room, distorted sounds, white noise and high-pitched parts. Provocative was his intention. In the context of this performance, the demonstration reminded me of a swamped brain with too much information, unable to turn off. He told us that it should represent sensory overload, which is common in autism. He wanted to bring across a feeling of, and a representation of how an autistic person could possibly experience the world. Bits of speech were included, but the semantic qualities were put away in this project to show how an autistic person could perceive speech.

The delayed reaction from the brain signal to the output corresponded to the feeling he described, when going out of the house, every day it feels like entering a new country, which takes up way more processing power in the brain than things we are used to.

While performing, he tried to be as neutral and passive as possible, although he said a feedback loop was created. His reaction to the outcome, the flashing lights and provocative sounds, creates more chaos in his mind, which creates a more chaotic outcome. So the circle continues.

by Anna Semmelrath -

Making Electronic Music Inclusive: A Virtual Studio for Visually Impaired Composers

This presentation by Joseph Butch Rovan took place during the IRCAM Forum Workshops in Paris in March 2024.

For this project, Joseph Butch Rovan works together with a visually impaired composer. He addresses the issue that existing tools for electronic composing using interactive technologies such as Max/MSP present a barrier to visually impaired composers. While graphical patching software aims to make it easier for the user, they create obstacles for blind people and makes it even more user-unfriendly for them. In this presentation, Rovan, introduces an interface and Max programming environment specifically crafted to be fully accessible to visually impaired composers.

While working with the composer, he realized that screen reading is absolutely no help when it comes to complex audio programming software. Also, interfaces with symmetric buttons are not optimal for visually impaired people, as they have to count in order to know the location of their hand. His interface is crafted towards blind people, it is equipped with tactile and audio feedback.

It has different types of pots, like buttons, switches and dials, which are easy to differentiate. Potentiometers are in specific groups to make them easier to locate. The interface works with audio feedback, which can be put on a different channel for personal feedback or can be turned off. With the audio feedback, the user can go through menus before working with the pots.

This project enables visually impaired people to work with electronic music composition in a way that they can access tools that sighted people use on a regular basis.

The composer Rovan worked with, successfully held a concert with this tool using her own presets and voice.

This system not only helps blind artists but also opens new creative avenues for sighted composers.