by max.fruehwirt - 22. Februar 202422. Februar 2024

10 | Auditory and visual storytelling – a case study on different anime and video games

What this blogpost is all about

As my final entry, I wanted to continue looking into immersion and what may cause it.
As an avid anime and video game fan myself, I have always been fascinated by how some of my favourite pieces of media managed to catch the audience and deepen immersion by not only having a compelling story, but also mange to make use of various different auditory and visual storytelling elements to underline it. This lead to this blog-post, where I want to explore some of my favourite recent examples and explore some ideas, on what makes them work. By looking into these examples, analysing them and learning from their respective examples, I hope to also gain insight into how to make my own work more immersive.
Also, while I will try to keep my entry as spoiler-free as possible, there may be some spoilers for the analysed media, you have been warned!

Case 1 – Sousou no Frieren: A slow burner with a twist

Sousou no Frieren – translation: Frieren at the funeral, official English title: Frieren: Beyond the journey’s end – is a currently airing fantasy anime, that deals with a very simple premise: What happens to the heroes, after the big bad boss at the end of every fantasy story is slain and the world is saved? What comes afterwards, when the journey that should define a lifetime, is over?

In Sousou no Frieren, that is where our story begins, with the hero’s party, consisting of the noble human hero Himmel, the ever-unfazed dwarven warrior Eisen, the corrupt human priest Heiter and the stoic, century old elven mage and titular character, Frieren, having just slain the Demon Lord, thus bringing peace to the lands. And, as the title suggests, Frieren and her emotional journey from this point, is also the focus going forward.

Because, in the first episode already, as the heroes party is celebrating their victory in the capital, they manage to catch a very rare sight: A meteor shower, that only appears every 50 years. It is at that time, that they all agree to see it together next time too, though in a place with a better view, because Frieren remarks, how she knows such a place and that they should just meet. Afterwards, the party splits up, and our elven mage embarks on another journey. It should be 50 years before she sees her companions again, a time, that passes in the blink of an eye, barely even 2 minutes pass in the episode, and it is in that moment, that you, as a viewer begin to realise, how different Frieren perceives time and its meaning. When meeting her friends again, the two humans, Himmel and Heiter, have become old men, barely recognisable from before. However, they all keep their promise and watch the meteor shower together once more. Shortly after, Himmel dies, and it is at his funeral, where the usually stoic and composed Frieren breaks down, crying, wondering, why she feels so sad about someone, that „she only spent a 10 year journey with“, which to her, barely is a blip in her long, long life, but to her human companions, is one of the most important times of their lifes. She regrets, not spending more time with them, connecting more with them, and getting to know them better. It is at this moment, that Frieren decides to retrace her former parties steps and go on a journey of self-discovery, to deal with the question, on what it means, to form a connection, when it will eventually be lost anyway, how precious time together can be and how to deal with loss and her emotions in general.

Duality between Action and Slice of Life

During this journey, Frieren will come to experience the change, that her actions have brought, take on an apprentice, face all kinds of hardships, old and new foes and other battles, and generally grow as a person.

Screenshots from the first 6 episodes of the anime:
Sousou no Frieren’s backdrops (top) and frequent wide-angle have an almost pittoresque quality to them, inviting the viewer to stop and gaze, slowing down the pace, without appearing boring
The fight scenes (bottom), however, are usually much more dynamic, the art style changes, it features clear focus points, harsher lines, dynamic movement.

Now, it is important to note, however, that, while there are some amazingly animated action secquences, Sousou no Frieren is not an action anime at all. If anything, it would be more appropriate to call it a fantasy slice of life show, due to how Frieren’s journey is more about her everyday life and how she finds her way in a world, where elves are increasingly rare and thus, nearly everyone she meets, will die before her. This duality in moods, and how to approach them from a storytelling perspective, is by no means an easy feat, and while the aforementioned change in artstyle certainly sells the change in tempo and engage the viewer, it is without a doubt also the amazing soundtrack, composed by Evan Call, that helps the change along in a very unique way.

American composer Evan Call composed most of the Original Soundtrack of Sousou no Frieren, and already released part of it as a pre-release on Youtube. The full soundtrack is set to release in April

To understand how Call achieves that, it is first important to know, how he approached the Soundtrack in general. It is remarkable to see, how, to fit the music into the world, he actually makes use of medieval instruments, as well as choirs, and combines them with more modern composition techniques to achieve the tempo and response that is appropriate in any given scene. To make it possible, to seamlessly switch between slow-paced and fast-paced sequences, between slice-of-life and action, instruments, and sometimes even whole melodies get used in different paced pieces, while only slightly changing the tact or rythm, adding or substracting drums, but still following the same leitmotifs. A good example of this, when comparing different songs from his pre-released soundtrack, were the following.

Time Flows Ever Onward – a slower piece

Fear Brought Me This Far – an uplifting battle theme, that ends in a crescendo

The similarity between the two pieces, when experienced while watching the anime, is palpable and helps give the experience a more holistic feel. It allows for a change of pace, without one or the other feeling out of place. Of course, it would be possible, to further analyse these two pieces, as well as the rest of the soundtrack in depth, but this is not the goal of this blogpost. Rather, it is to understand, how sometimes seemingly subtle things can influence an experience.

In that regard, it is also very interesting, to see, how the anime plays with the passage of time, as it follows Frieren’s perception of it. As such, sometimes, within a few minutes, years can pass in the blink of an eye, while in others, time seems to almost grind to a standstil. Within a few frame and scenery changes, years can pass, while in others, especially during battles or important dialogues, the passage of time can be experienced almost in real time. It plays with the topic, of how even to someone as epheral and timeless as Frieren, time can pass in the blink of an eye, while still, if you’re aware of them, small moments still can have meaning, even to a life as long as Frieren’s. It is almost melancholic sometimes, to see what an impact memories and what may seem like such small gestures or occasions at the time can have. And of course, Evan Call also provides the audience with more than one other piece, that capture this sometimes epheral and timeless, yet oftentimes even melancholic quality quite well.

For 1000 Years – a slow piece, with an almost epheral quality to it, that is a play on the passage of time

It is very hard to put into written words, what it feels like to experience these pieces while watching, as such it is probably best to experience them oneself. Hopefully, by being a little bit more aware of them now, the experience can be made that much more enjoyable, however.

Master and apprentice

One last thing to add, that deeply fascinated me when watching the anime, was the relationship between Frieren and her apprentice, Fern, as they display both a remarkable similarity in some aspects, despite one being human and the other being an elf, while being completely different in others. It is funny to see their interactions and how they experience the same things sometimes differently and sometimes the same, especially when it comes, again, to the passage of time. In this sense, Fern grounds Frieren in the here and now, and might be the first proper human connection she has made since the passing of her party. To see, how that changes Frieren and to generally see, how they influence each other, how they copy each other’s mannerisms and small little details in their behaviours is a joy to watch. In this regard, the most fun little detail, that shows, how deep of an impact Frieren probably had on Ferns life, has to be their respective themes, both of which follow the exact same melody and leitmotif, with only small changes to singular notes, that still somehow manage to show both of their characters perfectly. Sadly, so far, only Fern’s theme has been released, but, as mentioned before, these songs are probably best experienced live and in person, while watching the anime, anyway.

Zoltraak – a name, that describes the offensive spell both Frieren and Fern use during their journey, that has still, due to stroy progression, become Fern’s theme, with Frieren’s being very similar

Case 2 – Nier: Automata: The ultimate humanist fable?

While Sousou no Frieren follows a more conventional Fantasy setting, albeit one with a new spin on it, Nier: Automata falls into the Science Fiction genre, while being no less fantastical. Set in the year 11.945, Earth has been invaded by advance aliens and their mechanical soldiers – machine lifeforms, whose whole purpose is the conquering of the planet and the extermination of humanity. In response, after endless battles, humanity has been brought to the brink of extinction and had to flee to the moon, from where they now wage a seemingly endless war, using their own placeholders – highly sophisticated battle androids called YoRHa – , to reclaim their planet and defeat the occupiers. In Nier: Automata, the player gets thrust in the middle of that conflict and takes control of two such YoRHa units: A battle unit, called 2B and a scouter, called 9S. Together, they take on the invaders, start to uncover the truth behind the invaders, their human masters, the war they’re part of, why the mysterious unit A2 desserted YoRHa as well as their main adversaries – the machine lifeforms. And while the player experiences the conflict through their eyes, making use of all of its assets and gameplay elements, the game asks its ultimate question: What does it mean, to be human?

A stylised cast and environment, and a large, varied number of tracks

As mentioned in the video, the game delivers a strong message and uses all of its gameplay elements towards it. These range from the wide variety of stylisation in its cast members – which, given its sometimes risque nature, admittedly might be slightly off-putting to some people at first – , its varied, open and completely different environments – from overgrown and reclaimed by nature urban areas, to dry desserts to amusement parks, where machine lifeforms celebrate – , it’s different gameplay elements – featuring mainly hack and slash action game, with top down or 2,5D sections as well as bullet hell and other mini-games added to the mix – all the way to its soundtrack, which was composed by studio Monaca, with its lead composer Keiichi Okaba, and which has earned multiple rewards, most notably „Soundtrack of the year“ at the yearly Game Awards.

Japanese composer Keiichi Okabe is founder and chief executive officer of music production company MONACA, which provided the Original Soundtrack for NieR: Automata

What makes the soundtrack and Okabe’s work so impressive is, how it incorporates seamlessly into the story and extends it, telling its own tale in an additional layer, that gets added to the other storytelling elements. It does so, using a variety of different languages, styles – instrumental, vocal as well as using 8Bit sounds, among others – and musical elements, that create a unique, but strangely holistic sounding composition of tracks. Of course, what makes this soundtrack even more impressive, is its sheer length and variety of track – Keiichi Okabe provides the player with more than 6 hours of total track length, each track finely tuned to the exact moment it appears ingame, which might only be once during the whole playthrough, for a short time, but still, adding to the told story.

意味/無 – meaning/nothing, which could also be read as meaningless when put together, is just the first of 119 tracks the composer has uploaded to Youtube. And the name already hints at one of the games hidden messages

A shift in perspective and a strong social element that transcends borders and time

Another aspect, that makes the game very unique is, that it has to be played through a total of 7 times in order to get the complete experience and get the complete picture. And while this certainly can be another factor, that could be off-putting for many people, it is certainly worth every second spent in the game world.

A review by Skill Up, that encapsulates, what maakes the game so special, aptly titled „The Masterpiece You (Probably) Won’t Play“, due to the multiple playthroughs the player has to complete first

While it may seem ridiculous at first, to expect the player to play the same game multiple times, there is a logic and approach behind it, as with each playthrough, you take control of a different character and experience the story through their eyes, each time adding different things to the experience, putting emphasis on different things, unveiling more of the truth. And, while it might seem, that each playthrough is the same at first, that is only really true, as only those culminate in what appears to be the same ending. On the third playthrough, however, an additional secret is revealed, namely, that the player actually hasn’t experienced the same story twice, but rather a repeating cycle of the same occurances. This may seem like a negligible difference at first, but will become very important to the game’s overall message as the story progresses. And while the different endings may appear similar at first – though of course each character adds thier different experiences – here too, Okabe’s music hints at a deeper message.

Weight of the world – the first ending’s song, in English

„壊レタ世界ノ歌“ (Kowareta Sekkai no Uta, lit.: „Song of a broken world“) – the second ending’s song, in Japanese

It does so via the final soundtrack for each ending, that, while being the same melody and song, gets completely transformed by its lyrics. Because, while the first ending’s song, „Weight of the world“, is sung in English from 2B’s perspective, the second ending’s „壊レタ世界ノ歌“ (Kowareta Sekkai no Uta, lit.: „Song of a broken world“) is sung from 9S’s in Japanese and delivers a completely different meaning to the just experienced playthrough. And just like this, each added ending song, of which there are 4, for reasons I do not want to spoil right now, gets another added layer.

Nouveau – another ending song in French

This slow change in perception by the characters – via a musical expression – may be somewhat unnoticeable by the player at first, especially during the initial playthroughs, however, it is an important element, that leads up to its final end, its final crescendo, in „the End of YoRHa“. What makes this song special, is, that it makes the players aware of a gameplay element, that they may have not consciously realised was part of the game – while, of course, still making use of it – and brings them together in a way, that I have not yet seen in a video game since. It is an adequate end to a very emotional journey, that is made all the sweeter by its final, moving piece, that has all of the game’s development team joining in to a choir – a small detail, that once more highlights the togetherness and sense of belonging, of being part of a bigger whole the player may feel in that moment.

the End of YoRHa – the final track in the soundtrack, combining all previous experiences, before presenting the player with the ultimate choice

It is in this moment, after the player has overcome the so far biggest challenge, all the different story-hints and paths come together and they are at the height of their emotional investment, that the game shows its final message to the player, while presenting them with one, final choice. It is a masterclass, on how investment and immersion can work.

Case 3 – Arknights: „Lone Trail“ and „Under Tides“: A holistic approach to ingame events

As a final entry to the presented case studies, Arknights is a multiplayer mobile tower-defense game developed by Chinese developer Hypergryph, that is set in the world of Terra. On Terra, which could be seen as a more grimmdark fantasy copy of our own worlds, which is made up of multiple factions and nations, the population lives in constant fear of catastrophes – cataclysmic events, that can wipe whole cities of the map – as well as Oripathy – a cancerous desease caused by a catastrophe’s aftermath, that is lethal – while also struggling for survival due to prejudice, scarce resources and the different nations being in conflict. These nations, factions and conflicts are oftentimes heavily inspired by our own world, while being placed in an environment, that is far more hostile and unforgiving than ours – a dark spin to the world, so to speak. Similar is true for the game’s ingame events, that add to the overarching story, push the narrative forward and reveal more of Terra, its people and their struggles to the player. Two such events, as well as their aural theme – Arknights plays heavily with music and its themes, with its developer Hypergryph even building a whole sub-sector wholy dedicated to creating and promoting music in their company -, are the focus of this study. Coincidentally, the final theme of both events has been composed by the same composer – America-born Steven Grove

Composer Steven Grove composed the final soundtrack / boss – theme for both Lone Trail and Under Tides

Lone Trail

The official trailer of Arknights – Lone trail already hints at its theme

Lone Trail is a ingame event, that has been inspired by the space race of the 1950s and 1960s between the Soviet Union and the United States. Nowhere is that more apparent than in its visual design, which can be described as retro-futuristic and which aludes to its inspiration. The following video perfectly encapsulatey and exemplifies, how Lone Trail manages to engage the player while drawing its connection to our real, historic counterparts.

An analysis of the Visual Design that make up the Lone Trail event in Arknights

While the visual design of Lone Trail is one part, that needs to be mentioned to understand Hypergryphs holistic approach when it comes to their game design and the design of their ingame events, all with the intent purpose of engaging and investing the player in their built world, as previously mentioned, another is Steven Grove’s music-addition to it.

Steven Grove – Control’s Wishes are both the first and final piece of music the player hears when experiencing the story of Lone Trail

The final soundtrack, Control’s Wishes, which can already be heard in the trailer, gains a whole new meaning, when connecting it to both the just experienced story and its overall setting. It is a multi-layered and multi-facetted track, that can both add to the existing story, thus boosting the players immersion at the end, but can also be understood without much context, just through its composition and musical elements alone. It is once more interesting to see, how much a well-composed track can add in terms of perceived information and mood, without the user having to necessarily be consciously aware of it.

Professionally trained opera singer and voice actor MarcoMeatball reacts to Control’s Wishes and analyses the main elements of the song

Under Tides

The official trailer of Arknights – Under Tide features many of the aquatic elements – both visually and aurally

Another event, that follows a similar scheme in terms of its visual design and aural depth – and whose first and final piece also was composed by Steven Grove, is Under Tides. This final piece is named „Under Tides“, and while the player may be confronted with it during the initial trailer already, it is only during the final boss battle, that its true meaning becomes apparent.

Steven Grove – Under Tides is a boss battle theme, that has a slight twist to it

Because, while, due to the audiovisual storytelling of the trailer, the setting of the event and its connection to the sea already become obvious, it’s in the final boss battle, that all the aural and visual hints, that have been dropped before, become obvious.

Voice actor and opera singer MarcoMeatball also analysed the musical elements of Under Tides, revealing the true meaning, that the player is exposed to, without having played the event himself

It is during the final fight, or rather, at its conclusion, that the final twist of the event is revealed: Under Tides isn’t the battle theme of the enemy you are fighting, but rather of the player, more specifically of the Abyssal Hunters at his side, that he commands throughout the battle and most of the event. „The player is the boss“ is an endearing and exciting new twist on how battles can work in general, as letting them experience being the final obstacle, the ultimate foe to overcome, is something not many games have attempted so far and Steven Groves music provides a very fitting backdrop for the experience.

A short summary and reflection of my journey so far

Finally, it is important to mention, that all these case studies should, at best, provide a broad overview of my experiences to the reader and are in no case to be understood as an in-depth analysis of the discussed material, as doing so would require a much deeper engagement of each of the discussed elements and source materials at hand. It should, however, hopefully engage the viewer and show, how different small details and used design elements can naturally come together to create a bigger whole, all with the goal of creating an immersive, engaging and holistic experience to the user.

On my journey so far – which included research into storytelling, world building, animation, rigging, extended realities and more – I have become deeply fascinated by the topic of immersion and how to invest someone into a story. I am especially fascinated by experiences inspired by modern games and media and set in the virtual. As such, after reflecting on my findings, I have decided to further pursue immersion as my goal, with the final goal being to create a virtual reality experience, that hopefully manages to catch the user’s interest and allow them to completely immerse themselves in the world I am building, whichever shape or form it ends up taking. To refine my goal and to achieve it, I will continue my research into various topics around it and start first studies and small-scale tests into different applications and designs, that will then hopefully help me carve out my path and help me find my final world.

Sources:

1. Kanehito, Yamada: Sousou no Frieren, Japan 2023-2024, Anime by: Studio Madhouse, Episodes 1 – 6, streamed at: https://www.crunchyroll.com/de/series/GG5H5XQX4/frieren-beyond-journeys-end, in: https://www.crunchyroll.com/de/ [22.02.2024].
2. Evan Call – Topic: Frieren: Beyond Journey’s End – Original Series Soundtrack EP, https://www.youtube.com/playlist?list=OLAK5uy_n9086XIv_nf3_qDXrEYnUm_dwW1KGxyrE [22.02.2024].
3. n.a.: Evan Call, https://anilist.co/staff/119999/Evan-Call, online in: https://anilist.co/ [22.02.2024].
4. Saba, Michael: How NieR: Automata Tells the Ultimate Humanist Fable, https://www.youtube.com/watch?v=63PzQIbTrM8 [22.02.2024].
5. n.a.: Keiichi Okabe, https://myanimelist.net/people/42543/Keiichi_Okabe, online in: https://myanimelist.net/ [22.02.2024].
6. Keiichi Okabe – Topic: NieR: Automata complete OST, https://youtube.com/playlist?list=PLdR7m7PFLzQ7RqOVfxk2Fr2F-a7iWzovn&si=90M0GsB9J6ZDHFt2 [22.02.2024].
7. Skill Up: Nier: Automate | The Masterpiece You (Probably) Won’t Play, https://www.youtube.com/watch?v=-NiyfG8Ctbo [22.02.2024].
8. Grove, Steven: Steven Grove Music, https://www.stevengrove.com/ [22.02.2024].
9. Arknights Official – Yostar: Arknights Official Trailer – Lone Trail, https://youtu.be/U_laI-G21fo [22.02.2024].
10. HalfACupOfRice: The Visual Design of Arknights Lone Trail, https://www.youtube.com/watch?v=aS8nnjrMlQc [22.02.2024].
11. 塞壬唱片-MSR – Topic: Control’s Wishes, https://www.youtube.com/watch?v=KKhEwcWMH9M [22.02.2024].
12. MarcoMeatball: Arknights – Lone Trail: Control’s Wishes || MUSIC REACTION, https://www.youtube.com/watch?v=s7NdkeSQcjI [22.02.2024].
13. Arknights Official – Yostar: Arknights Official Trailer – Under Tides, https://www.youtube.com/watch?v=0hqKmICjh0g [22.02.2024].
14. 塞壬唱片-MSR – Topic: Under Tides, https://www.youtube.com/watch?v=4uTMR6vZqFM [22.02.2024].
15. MarcoMeatball: Opera Singer Reacts To The Music of Arknights, https://www.youtube.com/watch?v=4l7nivLqz4Q [22.02.2024].

by max.fruehwirt - 21. Februar 2024

09 | More than just buttons – Interactions and interfaces in VR, AR and MR experiences

What this blogpost is all about

Building upon my previous blogpost on immersive level design, this post also aims to explore the topic of immersion and interaction within VR, MR and AR a bit more by looking into different interface and interactivity solutions currently available, some previously mentioned already, some new, that may increase intuitiveness and engagement from the user. While also still covering AR and MR solutions, the main focus of this blog post will be on VR, as it is more in line with my future plans and planned research.

Tracking, controllers, recognition and other interaction methods

With a wide variety of different VR, AR and MR headsets and technologies comes an equally wide variety of different input devices, interfaces and ways to interact with the created virtual environment. In VR alone, there exists a seemingly endless amount of different controllers, with each headset developer putting their own spin on it.

Different headsets, different controllers – HTC Vive Pro, Meta Quest 3 Pro, PS Move and Valve Index Pro (left to right)

However, controllers like these are by no means the sole mean to interact with virtual environments anymore, as with advancements in tracking, movement- and voice recognition, a vast variety of different input and interface methods has been developed alongside the conventional controller-based inputs.

Hands, eyes, voice and haptic feedback

As previously mentioned, constant advancements in available computing power, frequent optimisations as well as new technologies make it possible to create virtual experiences that are more immersive than ever.

One such advancement lies in tracking and how the tracked movement and data get processed. While hand and gesture tracking has been a long-time staple especially in AR headsets, due to their inbuilt sensors and tracking, it has also become one in VR and MR applications. To give a differentiation between hand-tracking, controller-based-tracking and gesture-tracking, more commonly known as gesture recognition, which all may appear similar at a glance, is quite simple. Hand-tracking, as the name suggests, track the actual movement of the hand within the virtual space.

Ultraleap’s 3Di, a small tracking camera for handtracking, comes with its own intergrated interface

Different than controller-based-tracking, it frees the hands for interactions, without relying on buttons or other inputs. Controller-tracking, in comparison, also tracks the hands movement, but instead of doing so directly, it tracks the hand-held controllers instead. These controllers usually come with a wide variety of buttons, joysticks and other triggers, that can be programmed and used to interact with the environment and input information.
Last but not least, gesture recognition interprets specific hand movements or gestures made by the user and reacts in specific ways, allowing interaction and enabling control over certain parts of the virtual space. It can be understood as a specific form of hand-tracking, as specific parts of the hand get tracked, though, in this case, the gesture made is usually more important than the position of the hand compared to the rest of the body.

Ultraleap Leap Motion Controller 2, a gesture tracking controller with a wide variety of applications

While it may seem now, that the main focus of current interaction lies in tracking movement of extremities, mainly the hands, this is not true. Eye tracking, for example, is a gaze-based form of interaction, that makes use of tracking monitors, that follow the user’s eye movements to enhance realism, allow interaction and render specific parts more or less, thus deepening immersion as needed, while devices like Amazon’s Alexa, Microsoft’s Cortana or Google’s own Voice Assistant can long sicne be used in VR and MR as well, to control and interact with the virtual environment using vocal commands. Using a combination of these different tracking technologies can make the user’s environment feel much more responsive.

But not only the users inputs can be enhanced by new technologies to create a more immersive experience. Using haptic feedback systems, spatial computing as well as hardware solutions, that creates the illusion of seemingly endless virtual spaces even within a very limited physical environment, the immersion of the user and responsiveness of the environment can be increased even further. While haptic feedback gloves have already been mentioned in a previous blog post, it is important to mention that haptic feedback, in the broader sense, is in no way limited to the tactile layer anymore.

teslasuit – a full body haptic feeback suit, that also tracks movement and can be used for motion capture

Haptic feedback suits, like the one shown above, can provide live-responses of the virtual environment to the user via vibrations. This can create the illusion of a physical component being present in the environment. Furthermore, spatial computing, especially in combination with multimedia-rooms, like the TU Graz‘ VR Cave, which can be used to more seamlessly merge physical and digital worlds, allowing physical objects to be tracked and influence the virtual, while also allowing virtual objects to interact with the real environment. Additional use of hardware, that allows for extended movement of the user through the virtual space even when limited by smaller real spaces, like an omnidirectional treadmil, can further blur the line between the virtual and the real.

Virtuix Omni One – a Omnidirectional treadmil developed for gaming in VR

Things to consider

When presented with all these different options to track the user’s input and the data they provide, it can be very easy to be overwhelmed or lose track of the reason of why these movements should be tracked and / or used in the first place, namely, to provide a smooth, fun and immersive experience, that allows a maximum amount of user interaction with a minimum amount of effort on the users side. To ensure that, there are a few important steps to consider when designing such an interaction.

Abstraction, intuitiveness, optimisation and sense of security

A good first step when approaching the design of user interaction and interfaces for the virtual is mimicking the real world and its interactions, to increase both intuitiveness for the user and clearness when providing feedback. By adapting the sense of realism or choosing a certain level of abstraction for the inputs and / or interfaces, they can be simplified and made to fit the desired experience without distracting the user.

Frequent user testing, followed by refinement and optimisation of the employed systems, can increase responsiveness, accessibility and create a sense of security in the user when confronted with the virtual environment. Furthermore, the higher the continuity of the created content, both in design and in experience, and the more seamless the transition between the physical and the virtual, the easier it is for the user to engage, which also boosts self-confidence and security.

All in all, by making use of the different technologies described above, while being aware of the challenges and chances they bring and optimising and adapting the desired experience according to the user’s needs, it is nowadays possible to create amazingly responsive environments already. It is, however, still important to be aware of the ever present limitations of current hardware, but with how rapidly technology and development keeps progressing, the next solution might already be around the corner.

Next steps:

Look further into different VR and MR solutions and their respective issues
Research essential tools for creating immersive virtual environments as well as different game engines and their advantages and disadvantages
Check methods of engagement and interaction within these digital environments
Look into accessibility and how to ensure it
Research into immersion and storytelling

Sources:

1. Springer / Gabler.: Virtuelle Realität, in: Gabler Wirtschaftslexikon, n.y.,
https://wirtschaftslexikon.gabler.de/defi niti on/virtuelle-realitaet-54243 / online in: https://wirtschaftslexikon.gabler.de/ [08.02.2024].
2. n.A.: Was ist Augmented Reality?, in: Omnia360, 2020, https://omnia360.de/blog/was-ist -au gmented-reality/, online in: https://omnia360.de/ [08.02.2024].
3. n.A.: Mixed Reality: Wenn sich Reales und Virtuelles vermischt, in: Omnia360, 2023, https://omnia360.de/blog/mixed-reality/, online in: https://omnia360.de/ [08.02.2024].
4. n.a.: Extended Reality, in: RyteWiki, n.y., https://de.ryte.com/wiki/Extended_Reality, online in: https://de.ryte.com/wiki/Hauptseite [08.02.2024].
5. Hayden, S.: Vision Pro Teardown Shows Balancing Act Between Cutting Edge Tech & Weighty Design, in: ROADTOVR, 2024, h ttps://www.roadtovr.com /apple-vision-pro-teardown-ifixit/, online in: https://ww w.r oadtovr.com/ [08.02.2024].
6. Hayden, S.: Quest 3 Teardown Shows Just How Slim the Headset Really Is, in: ROADTOVR, 2023, https://www.roadtovr.com/meta-quest-3-teardown-ifixit-repair/, online in: https://www.ro adt ovr.com/ [08.02.2024].
7. Hayden, S.: Vive Ultimate Tracker Gets Beta Support for Third-Party PC VR Headsets, in: ROADTOVR, 2024, https://www.roadtovr.com/vive-ultimate-tracker-quest-index-pico/, online in: https://www.ro adt ovr.com/ [08.02.2024].
8. n.a.: What to Watch: February 2024 Highlights, n: Meta Quest-Blog, 2024, https://www.meta.com/de-de/blog/quest/what-to-watch-free-meta-quest-tv-vr-film, online in: https://www.meta.com/at/ [08.02.2024].
9. CGV Channel: TU Graz Virtual Reality Cave, https://www.youtube.com/watch?v=aeTHlAZtlAI [08.02.2024].

by max.fruehwirt - 19. Februar 2024

08 | Designing realities beyond screens – World building and immersive level design in VR, AR and MR

What this blogpost is all about

To better understand, how to design immersive and engaging experiences in VR, AR and MR, and building on the previous post about issues and how to solve them, it is necessary to understand the importance of world building and level design on the overall experience. As such, this blog-post aims to provide a short overview on different considerations to be aware of, when dealing with the challenge of creating an immersive experience for VR, AR and MR. Same as with my previous post, however, I would like to dive even deeper into immersive world building and continue my research, so this entry is only a momentary summary and my research into this topic – especially since I plan to eventually read all of Richard A. Bartle’s „Designing Virtual Worlds“ – is still incomplete.

How to create engaging, immersive worlds using level design

Understanding the importance of space and spatial awareness

When designing VR and AR environments, it is crucial to remember that, different from other virtual experiences in front of a monitor or screen, such as games, videos or the like, in VR and AR the user moves his own body and his virtual avatar through the real and the virtual world. This may seem like an obvious and maybe not so important difference at first, but it has some implications, as the user’s physical space and interactions within it need to also be considered when designing. Furthermore, the user’s real life comfort zone, field of view as well as the virtual viewable distance need to be considered as well to understand how to engage with the space.

Using realism, stylisation and interactivity to boost immersion

Utilising the capabilities of modern graphics cards as well as making use of optimisation and / or stylisation options when designing virtual environments for VR and MR, it becomes possible to create both realistic and semi-realistic experiences, that still keep the user immersed. No matter the style, however, maximal attention should be given to light, sound and eventual haptic or other tactile feedbacks, as only a wholistic experience allows the user to perceive the created environment as real and thus allow for maximum immersion. In this regard, it is also important to be aware of how the user interacts and influences the created world. Integrating easy, intuitive and natural-feeling interactions, that take advantage of all the capabilities of VR and AR input devices, such as controller movement and position tracking, gesture tracking or even full-on inside-out tracking can influence how the user interacts with the virtual world.

Design intuitive spaces and provide clear clues

Understanding the perception of different spaces and how it may influence the user, be it in an AR environment, in MR or even completely virtual in VR, is important, as all these environments, compared to our usual computer screen, are perceived as 3-dimensional and thus need to be approached accordingly. By providing clear cues for navigating the virtual space, it becomes easy and safe to explore. To counter the limitations of a user’s real, phyiscal environment vs. its virtual counterpart, a wide variety of movement methods can be explored, such as teleportation, locomation or even simple walking, should the space allow it. This allows for a different approach to space than usual, as it further opens the environment up for exploration.

It is, however, important to always be aware and understand the limitations of such an experience, both in hardware – e.g., tracking and sensing capabilities and physical boundaries – and in the user’s own perception – e.g., how much they can handle without being overwhelmed by the stimuli and cues. A good example of such an overwhelming experience, that is sure to cause fatigue in the long term, is the short-film „Hyper-Reality“ by Keiichi Matsuda.

Hyper-Reality by Keiichi Matsuda

Be aware of storytelling and how it is affected by 3D

Last but not least, after exploration of both the virtual and the physical space, depending on whether the created experience has been in AR, VR or MR, has been opened to the user, it is paramount to understand, how storytelling may be affected by the 3D environment. A good approach to this would be to learn from the gaming industry, as it has created explorable semi 3D and 3D environments for a long time now. Of course, with AR, VR and MR, these spaces – these levels – get extended even further, with more feedback being added, more information being perceived, and more interactions being possible, so careful consideration should be given on which tools may be applicable and which may be unsuited.

The end goal, in any case, should be to consider innovative ways to tell a story, that seamlessly incorporates its narrative elements within the created user-experience. To make sure that the experience remains as complete as possible while offering free exploration remains the challenge when creating such a virtual space.

Next steps:

Look into interfaces unique to VR, AR and MR
Check how interaction with VR, AR and MR may work
Research how to boost engagement and create intuitive interactions
Research into immersion and storytelling

Sources:

1. Bartle, R.A.: Designing Virtual Worlds, New Rider, 2003, ISBN: 978-0-1310-18167.
2. Matsuda, K.: Hyper-Reality, online in: http://hyper-reality.co/ [08.02.2024].
3. n.a.: VR Design Principles, online in: https://viro-community.readme.io/docs/vr-design-principles [08.02.2024].

by max.fruehwirt - 18. Februar 2024

07 | Beyond the virtual: Motion sickness and other accessibility problems when navigating the virtual realm

What this blogpost is all about

Continuing the notion of my previous blogpost of understanding VR, AR, MR and XR along with their strengths and weaknesses, this post aims to also provide insight into the different problems one may be confronted with when designing for or engaging with these technologies on the regular and how to best solve them. At this point, it is necessary to mention already, that, while this blogpost will provide a broad overview about the different challenges one may face along with some possible solutions, it is in no way a complete guide yet, as my research into this topic is still ongoing and thus, incomplete.

Discomfort, accessibility and other challenges

To provide a widely accessible, engaging and also comfortable and immersive experience it is important to address a multitude of issues that the user may experience, when engaging with VR, AR or MR. By being aware of said issues as well as their implications to the experience and how to solve them, a much more complex and immersive experience can be created.

Motion sickness and other discomforts

Extended exposure to virtual environments, be it via VR, AR or MR, can lead to a wide variety of discomforts, when not handled and prepared for accordingly.

Simulator sickness, which is a special type of motion sickness, occurs, when our visual and vestibular system receive conflicting information. This can happen, for example, when the visual display or overlayed displayed object suggest rapid motion, but the user’s body senses no corresponding movement. The resulting mismatch in information can lead to disorientation, nausea and discomfort.

Furthermore, extended exposure to close-up virtual objects or displays can leade to both eye strain and general fatigue, as the eye has to constantly strain to focus. This strain may be even increased, if the eye has to constantly focus on rapidly moving objects or re-focus due to overlays, interfaces or similar displayed objects moving back and forth or rapidly in and out of focus.

Last but not least, when switching between AR and VR – which may mainly occur when working with MR solutions – ,but also when switching between AR or VR and the real world, user’s may experience a short bout of spatial disorientation. This may happen, when the transition between the virtual and the real environment is either to abrupt or not enough spatial clues are present in either the virtual or the real environment for the user to orient themselves. This can then cause disorientation, dizziness or short emotional distress.

Accessibility and impairments

Another thing to consider when dealing with artificial environments is accessibility for people with certain sensibilities or impairments. To create an inclusive and immersive experience, a number of challenges have to be considered.

As VR and AR experiences currently heavily rely on visual elements and their space cues, it is very easy to exclude users with visual impairments. Whether it be complete blindness, colour blindness or other similar disabilities, a number of things needs to be considered to make the experience as inclusive as possible.
In a similar vain, people with auditory, cognitive or motor impairments may have trouble interacting with the created environment as well. Be it challenging motion-based inputs, complex interfaces, information overload or lack of adaptability, users may find an interaction to be needlessly challenging for them and thus feel excluded from the experience, unless ample consideration is given.
This, of course, is also true for users with special sensory sensitivities, such as to bright lights, loud sounds or intense vibrations.

As such, accessibility needs to always be considered when designing in and for such environments.

Possible solutions and their integration

These considerations may seem challenging and / or limiting at first, however, they can also be understood as a chance to create a more immersive, comfortable and holistic experience for the user. When designing virtual environments, it is important to constantly question oneself, whether or not ample consideration for the described challenges has been given and remember some of the solutions described below.

Comfort and customisation

To increase the user’s long-time immersion, and comfort while doing so, a number of small changes and customisation choices can already make a huge difference.

One such change is in how the software is structured in the first place. To give an example, by including different options to adjust volume or brightness of the headset, but also providing different levels of haptic feedback, the whole experience can be customised to fit people of different sensibilities. In the same vein, allowing for the adjustment of the displayed colours or sounds or adding audio descriptive elements can also provide more inclusivity for people with auditory or visual impairments. Following this approach, the chance of motion sickness occuring can also be decreased, by allowing the user to choose settings such as movement speed, field of view and add comfort vignettes, that allow the user to find settings, that minimise their overall discomfort when being exposed to VR or MR environments for an extended amount of time.

HaptX haptic feedback gloves

Sony’s accessability controller, compatible with the new PS5 and PS VR

Another way to increase comfort would be by making adjustments to the hardware and how the user may engage with it. Using headsets with reduced latency or the currently trending inside-out tracking, for example, can both reduce motion sickness by boosting the sense of being present within the current environment and also reduce the weight and thus discomfort the user is exposed to long-term while interacting with the digital environment, be it AR, VR or MR. Similarly, combining modern headsets with custom accessibility controllers or even haptic feedback gloves, a variety of different modes of interaction can be provided, that could potentially lead to a more inclusive environment. A few solutions hereby would be tracking gestures of different body parts and allowing for gesture or voice controls as alternative input methods for people unable to use more conventional controllers or adjusting the haptic feedback to provide more or less information depending on the user’s other sensory capabilities.

Control and adaptability

To conclude the described solutions, providing the user with a maximum amount of control, thus allowing them to adapt the experience to fit their specific needs, can greatly boost both inclusivity and immersion. This may seem like an obvious conclusion, at a glance, but the fact that there is still new technologies and ways being developed to create an even better immersive experience for the wide variety of user needs hints at the opposite. As such, it is paramount to always consider your targeted end user while developing and frequently question yourself not only how to best experience a created environment, but also the challenges some users may face when doing so and to react accordingly.

Next steps:

Look into immersive environments and how to create then
Check methods of engagement and interaction within these digital environments
Research into immersion and storytelling
Look into interactivity in VR, AR and MR
Check interaction solutions for VR, AR and MR

Sources:

1. Creed, C., Al-Kalbani, M., Theil, A. et al. Inclusive AR/VR: accessibility barriers for immersive technologies. Univ Access Inf Soc (2023). https://doi.org/10.1007/s10209-023-00969-0 online in: https://arxiv.org/ftp/arxiv/papers/2304/2304.13465.pdf [08.02.2024].
2. n.A.: Why VR (& VR Headsets) Can Cause Serious Eye Strain & Pain, in: NVision, 2023, https://www.nvisioncenters.com/education/vr-and-eye-strain/, online in: https://www.nvisioncenters.com/ [08.02.2024].
3. n.A.: XR Today’s Guide to Stopping VR Motion Sickness, in: XR Today, 2023, https://www.xrtoday.com/mixed-reality/xr-todays-guide-to-stopping-vr-motion-sickness/, online in: https://www.xrtoday.com/ [08.02.2024].
4. n.a.: VR Design Principles, online in: https://viro-community.readme.io/docs/vr-design-principles [08.02.2024].

by max.fruehwirt - 9. Februar 202418. Februar 2024

06 | VR, AR, MR, XR – Exploring the future of extended reality and its technologies

What this blogpost is all about

To explore one of my possible research topics further, this blogpost will look into the strengths and weaknesses of virtual, augmented, mixed and extended reality systems as well as their exact definitions and current technological trends, to gain a better understanding of which system / technology to use in future endeavors when going for different experiences.

VR vs AR vs XR/MR – a comparison

To better understand the difference and similarities between the different technologies, it is first important to understand their definitions as well as their strengths and weaknesses. For this reason, a short overview will be provided.

Virtual Reality (VR)

Virtual reality creates a computer-generated environment that immerses the user in a completely digital, three-dimensional space, often experienced through specialized VR headsets, providing a sense of presence and interaction within the virtual world. The user, thereby, is completely seperated from the real world and any stimuli they experience is completely computer generated. As a result, it usually allows for a much deeper immersion than the other solutions researched in this blogpost, but has its own strengths and weaknesses to be aware of.

Strengths:

Offers a completely immersive experience, perfect for training, gaming and simulations
Can easily create spaces, that are not accessible normally and / or provide space where normally there would be none
Can be used in healthcare, especially in therapeutic applications, to provide immersive therapy, exposure therapy, pain management and rehabilitation

Weaknesses:

Isolation from real world may cause emotional distress (solitude)
Needs special equipment, that may be costly and / or not readily available

Augmented Reality (AR)

Augmented reality overlays digitally created content onto our real world, enhancing the user’s perception of their surroundings by integrating computer-generated information such as images, text, or 3D models into the real-time environment. These are typically viewed through devices like AR glasses, tablets or smartphone screens, though in recent years, more applications have surfaced. Especially in the automotive industry, heads-up displays also make use of AR to display necessary information directly to the driver while projecting said information directly onto the windshield.

Strengths:

Real world information overlay, that relays information in real-time and provides additional input
Allows for hands-free interaction, which allows the user to engage the digital content while also staying aware in the real world
Useful for product visualisation and trying out products before making a buying decision

Weaknesses:

Limited field of view, especially on smartphone screens or tablets
Mobile dependency means less computing power, limiting display performance and causes need for optimisation

Mixed Reality (MR)

Mixed reality combines elements of both virtual and augmented reality, allowing digital and physical objects to coexist and interact in real-time. It seamlessly blends the virtual and real worlds, and allows switching between them, enabling users to engage with both types of content simultaneously. While this, of course, can provide difficult to understand with at first, it also allows for a much deeper influence of the user’s perceived reality as a result.

Strengths:

A high level of versatility, as it combines both VR and AR, it allows for a broader range of experiences to be created
Enables both in-room and virtual connection, communication and collaboration
Can, same as VR, be used in a wide variety of industries for training purposes, while also allowing for direct testing in the real world in AR

Weaknesses:

Different technologies and their implementation can cause performance and optimisation issues, posing technical difficulties
Cost of adaption currently still very high, especially when compared to pure VR or AR solutions

Extended Reality (XR)

Finally, extended reality is an umbrella term, encompassing VR, AR, and MR. It referes to the spectrum of immersive technologies, that extend, enhance, or blend reality. XR is a comprehensive term covering the entire range of experiences, from completely virtual to fully augmented. It aims to offer a holistic approach to immersive technologies. As such, it comes with all of the previously strengths and can, if used correctly, mitigate some of their weaknesses. The opposite, when used incorrectly, is also true, however.

Current trends and technologies

While of course, the market around VR, AR, MR and other technological solutions like that is constantly evolving, it is still important to understand the general direction of these developments in order to better understand them and work with them. As such, some of the most important ones will be listed here.

Stand-alone, wireless VR, AR and MR headsets without the need for external trackers:

HTC recently presented their new inside-out tracker, that would allow for inbuilt-tracking on a multitude of existing 3rd party headsets

While some of the previous generation’s systems still require for external trackers / tracking stations to be set up or for the headset to be connected to a PC via cable in order to ensure a smooth and immersive experience, current trends have since begun to diverge from that. The current trend seems to be to offer stand-alone, wireless VR, AR and MR solutions without the need for a seperate computer or similar unit for calculations, that also offers tracking via inbuilt sensors instead of external stations. The tracker shown in the picture above, for example allows for complete controllerless tracking of a person’s arms, legs, torso and even head, all by attaching it to the limb in question and up to 8 trackers can currently be used together to provide a smooth and easy experience.

Hybrid systems:

Both the Apple Vision Pro and the Meta Quest 3 offer hybrid solutions when it comes to VR and AR.

While there are still certainly solutions, that focus on either VR or AR in particular, MR seems to be the much more common trend, with passthrough headsets becoming more and more widespread. This is especially obvious when looking at both Apple’s recently released Vision Pro, but also Meta’s Quest 3.

Wider application in everyday life:

When looking at the adaption rate of VR, AR and MR solutions, it quickly becomes apparent that the possible applications for it have skyrocketed. While originally more of a niche development, nowadays a wide variety of experiences are offered, ranging from immersive nature documentaries, sport events, games, movies and more. Furthermore, with both the increasing variety of headsets to choose from and their increasing calculation power comes a lower price – except for the apple solution, of course – with an also increased quality in the experience, making the technology much more accessible in everyday life.

Next steps:

Look further into different XR solutions and their respective issues
Research essential tools for creating immersive virtual environments
Check methods of engagement and interaction within these digital environments
Look into accessibility and how to ensure it
Research into immersion and storytelling

Sources:

by max.fruehwirt - 8. Februar 20248. Februar 2024

05 | Character animation and rigging in Cinema 4D – 3. Advanced tools and techniques

What this blogpost is all about

This blogpost will be the final entry on the topic of character rigging and animation in Cinema4D. It will cover some of the more advanced tools Cinema4D provides for character rigging and animation and how they may be used to either achieve even better results or speed up the production process. Finally, it will give a short summary of the covered topic.

Deformation and changing of topology based on movement

Muscle Object

The muscle object follows its name by allowing for the set-up and simulation of anatomically correct muscles and sinews. These muscles can be attached to the joints and bones (the rig) allow the simulation of muscle movement beneath the mesh. Since the muscle object is a very complex and advanced tool, that requires lots of time to set up, however, it is only worth the effort when complete anatomical accuracy is paramount. In these particular cases, however, it is possible to simulate dynamic movement changes, skin deformation, different anatomies and their anatomical changes when influenced by different muscles, programm simulations and many more features that would require a whole seperate tutorial series.

Needless to say, the muscle object is a very powerful tool, if mostentimes a bit too complicated, especially since, as long as one’s goal is a more cartoonish style of animation – or rather just not 100% anatomical correctnes – simulating muscle movements by using the pose morph is usually a much quicker and easier alternative.

Pose Morph

As previously described, when working on muscle movements in a rig, where anatomical correctness is not the main focus, the pose morph can be used to transform mesh topology and animate this transformation. By adding the pose morph tag to the obejct or topology one wants to animate, it is possible to switch between different defined poses by simply moving a slider. The poses, between which the slider switches, can be defined in the edit tab of the tag.

While having the pose morph in edit mode, and while working in point mode on the topology, the pose-morph tag records the initial position of the topology / mesh as original position. Now, when working in point mode with the pose morph tag active in edit mode, any changes made to the mesh, e.g., movement of points or any other disformation, will be recorded and the new, updated mesh will be set as target pose. Now, by accessing the animate tab of the pose morph, the object can be blended seamlessly between the two set poses using a slider. This is also true for any and all affected points of the object – to animate the changing topology parameters like rotation, UVs, scale, normals, and many others can be simply changed and the change recorded. Furthermore, it is important to note, that not only one, but a multitude of target poses can be added, which makes it possible to easily animate and / or blend the same object from its starting position to a wide variety of different target postitions via sliders for all the different set poses. It is, in this regard, very similar to the blend-shapes option in other animation software
This shows, that the pose morph is a very powerful tool, that allows for easy animation and keyframing of the different poses and blend between them seamlessly, by simply keyframing the blend-slider in the animate tab. This, in connection with Cinema4D’s intuitive auto-keyframe option, allows for a very easy, smooth and fast animation workflow while still producing remarkably realistic results.

Parametric and node based animation

As parametric and node based animation is a very deep and complex topic, this chapter will only cover the basics of the available tools, without diving too deep into the programming or concepts behind it, but aims just to show, which tools exist and, in simple terms, how they work and may be used to improve the animation workflow.

C-Motion

C-Motion is a tool so far unique to Cinema4D. It is a tag, which has been specifically created and optimised for animating repeating or cyclical motions, e.g., someone walking, a bird’s wings flapping, the pistons in an engine moving, etc. This tool works using hubs and targets, which have their own hierarchy similar to the already known parent and child hierarchy. When comparing the two, it would be adequate to say that Hubs could be seen as equal to parents, while targets are their child / children.
Within the C-Motion object, to animate a cycle, it is first necessary to set a hub-joint and a target joint. These two can then be continuously animated in a cycle by adding rotations, movement or other actions to the target. The phase tab can then be used to set the length, strength and the start- and endpoint of the desired cyclical movement via a curve.

Using the C-Motion and making sure the movements are either in-phase or out-of-phase to each other makes it easy to make walking appear natural, e.g., to also animate the movement of the arms and legs while walking, so when one leg moves back, the other moves forward and vice versa. The ease, with which a movement can be adjusted by simply adjusting either the strength or entry and exit point of the phase makes the C-Motion a very powerful tool for parametric animation.

XPresso

XPresso is a node-based programming language within Cinema4D. Due to its node based character, all interactions happen within the XPresso shader graph. Within the shader graph, XPresso allows for the connection and relation of different parameters between different objects, which basically allows for interactions between them to be programmed freely via their relations. This way, it becomes possible to, for example, programme, that, whenever object A moves or behaves a certain way, object B behaves a certain way as well. It is, in this regard, similar to constraints, though of course it allows for a much more precise and free interaction, as any relation or behaviour can, with adequate skill, be programmed exactly how the user desires.

To make use of XPresso and manually programme node-based interactions between objects, it is first necessary to add the XPresso tag to the target objetct(s). Then, by double clicking the tag, the shader graph can be access and the desired interaction programmed. When working with XPresso, within the shader graph, it is important to be mindful of the different in- and output parameters and make sure, that values match the desired results, e.g., when working on rotation, making sure whether or not the output of an angle in in degrees or radians, since it can have a huge influence on the later interacations. As such, proper use of re-calculation and re-mapping nodes, where applicable, is crucial.

Now, while it is possible to programme every single interaction manually with XPresso, Cinema4D also offers a shortcut for it. To simply connect two parameters via XPresso, it is also possible to right click the first parameter, which is supposed to influence the other, and chose XPressions -> Set as a driver. Then, simply right click the second parameter, the one to be influenced, and choose XPressions -> set as driven (absolute). This will automatically create a XPresso tag on the target and connect the two parameter in it. It will also change the parameters keyframing appearances to indicate the driving, where the position of the triangle indicated, whether it’s the driver or the driven. This is a very simple way to automatically connect two parameters, though of course it is also prone to error, since there is much less control over what is happening. At this point, it is important to note, however, that, if one is unhappy with the automated results or if the automation produces undesirable results, the XPresso tag can now be accessed manually as well and the interaction refined the same way as if programming it manually.
As such, XPresso can both be a very easy, but powerful tool, but also very deep and complex at the same time, as it allows for interactions between any objects or parameters within Cinema4D to be made as simple or complex as one’s skill allows. With it, a wide variety of complex interactions can be programmed, e.g., the influence of a light source on an object and its reflections or similar complex ideas.

The character object

Cinema4D’s character object is basically a built-in auto-rigging tool, that combines many of the different tools and techniques previously described and automates them to a certain degree. To understand what it is doing and how it works, however, it is important to understand all the previously described concepts, as it will help use the character object to its maximal potential.

Simple character rigged using Cinema4D’s character object

The character object comes with a wide variety of pre-defined tools and settings, that can be a shortcut when working on very basic meshes and can further cut production time when animating. It is important to note, however, that, same as many other tools in Cinema4D, it can be manually extended or adjusted however needed by the user. With the character objects, the rig can be set up manually or automatically, in a similar way to how normal rigging with bones and joints works. The character object, however, has the advantage that controllers, IK chains and similar controlls / additions can be added and adjusted much more easily than by adding them one by one in a very holistic manner., thus speeding up both the rigging and the animation process. It is one of Cinema4D’s most powerful tools when it comes to animation, as it can be adapted to fit almost any topology and comes with as wide variety of presets and supports, that make adding, for example, mo-capped animations to your mesh very easy.

Summary

To sum up the previous three blogposts, Cinema4D is a very powerful software solution, with a wide variety of tools and approaches to work with, that, when used correctly, can achieve remarkable results when animating. To achieve said results, however, it is crucial to both know about the described tools and be steadfast and secure in their use, which is something, that I am still working on. It is also always important to keep in mind, what the desired endgoal for the animation should be like, as some tools, such as, for example, the muscle object, may yield very little gain for the considerable amount of effort that goes into setting it up. When keeping all these things in mind, however, it can be remarkably simple to achieve great and fun results when animating with Cinema4D and I am very much looking forward to the results my journey with this software may still lead me to.

Sources:

1. Maxon Training Team.: Intro to Rigging in C4D: Part 3 – Constraints & C-Motion, https://www.youtube.com/watch?v=sUkRFEQGTqU [26.01.2024].
2. Maxon Training Team.: Intro to Rigging in C4D: Part 4 – Character Rigging/The Character Object, https://www.youtube.com/watch?v=Co2hPxJhgT4 [26.01.2024].
3. Maxon Training Team.: Intro to Rigging in C4D: Part 5 – More on the Character Object and Features, https://www.youtube.com/watch?v=QuAJRZKeRe8 [26.01.2024].

by max.fruehwirt - 30. Januar 202431. Januar 2024

04 | Character animation and rigging in Cinema 4D – 2. Rigging and animating simple geometries

What this blogpost is all about

This blogpost will expand further on the previous topic of character rigging and animation in Cinema4D. It will tackle the question, how a rig can be connected to a modelled geometry, how the created character can be animated and what role topology and edgeflow play when rigging a model. Furthermore, it will also explain the role of controllers and constraints when animating a rig.

Connecting the rig to the character / geometry

Parent – based rigging

Parent-based rigging is the most simple way to connect a piece of geometry and can be great for animating usually inanimate objects, like lamps or balls or smaller robots, such as claw-hands or the like. To parent – rig a geometry, simply make the geometry you want to animate a child of the joint you want it to affect. This allows for the geometry to the be influenced by only its parent joints, which makes it a great tool for animating basic shapes and geometry quickly. Furthermore, it also allows for the use of parametric objects when animating, since there is no need to access the points / vertices of the object.
As a result of its rudimentary functions and due to these limitations, however, it is not suitable for animating more complex geometry.

Point weighting, weight management, normalisation and point – weight – based animation

Point weighting, in comparison to parent – based rigging, is much more precise in how it affects the rigged geometry. The main difference between the two is, that instead of each mesh / piece of geometry being assigned as a child of one joint, with point weighting you can, as the name implies, weigh each singular point / vertice to multiple different joints. Point weighting, in this regard, controls how the edge flow and overall topology react to the movement of the joint. As such, for effective point weighting and weight – based animation to be possible, it is imperative to have a clean, artifact-free topology and consistent edge flow.

The weight – assigned in percentage – thereby describes, how much the movement of a joint affects the weighted geometry. The difference hereby is, that one point / vertice can be weighted to multiple joints at the same time and thus be affected by their movement. The percentage of the weight, in such a case, controls how much the point gets affected by the joints movement.
To give an example, if a point were to be assigned 50% weight of one joint A and 50% weight of another joint B, if you were to rotate joint A by 90° along the z-axis, but did not rotate joint B at all, the points would only rotate 45° (or, as assigned before, 50% of the weight). It is thus possible to say, that the assigned weight displays, how much control any given joint has over a point. It is important to note, however, that, in order to get clean animations and proper control over the points, each point needs to be assigned a total weight of 100% to the different joints, so it needs to be weighted at 100%. This is called normalisation and it ensures, that no point remains unaffected by the movement of the joints / remains stuck in place, when you move the rig along and thus deforms the topology and mesh in undesirable or unforeseen ways. As such, 100% absolute distribution is the goal when point weighting a mesh.

Simple weighted character mesh of a flying bird. The different colours show the weight distribution of the mesh to the different joints of the rig.

Of course, it is possible that, even with 100% absolute distribution, some parts of the mesh get squished or deform in a weird way when moved / rotated past a certain point. To prevent that, the fan-bones, mentioned in the previous blog entry, can be used. By simply assigning weight to the fan bone instead of its two parent joints, the rotation and movement of the mesh can be smoothened out and blended together in a way that prevents the undesirable squishing and / or deformation from occuring. Furthermore, a fan bone’s pole vector allows for a more precise control of the direction the topology deforms in, similar to how a joint’s pole vector allows for a more precise setting of the bend.

Constraints and controllers

To prevent self-intersections during movements and, as another way to influence and refine how a rigged mesh operates when animated, it is possible to use constraints. Contraints, as the name implies, constantly update and constrain certain parameters or properties of any object, joint or bone they are assigned to. This way, the assigned object can, for example, be made to follow a selected target or change its rotation or position depending on different situations. The set target for any given constraint is what is most commonly known as a controller.

Two good examples on a possible application of constraints would be the AIM and the PARENT constraint.

The AIM constraint

By adding the aim constraint to joint and setting a random object (e.g., a cube, sphere, camera, null, etc…) as a target (controller), the joint’s general direction and rotation as well as the direction of that joint’s bone will always follow the movement of the selected target, no matter, where in the workspace it is moved to. This can, for example, be used to rig a character’s head or eyes, so they follow an object’s movement or make them always focused on the camera.

The PARENT constraint

While the aim constraint mainly limits the direction an object is facing, without affecting its actual position, the parent constraint constrains the object it is applied to to the selected parent object and ensures that the position and rotation they have relative to each other remains the same. That way, if you move the parent object, you also move the constrained object / joint the same way, while, if you were to rotate the parent object, the constrained object would be rotated and moved around, so their relative postion would remain the same.
Here, it is important to note that, unlike the aim constraint, which assigns just one target for the object to follow, with the parent constraint it is possible to set multiple such targets. When doing so, it is possible to regulate the strength with which each target affects the constrained object and even to blend two different influences together.

Just by looking at these two examples, it should already become obvious why, when rigging an object, proper use of constraints can not only help make the animators work easier, but become outright necessary to ensure a smooth and impeccable animation flow.

Controllers

As mentioned before, the set target for a constraint is usually called a controller. Here it is important to note, however, that, by setting a null object or similar target as a controller, and using the proper constraints, the animation of a mesh can become way easier since, instead of having to animate each movement of any given part of the mesh by hand, it becomes possible to just animate the respective controller and, due to the previously applied constraints, have the mesh react accordingly and, if set properly, correctly and as desired. This is another reason, aside from ensuring proper movement and deformation, that makes constraints and controllers paramount to rigging and character animation in general.

Next steps:

Advanced tools:

Muscle Object
Pose Morph
Character Obect
Phase
XPresso and User Data sliders

Sources:

1. Maxon Training Team.: Intro to Rigging in C4D: Part 2 – Point Weighting, https://www.youtube.com/watch?v=oiEUwM7vdCg [26.01.2024].
2. Maxon Training Team.: Intro to Rigging in C4D: Part 3 – Constraints & C-Motion, https://www.youtube.com/watch?v=sUkRFEQGTqU [26.01.2024].

by max.fruehwirt - 30. Januar 202430. Januar 2024

03 | Character animation and rigging in Cinema 4D – 1. Introduction and basics

What this blogpost is all about

Once I started further looking into the topic of character animation and rigging, it became increasingly obvious, how big a topic this would be. As such, a decision was made to currently limit my research only to rigging and animation with the tools Cinema4D provides, as it is my current go-to software solution when it comes to modelling, rendering and animation. As such, while this will aim to provide a general picture and insights on rigging, the tools described will be following a standard rigging process in Cinema4D.

After working through all of Maxon’s „Hands on“ series, which is more than 13 hours of lectures, dealing with the topic Animation and rigging, and since Cinema is such a powerful software, that provides lots of different tools, my findings will be split into multiple parts:
The blog entries will explore what rigging even is, how it works, why it is important for character animation, how to build a rig, how to connect it to geometry and how to animate it. It will start with the very basics, giving an introduction and explaining concepts, and go all the way to the most advanced tools an animator can use when animating with Cinema4D.

First of all: What is rigging, and why is it important for character animation?

Rigging is a technique used in skeletal animation to move and / or present a character model using a series of interconnected digital bones and joints. More specifically, rigging refers to the actual process of creating a digital skeleton for a character model, that can then be linked to said model in order to move, rotate or animate it otherwise. The linked 3D geometry, in this case, then becomes almost like puppet of the created rigged skeleton and deforms and moves along with it. These movements of singular joints and bones, or even the complete model, can then be keyframed or otherwise recorded in order to create longer, continuous animations.

Great, but how does it work?

Bones and joints

To rig a model in Cinema4D, you first have to create its skeletal rig aand then assign you modeled character or geometry to it. By using the joints tool in the character menu, you can, as the name implies, place joints within your scene. These joints then get connected via bones and are what you would move around and / or rotate in order to animate your character later on. To connect two joints with a bone, simply make one joint into a child of the other joint in the object manager. Here, it is important to note, that hierarchy between the joints is VERY important during the rigging process and also later in animation, because, generally speaking, the direction of the bone goes from the object higher in the hierarchy to the object lower in the hierarchy, so always from parent to child. As such, it is usually a good idea, when animating a humanoid character, to start somewhere close to the center, where the spine would be, usually around the pelvis or the chest region, and go outwards from there creating shoulders, elbows, hips, knees, all the way to hands and toes. This is so important, since the direction of the bone also determines, in which way two joints influence each other.

Forward kinematics, inverse kinematics and their hybrid solutions

As mentioned before, the hierarchy between the joints determines their influence on one another. Generally speaking, a rig is dependent on the direction of the bones, as they are forward kinematic, meaning that, when you move or rotate a joint higher in the hierarchy (parent), all children of that joint (other joints and bones) get moved and / or rotated along with it. However, if you move a joint lower in the hierarchy (child), its parent does not get affected and remains in place. This is, in fact, intentional and desired and can be easily understood by imagining the following:
When you turn your upper body / torso, usually your neck and shoulders along with your arm rotate with it. However, when you rotate only your wrist, your upper body / torso usually remains unaffected. The same principle applies here.

There are, however, situations, where it may become desirable for a lower-hierarchy joint to affect their parent, e.g., when animating a character walking or jumping. When lifting or otherwise moving the foot – which is usually lower in the hierarchy than the knee or the hip – in such a situation, it is only obvious that the knee should bend accordingly at the same time and maybe even the hip to slightly rotate or move upwards, in order to look natural and avoid uncanniness.
To achieve that, it is necessary to create what is known as an inverse kinematic chain between the child-joint and its parent-joint. This chain can be understood as a direct reference between the goal joint, which is the end-joint of the chain (parent, upper hierarchy) and the influencing joint, which is the start-joint of the chain (child, lower hierarchy) and it enables for more fluid and natural looking movements to be created without the need of animating each joint seperately. It also is possible to define, which way a joint higher in the hierarchy / chain should bend or move by setting a pole vector.

So, to sum things up and break them down a little bit:

1) Forwards kinematics work from the top of the hierarchy downwards through the joint-chain, you have to tell each chain-link how to
move seperately and animate every joint as you need it while manually forwarding the movement down the chain.
2) Inverse kinematics work from the bottom of the hierarchy upwards through the joint-chain, you influence the last joint in the chain
and all the joints, that are connected and higher up in the chain react accordingly. This is especially helpful when animating legs, as it
helps them stick to the floor without moving through it, while maintaining proper form.

Understanding how forward and inverse kinematics work and making use of both systems within a rig (hybrid solution) is paramount for proper, good looking animations, that don’t look uncanny or otherwise off.

Simple joint and bone rig using both forward and inverse kinematics

Fan bones / joints for smoother rotations

Fan joints and fan bones are a form of controller – more on controllers will be explained in a later blog post – that basically are a smaller bone and / or joint. These joints, while being the child of another joint and being zeroed-out towards it – meaning it shares its parent’s location, rotation and scale – , act as a support, that blends the rotation of two joints together. It does so, by using a rotational constraint (rigging tag), where the rotational strengths of the two joints can be set. The blending between the joints is dependent on the strength set. This may seem confusing at this point, but its purpose will become clear in later entries. All that is important at this point is to know, that fan joints / bones exist and that they can blend the rotation of two other joints together.

Fan bone sticking out of a simple rigged pipe

Next steps:

Connecting the created skeletal rig to the character model / geometry
Parent – based rigging vs. point – weighting and binding
Edge flow and topology and why it’s important
Constraints and controllers

Sources:

1. Petty, J.: What is 3D Rigging For Animation & Character Design? , n.y.,
https://conceptartempire.com/what-is-rigging/ online in: https://conceptartempire.com/ [16.01.2024].
2. Maxon Training Team.: Intro to Rigging in C4D: Part 1 – Fundamentals of Rigging, https://www.youtube.com/watch?v=rN_5weIhiMY [12.01.2024].
3. Maxon Training Team.: Intro to Rigging in C4D: Part 2 – Point Weighting, https://www.youtube.com/watch?v=oiEUwM7vdCg [12.01.2024].

by max.fruehwirt - 15. November 202317. Januar 2024

02 | AI supported animation of rigged models

What this blogpost is all about

When looking into different methods of 2D and 3D character animation – be it with the goal to create a short scene or to create a responsive, engaging character – and possible workflows of character creation – the following video came up:

Video showcasing an AI-assisted keyframe-based animation workflow using Cascadeur

Thus, the idea of using AI to assist in different steps of an animation workflow was born. Hereby, the goal was to look for tools that do not completely automate the animation process, but rather that support the artist in a variety of subtle ways by automating some of the most time consuming or tedious processes of an animation workflow. The found results and software solutions will be shown in this blogpost.

AI and MoCap

It rapidly became apparent, that one of the most common applications of AI currently is in the automation of Motion Capture. Be it by using videos as a basis, employing MoCap suits for tracking or even live-tracking a movement via camera, a wide variety of softwares already use AI to automate the motion capture process and provide fully animated and rigged models at the same time or transfer the captured movement to a pre-existing model.

Radical – Real-time 3D human virtualization for everyone, everywhere.

https://radicalmotion.com/

Deepmotion – Create 3D Animations From Video Using AI

https://www.deepmotion.com/animate-3 d

Plask Motion – AI powered MoCap animation tool

https://plask .ai/

Now, while these softwares certainly differ in a small variety of ways, such as Radical promoting their seamless integration into a wide variety of existing 3D software, Deepmotion advertising the possibility of refinement via a rotoscope editor and their big community and Plask even offering additional outsorcing service for certain tasks, their overall purpose largely remains the same – namely, to capture a real life motion and automate its transfer into a digital environment, thus cutting this part of the workflow short. Given the wide variety of tools available, the question remains, however, on how to proceed from there or how to create new animations from scratch without using motion capture?

AI assisting in animation

Cascadeur – AI assisted keyframe animation software

https://cascadeur.com/

Cascadeur aims to ease the process of keyframe-animating already rigged characters by using AI to simulate existing physical forces that may influence a characters movement, such as gravity and momentum. It does so by automatically calculating the appropriate movement of all the different nodes within the rigged model in response to the movement of a single other node. The following tutorial video more accurately demonstrates, how this works:

Next steps

More research into different character-animation-workflows

Try out different workflows to see the advantages and disadvantages

Model a character, rig it and creat a small clip using the learned tools

by max.fruehwirt - 14. November 202314. November 2023

01 | Topic Exploration

1. Optimisation of production workflows in Western and Japanese animation:
Rotoscoping, Motion Capture and other Hybrid 2D/3D Techniques

This topic deals with the question of how Western and Japanese animators and animation studios try to optimise their production workflows and how they incorporate technologies such as Rotoscoping, Motion Capture as well as other Hybrid 2D and 3D techniques into their pipeline.
The aim is to provide accurate insight into different 2D and 3D production workflows and compare them along with the results they achieve. The main interest will therefore be the practical applications of such techniques and the comparison of the different results they achieve within their cultural, narrative and genre-specific context.
To summarise, after an in-depth study of the material, which will hopefully show both the advantages and the disadvantages of the different approaches, the next step will be to look for ways to incorporate the learned into my own design. This will be done via different case studies and short projects, such as short animations, character models or even a short movie.

1) History of Rotoscoping (https://www.youtube.com/watch?v=IS1hCSsmH1E)
2) Film “The Case of Hana and Alice” (https://variety.com/2015/film/festivals/the-case-of-hana-and-alice-film-review-1201544811/)
3) Series “Aku no Hana” (https://www.cbr.com/aku-no-hana-why-unpopular-anime/)
4) “Fate/Grand Order Memorial Movie 2023” (https://www.youtube.com/watch?v=L02aPFAhTCk)
5) Rotoscoping in Avatar: The Last Airbender (https://www.youtube.com/watch?v=DJ86QWzI2Wo)
6) Motion Captured Film “Kingsglaive Final Fantasy XV” (https://www.youtube.com/watch?v=UblcNFVVHT8)
7) Motion Capture at the set of “Ready Player One” (https://www.youtube.com/watch?v=6jo_SfurgkE)

1. Agarwala, A. / Hertzmann, A. / Salesin, D. / Seitz, S.: Keyframe based tracking for rotoscoping and animation . In: “ACM transactions on graphics, 2004, Vol.23 (3)”, ACM, New York 2004 , DOI: https://www.doi.org/10.1145/1015706.101576
2. Bratt, B: Rotoscoping, Routledge, 2021, ISBN: 9780240817040.
3. Davison, A. / Deutscher, J. / Reid, I.D.: Markerless Motion Capture of Complex Full-Body Movement for Character Animation, in: “Computer animation and simulation 2001”, Springer, Vienna 2001, ISBN: 3211837116.
4. Dunlop, R.: Production pipeline fundamentals for film and game. Focal Press, Burlington 2014, ISBN: 1-317-93622-1.
5. Kitagawa, M. / Windsor, B.: MoCap for artists: workflow and techniques for motion capture. Elsevier / Focal Press, Amsterdam / Boston 2008, ISBN: 9786611307301.
6. Menach, A.: Understanding motion capture for computer animation, Morgan Kaufmann, Amsterdam 2011, ISBN: 9780123814968.
7. Lanier, L.: Professional digital compositing: essential tools and techniques. Wiley Pub, Indianapolis 2010, ISBN: 9786612471834.
8. Lin, Z.: Research on film animation design based on inertial motion capture algorithm, in: “Soft computing (Berlin, Germany), 2021, Vol.25 (18)”, Springer, Berlin 2021, DOI: https://www.doi.org/10.1007/s00500-021-06001-y.
9. Miyakoshi, M.: Correcting whole-body motion capture data using rigid body transformation, in: “The European journal of neuroscience, 2021, Vol.54”, Wiley Subscription Services, France 2021, DOI: https://www.doi.org/10.1111/ejn.15531.
10. Roberts, S.: Character Animation Fundamentals: Developing Skills for 2D and 3D Character Animation. Routledge, London 2017, ISBN: 9786613267436.
11. So, Clifford, K.F. / Baciu, G.: Entropy-based motion extraction for motion capture animation, in: “Computer animation and virtual worlds, 2005, Vol.16 (3-4)”, John Wiley & Sons Ltd., Chichester 2005, DOI: https://www.doi.org/10.1002/cav.107´.

2. Gamification and Exploration in a VR environment

The goal of this topic would be to explore the different possibilities to work, play and design within a VR environment. With a very experimental approach, the final outcome of this topic can not yet be defined, but would very likely be some form of VR environment that allows for a playful and creative exploration of different applications and possibilities that such a digital space provides. Since VR can be seen as a completely different entity from our real world allowing for the addition of a second reality right within our lived-in one, with different rules, options and limitations, the current possibilities seem almost endless and are only restricted by our imagination as designers. As such and to set a boundary for the planned thesis, the exploration of two very particular topics is of special interest to me. These are:

2.1 An integrated approach to an interactive digital VR museum
Based on the current research project I am working on at the Graz University of Technology, funded by the FWF, the possibility of building a completely interactive, decentralised virtual museum within Virtual Reality could be researched. Possible topics to look into include but are not limited to: user interaction and how to increase engagements, tools to allow exploration, interfaces and information design, accessibility, visual narration and perception of space and many, many more. As such, more in-depth research of this topic could prove to be worthwhile.

2.2 Gaming and designing in / for VR
Another topic that seems quite promising is inquiring how gaming and exploration can even work in a virtual reality environment, both from a design standpoint and from the view of a user. To explore the different possible playful experiences that can be created and how to create them would be the main focus of this research. By looking into existing examples, such as VR games and the user-driven and collaboratively designed environment of VR chat, a deeper understanding of the topic could be achieved, which then in turn would be used to inform the design decisions shaping the new environment created as part of this research.

All in all, this topic is sure to be the most ambiguous and open among the presented, but also the one with the most potential for playful exploration and development. As such, VR and the inquiry of possible applications, is currently my favourite.

1) Virtual Reality in a Museum (https://www.museumnext.com/article/how-museums-are-using-virtual-reality/)
2) VR application Virtual Museum (https://www.artgp.fr/vr-application-virtual-museum-r.html?lang=fr)
3) “Half Life – Alyx”, VR Game (https://www.half-life.com/de/alyx/)
4) World / Rooms in VR Chat (https://www.uploadvr.com/10-amazing-vrchat-worlds-visit/)

Literature list:

1. Jerald, J.: The VR book: human-centered design for virtual reality, Association for Computing Machinery (pub.), Morgan & Claypool, San Rafael 2016, ISBN: 9781970001129.
2. Sherman, W.R. / Craig, A.B.: Understanding Virtual Reality: Interface, Application, and Design, Elsevier Science, Burlington 2002, ISBN: 9780080520094.
3. Caarrozzino, M. / Bergamasco, M.: Beyond virtual museums: Experiencing immersive virtual reality in real museums, in: “Journal of cultural heritage, 2010, Vol.11 (4)”, Elsevier SAS, 2010, DOI: https://www.doi.org/10.1016/j.culher.2010.04.001.
4. Lengua, D. / Lengua, M.: Moving Towards Digital Lands: on Processes of Interaction in Museum’s Virtual Spaces, in: “Meta space”, DISTANZ Verlag, Berlin 2022.
5. Plowman, J.: Unreal engine virtual reality quick start guide: design and develop immersive virtual reality experiences with unreal engine 4, Packt Publishing Ltd, Birmingham / Mumbai 2019, ISBN: 1-78961-504-6.
6. Mitsea, E. / Drigas, A. / Skianis, C.: VR Gaming for Meta-Skills Training in Special Education: The Role of Metacognition, Motivations, and Emotional Intelligence, in: “Education sciences, 2023, Vol.13 (7)”, MDPI AG, Basel 2023, DOI: https://www.doi.org/10.3390/educsci13070639.
7. Evans, L. / Rzeszewski, M.: Hermeneutic Relations in VR: Immersion, Embodiment, Presence and HCI in VR Gaming, in: “HCI in Games”, Springer International Publishing, Cham 2020, DOI: https://www.doi.org/10.1007/978-3-030-50164-8_2.
8. Yildirim, C.: Cybersickness during VR gaming undermines game enjoyment: A mediation model, in: “Displays, 2019, Vol.59”, Elsevier B.V, 2019, DOI: https://www.doi.org/10.1016/j.displa.2019.07.002.

3. 3D character modelling and narration in Virtual YouTubing / the V-Tuber-community:
How to make your character engaging via VFX and animations to underline your narration

Virtual YouTubing (V-tubing) has evolved over the recent years from a small niche community into an active and dynamic format of online content creation, where content creators interact with their audiences through animated 3D avatars. This topic delves into V-Tubing as a form of content creation, focusing on its combination of 3D character modelling and rigging and the narration and storytelling behind the characters of different creators. The main inquiry will be, how content creators use visual effects (VFX) and animations to create engaging characters that line up with their created narratives.

V-tubers, in this context, balance different skillsets, between technology and its capabilities to entertain and the storytelling aspect of their character. It is not surprising then, that some creators instead already opt to outsource one or the other, sometimes even both, to a digital artist, since the skillset required is expansive and the quality of many models in the competition is on the rise. Nonetheless, there is also still those that do everything themselves. The research aims explores the intricacies of character creation, using 2D character designs in combination with 3D modelling and rigging to craft distinct personas and then employing advanced animations and VFX to enhance their narrative expressiveness. By doing so, a deeper understanding of the complexity these creators are faced with, can be achieved.

The proposed research will then go on to explore the possibilities of viewers developing emotional attachment towards these digital personas, examining how the interplay of animations and VFX may enrich the viewer experience and engage the viewer. It will include studies into collaborations and manners of fan engagement – for example, fan art – that acts as the foundation of this community.

Finally, this topic aims to create one such V-Tuber model from scratch, going through all the most common steps from first draft to finished character. By doing so and by documenting the process, it will hopefully act as an investigation into the role of technology in facilitating the creation of these characters, including the software and hardware used to bring V-Tubers to life.

1) VTuber making a VTuber Avatar (https://www.youtube.com/watch?v=2THs99qG1Xc)
2) IronMouse Character Design (https://twitter.com/ironmouse/status/1654996287505268738/photo/1)
3) IronMouse Season 2 Model (https://virtualyoutuber.fandom.com/wiki/Ironmouse/Gallery)

Literature list:

1. Dazon, L.: Virtual Youtubers – What’s the appeal?, 2020,
https://cardiffstudentmedia.co.uk/quench/culture/virtual-youtubers-whats-the-appeal/ online in: http://cardiffstudentmedia.co.uk/ [14.10.2023].
2. Chen, J.: The Vtuber takeover of 2020, 2020, https://www.polygon.com/2020/11/30/21726800/hololive-vtuber-projekt-melody-kizuna-ai-calliope-mori-vshojo-youtube-earnings, online in: https://www.polygon.com/ [14.10.2023].
3. Lufkin, B.: The virtual vloggers taking over YouTube, 2018, https://www.bbc.com/worklife/article/20181002-the-virtual-vloggers-taking-over-youtube, online in: https://www.bbc.com/ [14.10.2023].
4. Shonzo: Making a Vtuber Avatar in BLENDER! Commission for Rakuu, 2021, https://www.youtube.com/watch?v=2THs99qG1Xc [14.10.2023].
5. n.a.: Ironmouse, 2019 – ?, https://virtualyoutuber.fandom.com/wiki/Ironmouse [14.10.2023].
6. n.a.: Shylily, 2019 – ?, https://virtualyoutuber.fandom.com/wiki/Shylily [14.10.2023].

What this blogpost is all about

Case 1 – Sousou no Frieren: A slow burner with a twist

Duality between Action and Slice of Life

Master and apprentice

Case 2 – Nier: Automata: The ultimate humanist fable?

A stylised cast and environment, and a large, varied number of tracks

A shift in perspective and a strong social element that transcends borders and time

Case 3 – Arknights: „Lone Trail“ and „Under Tides“: A holistic approach to ingame events

Lone Trail

Under Tides

A short summary and reflection of my journey so far

Sources:

What this blogpost is all about

Tracking, controllers, recognition and other interaction methods

Hands, eyes, voice and haptic feedback

Things to consider

Abstraction, intuitiveness, optimisation and sense of security

Next steps:

Sources:

What this blogpost is all about

How to create engaging, immersive worlds using level design

Understanding the importance of space and spatial awareness

Using realism, stylisation and interactivity to boost immersion

Design intuitive spaces and provide clear clues

Be aware of storytelling and how it is affected by 3D

Next steps:

Sources:

What this blogpost is all about

Discomfort, accessibility and other challenges

Motion sickness and other discomforts

Accessibility and impairments

Possible solutions and their integration

Comfort and customisation

Control and adaptability

Next steps:

Sources:

What this blogpost is all about

VR vs AR vs XR/MR – a comparison

Virtual Reality (VR)

Augmented Reality (AR)

Mixed Reality (MR)

Extended Reality (XR)

Current trends and technologies

Next steps:

Sources:

What this blogpost is all about

Deformation and changing of topology based on movement

Muscle Object

Pose Morph

Parametric and node based animation

C-Motion

XPresso

The character object

Summary

Sources:

What this blogpost is all about

Connecting the rig to the character / geometry

Parent – based rigging

Point weighting, weight management, normalisation and point – weight – based animation

Constraints and controllers

The AIM constraint

The PARENT constraint

Controllers

Next steps:

Sources:

What this blogpost is all about

First of all: What is rigging, and why is it important for character animation?

Great, but how does it work?

Bones and joints

Forward kinematics, inverse kinematics and their hybrid solutions

Fan bones / joints for smoother rotations

Next steps:

Sources:

What this blogpost is all about

AI and MoCap

AI assisting in animation

Next steps

1. Optimisation of production workflows in Western and Japanese animation:Rotoscoping, Motion Capture and other Hybrid 2D/3D Techniques

Literature list:

2. Gamification and Exploration in a VR environment

1. Optimisation of production workflows in Western and Japanese animation:
Rotoscoping, Motion Capture and other Hybrid 2D/3D Techniques

3. 3D character modelling and narration in Virtual YouTubing / the V-Tuber-community:
How to make your character engaging via VFX and animations to underline your narration