In this blog post, I want to showcase and discuss my finished prototype. Additionally, I’ll theorize about the future outlook and potential advancements in this area.

So first of, the showcase video:

Showcase.mp4

For this prototype, I used the sensors embedded in my smartphone, which I found to be a valid approach. However, for future prototypes, using a smartwatch or another tracking device may prove to be more useful. There is also the potential for creating specific health devices, such as a resistance band with sensors to detect how strongly the band is being stretched or pulled.

• Device Selection: The weight of the phone was noticeable during use. For a more refined prototype, a lighter device would be preferable. For reference, the phone I used, a Motorola Edge 30, weighs 155 g.
• Sensor Integration: Utilizing wearables like smartwatches can provide more accurate and convenient tracking. Alternatively, integrating sensors directly into exercise equipment (e.g., resistance bands) could enhance the user experience and data accuracy.

The research I did and the buidling of the prototype taught me a lot about gamification and therapy methods. It was the first time where I really took a look at how therapy exercises look like and how they should be performed. Also from the technical side I gained many valuable insights. Right now the prototype gives immediate feedback. However, in the future it could be enhanced by creating really gamifying it. This could happen in the form of characters, stories that are being told, audio feedback, rewards, etc. This is definitely something I want to explore more of in the future.

One of the proplems I personally have with most of the gamified approaches to therapy which I have seen during my research phase is that so many of them seem either too silly for my personal taste and that they are old and not up-to-date. Finding something new and novel that is both fun, engaging and yet still serious enough that people will see it as a legitimate form of therapy will be one of the challenges which need to be overcome.

Conclusion

The process of researching and creating the prototype has provided me with valuable insights into the potential of gamified physical therapy. Moving forward, I plan to refine the technology and explore new avenues for enhancing the user experience and effectiveness of these solutions. I am currently in contact with some experts in the field so hopefully there will also be the possbility of cooperation in the future.

Sources:

• https://thephysioshop.ca/wp-content/uploads/2020/09/Banned-Shoulder-Flexion.webm

This blog post details my first proper implementation of a gamified exercise prototype. I will explain the steps I took and share my thought process along the way.

As mentioned before, there are multiple ways of providing (gamified) feedback to a user. I briefly want to talk about the differences between short-term immediate and long-term delayed feedback:

• Timing: Immediate feedback is instantaneous, while long-term feedback is delayed.
• Purpose: Immediate feedback enhances the act of exercising, while long-term feedback enhances the overall experience by providing a broader perspective on progress.
• Type of Feedback: Immediate feedback is typically corrective and directly related to the action performed, while long-term feedback is cumulative and provides a broader perspective on progress.

Choosing the Exercise and Sensors

When creating an actual prototype, I first looked at:

a) Which exercise I could replicate, and b) Which sensors were available to me.

Selecting the Exercise

I researched various sources and training programs to find exercises that could be easily enhanced digitally. I chose this specific exercise because it seemed easy to track, the resistance band allowed space to mount the phone, and it was a good exercise mentioned in different therapy programs.

Testing the Sensors

I explored various sensors available through Sensors2OSC. After testing, I found that the gyroscope, gravity, and accelerometer provided the clearest feedback when performing the exercise. I decided to use the gravity.z value as it gave the most reliable data.

Developing the Prototype

I initially wanted to send data to Unity as a Vector3 (using Keijiro’s OSCJack) but encountered issues with proper message transmission from Max 8 to Unity. I eventually decided to send the data as an integer.

Structuring the Code

Before starting the actual coding, I planned how to structure it. I modified my previous approach and decided to handle all game logic in Unity, using Max 8 only to receive and send data.

Updated flowchart for the data processing. Now there is a manipulation step in Max 8, and multiple values are sent to Unity for detecting repetitions and triggering actions/events accordingly.

I then modified my script and scene from last time. The UI now tracks repetitions and sets, showing the user when they have successfully performed the exercise. The code has grown significantly to handle these changes. It includes checks to recognize repetitions, locking mechanisms to ensure the count increases only once per repetition, UI updates, etc.

Coding was mostly a matter of planning and trial and error to figure out exact values. Interestingly, some planned functionalities, like setting a base value when pressing the spacebar, turned out unnecessary as the gravity data didn’t require it.

Lessons Learned

For this first prototype, handling all aspects (receiving data, triggering events, checking values, updating UI, etc.) in one script was manageable. However, for future iterations, separating the code into multiple classes and following the principle of data encapsulation from object-oriented programming would provide more structure. Additionally, establishing a naming convention beforehand could help avoid confusion in the future.

In the next blog post, I will show and discuss my findings and results from this prototype, along with a video demonstration.

Sources:

• https://www.semanticscholar.org/paper/A-Participatory-Design-Framework-for-the-of-Systems-Charles-McDonough/4149205989849ac7baf0fd4cd26843923aa192e9
• https://core.ac.uk/download/pdf/55533955.pdf Games & Feedback
• https://www.healthline.com/health/shoulder-mobility-exercises
• https://www.whyiexercise.com/physical-therapy-exercises.html
• https://thephysioshop.ca/blog/upper-body-stretches-for-better-flexibility-and-posture/
• https://www.hingehealth.com/resources/articles/arm-stretches/
• https://medi-dyne.com/blogs/exercises/upper-arm-pain-exercises
• https://www.whyiexercise.com/physical-therapy-exercises.html
• https://www.dunsboroughphysio.com.au/physio-exercises-upper-limb-flexibility.html
• https://thephysioshop.ca/blog/upper-body-stretches-for-better-flexibility-and-posture/
• https://www.hingehealth.com/resources/articles/arm-stretches/
• https://medi-dyne.com/blogs/exercises/upper-arm-pain-exercises

The goal of this blog post is to explore the entire process of creating a gamified application for use in therapy. I will examine the implementation of gamified physical therapy, from identifying the target group to maintaining long-term motivation.

Understanding the Target Group:

The benefits of gamified physical therapy can extend to various demographics, including children, adults recovering from surgery, and elderly patients. Each group has unique needs and preferences:

• Children: Engaging games can make therapy feel less like a chore and more like play.
• Adults recovering from surgery: Structured challenges can help them regain strength and mobility.
• Elderly patients: Gentle, gamified exercises can improve balance and prevent falls.

Conditions such as stroke rehabilitation, chronic pain management, and musculoskeletal injuries are particularly well-suited to gamified therapy.

Designing the Gamified Therapy Program:

Designing a gamified therapy program requires careful consideration of the specific requirements of both the exercise and the user. Several design frameworks, specifically tailored for therapeutic applications, can aid in this endeavor. Examples include the GAME and PACT (People, Aesthetics, Technology, Context) frameworks. Traditional game design frameworks, such as the well-known MDA (Mechanics, Dynamics, Aesthetics), are also applicable and can help in making design decisions. However, the more specified frameworks provide a more nuanced perspective.

Initial Consultation with a Specialist:

The process begins with a consultation with a physical therapist, who assesses the patient’s condition, sets realistic goals, and determines if a gamified approach is suitable. Customization is key, as the gamification elements must align with the patient’s specific needs and preferences.

Training During Therapy Sessions:

During therapy sessions, patients engage in gamified exercises under the guidance of their therapist. Immediate feedback is crucial, as it helps patients correct their movements and stay motivated. For instance, a VR game might involve reaching and grasping objects to improve hand-eye coordination and strength.

Encouraging Patients to Train in Their Free Time:

Consistency is key in physical therapy, and gamified approaches can help patients stay engaged even outside of scheduled sessions. Mobile apps and home-based VR systems are effective tools for encouraging continued practice. Examples of successful home-based exercises include:

• Balance Games: Using a balance board and a connected app to navigate obstacles.
• Strength Training: Gamified exercises where patients perform repetitions to defeat virtual enemies or complete challenges.
• Coordination Drills: Interactive games that require precise movements to score points.

Maintaining Long-term Motivation:

One of the biggest challenges in physical therapy is maintaining long-term motivation. Gamification addresses this by:

• Setting Achievable Goals: Breaking down the rehabilitation process into small, attainable milestones keeps patients motivated.
• Leaderboards and Social Sharing: Friendly competition and sharing achievements with a community can boost motivation.
• Regular Updates and New Challenges: Introducing new games and challenges prevents monotony and keeps the therapy engaging.

Evaluating the Effectiveness of Gamified Therapy:

To ensure the effectiveness of gamified therapy, it’s important to regularly evaluate progress. Methods include:

• Patient Feedback: Gathering qualitative data on the patient’s experience and engagement.
• Progress Tracking: Using the data from apps and devices to monitor improvements in strength, mobility, and endurance.
• Clinical Outcomes: Assessing functional gains and comparing them to traditional therapy methods.

Research supports the efficacy of gamified approaches. For example, studies have shown that VR-based therapy can improve motor function and cognitive recovery in stroke patients more effectively than conventional methods.

Future Directions and Innovations:

The future of gamified physical therapy looks promising with emerging technologies and ongoing research. Potential innovations include:

• AI and Machine Learning: Personalizing therapy programs based on individual progress and adapting exercises in real-time.
• Wearable Technology: Integrating sensors to provide more precise feedback and data tracking.
• Mixed Reality (MR): Combining VR and AR for more immersive and interactive therapy experiences.

Conclusion:

Gamified approaches in physical therapy offer a compelling solution to increase patient engagement, motivation, and ultimately improve outcomes. By leveraging game design elements and cutting-edge technology, therapists can create more effective and enjoyable rehabilitation programs. If you or a loved one is undergoing physical therapy, consider discussing gamified options with your specialist to enhance your recovery journey.

Sources:

• https://www.semanticscholar.org/paper/A-Participatory-Design-Framework-for-the-of-Systems-Charles-McDonough/4149205989849ac7baf0fd4cd26843923aa192e9

In my last blog post, I discussed the technical aspects required to create a basic functioning digital prototype. In this post, I want to delve deeper into what a potential first prototype could look like.

Before I speculate on the prototype, I want to mention a game we made last semester called “Lupos,” a simple 3D jump ‘n’ run game. Its distinctive feature was its control method: we used an Xbox Kinect to track the player’s body position and gestures. Leaning left or right steered the player character, while raising both arms above the head made the character jump. Since it was a cooperative game, the gestures had to be made in unison. I mention this game because it shares similarities with games used in rehabilitation and therapy. Using body movements to control in-game avatars can be a form of physical exercise, especially actions like raising hands, which require significant flexibility and range of motion.

As mentioned in my previous blog posts, there are different approaches to gamification. One consideration is whether the service should be used during sessions with a medical professional or during the patient’s own training time.

Another decision is whether the gamification should focus on the exercise itself (turning the exercise into a game) or provide user motivation outside of these exercises (e.g., with leaderboards, visualizations, etc.). These approaches can also be combined.

For a first prototype, I will focus on creating an application that can reliably measure a specific movement and track how many repetitions a user has completed. Additionally, I will explore how this information is presented.

I do not yet fully know the direction I want to take this idea, but I plan to explore it naturally through the prototype development process.

The next step is to build a low-fidelity prototype, focusing on functionality rather than aesthetics. Accurately detecting repetitions without accidental triggers will likely be the most challenging aspect.

For this blog post, I explored various materials that might be relevant to my research, including articles, papers, books, and more. While I aimed to prioritize scientific sources, I did not limit myself exclusively to them.

The first source I examined was a paper titled “A Therapy-Driven Gamification Framework for Hand Rehabilitation.” This paper presents an innovative e-health framework that uses adaptive serious games to facilitate hand rehabilitation for individuals with disabilities. Their end product was a serious game where players control an airplane or spaceship using the rotation and movements of their hands. What I found particularly intriguing was that the system accommodates users with vastly different levels of hand flexibility. According to the paper, the normal range for wrist flexion is 75°, but some individuals may only be able to flex their wrist up to 20°. The system can register these varying levels of flexibility and then devise a plan to gradually improve the patient’s ability through multiple steps. Additionally, I was interested to see that they used Leap Motion as the controller, a piece of technology we received a workshop on.

“Digitale Gesundheitsinterventionen” was an extensive and challenging read. It covers various digital health interventions and options, with gamification being more of a side topic. The main focus is on IMIs, or “internet- und mobilbasierte Interventionen” (internet- and mobile-based interventions). Nonetheless, it was insightful to browse through the book as it addresses a wide range of health problems. There are also dedicated chapters on specific target groups, such as the peculiarities of childhood and adolescence. I found the chapters on “Persuasive Design” and “Ethische Risiken” (ethical risks) particularly insightful.

The paper “Gamification in Musculoskeletal Rehabilitation” caught my interest because it focused on analyzing the general usefulness of gamified approaches. Seven articles were analyzed, and the results showed that gamification can be genuinely beneficial and is not just a novel idea without real merit. Since I am considering this topic for my master’s thesis, it was important to me to find data that provides a solid basis for argumentation.

I also read passages from “Making Points the Point: Towards a History of Ideas of Gamification,” “Rethinking Gamification,” and “Actionable Gamification” by Yu-Kai Chou. These reads focused more on gamification in general and less on health. I skimmed through them, reading passages that seemed interesting. However, I am not yet sure whether I will include them in my further research as they did not seem as directly applicable as others might.

I believe it is important to continue conducting traditional research alongside these more hands-on explorations.

Sources:

• Afyouni, Imad, Faizan Ur Rehman, Ahmad Muaz Qamar, Sohaib Ghani, Syed Osama Hussain, Bilal Sadiq, Mohamed Abdur Rahman, Abdullah Murad, and Saleh Basalamah. “A Therapy-Driven Gamification Framework for Hand Rehabilitation.” User Modeling and User-Adapted Interaction 27, no. 2 (June 2017): 215–65. https://doi.org/10.1007/s11257-017-9191-4.

In this blog post, I want to explain my process behind starting the prototyping phase.

A lot of considerations and decisions had to be taken into account for this project. Since it wasn’t clear to me what I wanted to do exactly, I started with a technical feasibility-first approach. It was clear that regardless of what I decided to do, I needed a way to track data and then utilize it somehow. Therefore, I decided to start by getting data from a tracking device to a game engine. I chose to use OSC (Open Sound Control) to transfer information from one device to another since we have been using it extensively during our studies, and I already have a good grasp of the concept.

For the tracking device, I chose my phone (Samsung Galaxy S23, running One UI 6.1, Android 14) and the app Sensors2OSC (not available on the Play Store, but the .apk can be downloaded from their website https://sensors2.org/osc/, and it can also be installed via F-Droid). For the game engine, I chose Unity because I am already familiar with it, whereas something like Unreal would require time to get used to.

To get the OSC data into Unity, I used an external framework. After some research, I narrowed it down to these two choices:

• keijiro/OscJack: Lightweight C# implementation of OSC server/client
• Iam1337/extOSC: extOSC is a tool dedicated to simplifying the creation of applications in Unity with OSC protocol usage

Ultimately, I decided on OscJack for several reasons. It was very lightweight (unlike extOSC, which comes with a lot of features), well-documented, and relatively new. Furthermore, keijiro has a good reputation in the GitHub/Unity community.

With this settled, I started to create a simple technical proof of concept in Unity.

OscJack comes with an OSC Connection component where you can define the type of message, host address, and port. The host is left empty because it is acting as a receive-only connection right now.

The scene in Unity is very simple. The only necessary object is the GameObject titled “OSC Event Handler,” which has the OSC Event Receiver script attached to it.

This script allows you to receive data and invoke a Unity Event that utilizes this data. For testing purposes, I created a simple script that outputs a text message and the value to the console. In this case, I used a simple int, but other data types such as Vector3 are also supported.

To quickly test if the setup was working, I created a Max8 file that sends an integer to Unity. It worked almost instantly without much troubleshooting.

It should also be noted that a dedicated OSC monitor exists, which may prove very helpful for debugging in the future. Also if anybody is wondering why my Unity windows appear reddish – it is because I set it up so the colors is changed while in play mode. Can definitely recommend so that no changes are lost.

In the final step, I added my phone to the process. I sent the data from my phone into Max8 and from there into Unity.

I probably could have done this without Max8 as an intermediary, but it is a very useful tool as it allows for rapid prototyping and easy cleanup of the data before handing it to Unity.

To have a backup system in place and to also allow me to work on multiple devices (as I work on both a laptop and a PC), I set up a GitHub repository for this project.

This was an important first step in getting a functional prototype working. In my next blog posts, I want to explore the concept and idea behind a potential prototype rather than focusing on technical details.

In this blog post, I want to explore the existing approaches to gamification in the field of physical therapy. The goal is to identify current methods, examine the problems they solve, and determine the contexts in which they are most effective.

Additionally, I wish to investigate the question: “What kinds of data can be tracked and utilized in creating a gamified experience?” Depending on the technology used, a diverse array of factors can be monitored, including:

• Steps, GPS location, distance traveled
• Heart rate
• Calories burned
• Altimeter (vertical distance)
• Active minutes per day
• Specific exercises/activities
• Sleep time/quality
• Gesture detection
• Movement detection (x/y/z axis)
• Accelerometer

Some specialized hardware can track even more specific metrics, such as:

• Skin temperature
• Breathing rate
• Heat loss
• Sun exposure (UV light)

All these different types of data can be used to create various applications. Here are some examples related to the healthcare industry. But first, a quick definition of „gamification“ is in order. To paraphrase, gamification is the use of game-design elements in a non-gaming context with the goal of enhancing a service by incorporating game-related features that support and enhance the user’s overall experience . Here are some services I found, listed in no particular order:

• PreOperative Diet App (tracking of food intake)
• Immersive VR games
  • Football goalie game
  • Catching coconuts that fall from above
  • Laundry sorting
• Smartphone Apps
  • MySugr Junior app: An app for children and parents to track blood sugar levels for children with diabetes
• Wii Fit: The Wii Fit Board has been used for several healthcare applications
• ADHD video game treatment

Also this video is very interesting: Gamified smartphone app helps diabetes patients | mySugr develops amazing, gamified applications to help patients with diabetes manage their treatment. | By The Medical FuturistFacebook | Facebook

Another novel form of gamification, though not perfectly aligned with traditional academic/health settings, is the use of pedometers and step counters by Game Freak and the Pokémon Company. Over the years, there have been many attempts to capture the sense of exploration from the Pokémon games and bring it into the real world. They have created extra devices to enhance the normal Pokémon game experience. For every seven steps you take, you generate one Watt. These Watts can be used to improve your friendship with your virtual companion, similar to a Tamagotchi, or to exchange for in-game items. The first of these devices dates back to 1998 with the release of “Pokémon Pikachu.” Later iterations included mini-games and a deeper connection to the main game. In 2016, Pokémon GO was released, which became hugely popular. While more of a game than a gamified app, it nonetheless encouraged physical activity.

While researching for this blog post, I gained several interesting insights:

It seems there are two (or more accurately three) approaches to gamified therapy. One could create:

• Something to be used during training sessions
• Something to be used at home
• A holistic approach spanning both therapy sessions and exercise outside of a medical setting

Additionally, one can make the act of therapy rewarding by creating game-like applications for the exercises, or focus on the process as a whole and motivate users by showing them their progress through statistics, visuals, etc., and rewarding prolonged positive behavior.

Lastly, I want to discuss potential obstacles or problems with a gamified approach to therapy. One common problem is longevity—the need to be motivating over a long period. In the context of physical therapy, it can be hard to predict how long treatment will last as it depends on the type of injury. Estimates can range from 2 weeks to 12 weeks or longer. Physiotherapy sessions typically last 30 minutes to 1 hour. Another aspect to consider is creating the right product for the target group. Children and teenagers are generally more open to a playful, game-like approach, while adults or elderly people may find these applications strange or childish. Additionally, if there is a need to store data externally on a cloud, privacy and ethical concerns are important. For example, BetterHelp had to pay $7.8 million for selling sensitive mental health information to Facebook and other marketing companies.

At this point, I could delve further into detail, but I think this is a good place to conclude for now. Designing a gamified experience, especially in healthcare, requires careful consideration. As designers, we need to be aware of these considerations to create the best possible product that can truly help people.

Sources:

• https://medicalfuturist.com/top-examples-of-gamification-in-healthcare/
• https://restoreskills.com/take-your-therapy-to-the-next-level-with-gamification/
• https://portlandwellnesscare.com/blog/how-long-does-physiotherapy-take/
• https://www.lifewire.com/what-wearables-can-track-4121040
• [1] https://www.sciencedirect.com/topics/computer-science/gamification
• https://wirtschaftslexikon.gabler.de/definition/gamification-53874
• https://en.wikipedia.org/wiki/Pokémon_Pikachu
• https://bulbapedia.bulbagarden.net/wiki/Pokémon_Pikachu
• https://www.ftc.gov/news-events/news/press-releases/2023/03/ftc-ban-betterhelp-revealing-consumers-data-including-sensitive-mental-health-information-facebook

In my previous research, I focused heavily on virtual reality (VR). After my talk, I realized I could start with a more physical approach and explore the broader topic of gamification.

To begin, I want to start with something smaller in scope. I will look at physiotherapy exercises, setting aside mental health therapy for now, and explore existing exercises and how one might create a simple gamified prototype.

I delved into the field of physiotherapy. Personally, I have little experience with this field, except for some physiotherapy sessions for my lower back condition, which I’ve had since birth. During these sessions, I noticed a recurring problem: after each meeting, I had a short burst of motivation to do the exercises, but this motivation faded after a few days. There were no disciplinary structures in place to keep me doing the exercises. This seems to be a common issue that medical professionals can hardly control outside of therapy sessions. Gamification could promote discipline outside of these sessions. Even during sessions, gamification could make the experience more enjoyable, especially for target groups such as children.

Statistics show that in our sedentary society, physiotherapy is on the rise. In Austria, 21.8% of the population has consulted a physiotherapist. Given the increasing numbers, we can infer that the figures for 2024 could be even higher.

I looked into specific physiotherapy exercises, focusing on the upper body. I chose this region because the exercises are potentially easy to measure, mostly simple, and could be combined with VR in the future, as VR is traditionally controlled via hand gestures or hand controllers. Although it’s hard to assess their effectiveness, these exercises were recommended by several sources, so I believe they are effective. For the prototype, I will pick one or several basic exercises.

In the next prototype, I want to explore what gamification is and which gamified approaches to therapy already exist.

Sources:

• https://www.dunsboroughphysio.com.au/physio-exercises-upper-limb-flexibility.html
• https://www.youtube.com/watch?v=hnCwHI75uVk&list=PLMLo0FsBVlPlY9Vzzr44bP8CJGBfaaHVw
• https://www.csp.org.uk/campaigns-influencing/campaigns/stronger-my-way/i-want-feel-stronger/strengthening-exercises/upper

Notes & Observations:

• Flexibility exercises seem to be performed by professionals rather than patients themselves, likely depending on the severity of health problems.

In this blog post, I want to talk a bit about my current ideas of how I could organize and tackle this topic.

Right now, I feel like the next possible steps could be to narrow down the specific use cases, i.e. which mental or physical health problems I want to focus on. It would make sense to tailor the selection to the field where a potential partner operates. Preferably, I would like to work together with the AUVA Rehabilitationszentrum Meidling in Vienna. It would be a good pick for several reasons. I already know that they utilize VR treatment (and that there are aspects that could be improved). My brother was there and knows some of the staff. And lastly, the distance to Vienna, which isn’t that far from Graz.

However, I also feel like I shouldn’t limit myself to one hospital/potential partner because I have to consider what will happen if it doesn’t work out and no cooperation will be possible. I have tried to contact them several times via email or LinkedIn. However, I haven’t gotten a reply yet. So I will try the more direct route and call them directly. Nonetheless, I should also do research into any other medical facilities in the area. I am sure that there are other opportunities for cooperation.

Thinking more broadly, I also believe that it would be a good idea to take a look at all my previous blog posts from the first semester and get to the essence of what I researched and found interesting. Using a Design Thinking approach, the first blog posts were widening the research field and now I need to focus and narrow it down again.

Lastly, more practically speaking, I believe it is a good idea to schedule regularly time slots just for Design&Research and set aside a fixed time for that. Even though the schedule is already quite full, I can set aside a couple hours on e.g., Monday, to really think about this topic.