Human interactions with objects revolve the five senses—sight, hearing, touch, taste, and smell—playing a essential role in perception. With the digital age in full swing, multisensory design is gaining significance. However, the most unforgettable experiences are usually multi-sensory based.This approach, considering senses and their interrelations, transforms experiences, opening up possibilities in product design, immersive storytelling, and user engagement. Beyond Human-Computer Interaction (HCI), insights from contemporary artists highlight the potential of blending sensory research with biometric sensors for interactions through mobile, sensor, and wearable technologies.
Multi-Sensory in Action
Multisensory design transforms society and consumer markets, shaping product experiences and immersive storytelling. Examples in contemporary art, like Jenny Tillotson’s interactive scent outfit, blending biometric sensors with sensory research. Challenges arise in using smell in media, as seen in Iron Man 3 screenings. Museums benefit from multisensory design; Mark Harvey’s research highlights its influence on visitor engagement, while the Jorvik Viking Centre immerses visitors in York’s Viking history through touch, taste, smell, and sound.
The SCHI Lab’s involvement in the Tate Sensorium exhibition underscores their expertise in multisensory design, enhancing user experiences through lighting, soundscapes, haptic feedback, scents, and taste. A study with 2,500 participants revealed the positive reception of sensory-enhanced art, with interesting nuances in visual liking based on gender.
The tactile experience usingmid-air haptic technology was iteratively designed and tested in the SCHI Lab before it was integrated into the Tate Sensorium—a six-weekmultisensory art exhibition at the Tate Britain art gallery in London. Each user was presented with tactile sensations on the hand while looking at a painting (a poster printout for the lab testing) and listening to synchronized sound.
Challenges in HCI
In HCI, despite significant advances in knowledge of sensory systems and devices in recent years, understanding of multisensory experiences is still incomplete. The goal is to leverage this method to improve human-technology interactions by creating more immersive and meaningful experiences. There are specific actions take to meet this challenge:
Design Sensory Experiences: Identify which senses can be intentionally designed and how to stimulate them effectively in tech interactions.
Build on Existing Models: Use current multisensory design frameworks, while also innovating and introducing new ones for more comprehensive user experiences.
Integrate Senses: Create interfaces that recognize and accommodate interactions between different senses, potentially combining taste and smell for richer digital experiences.
Manage Multisensory Info: Explore challenges of users monitoring information from multiple senses simultaneously. Find solutions to enhance the usability of interfaces using diverse sensory modalities.
Exploring New Senses in HCI
Studying touch, taste, and smell, as chemical senses, is a challenging task compared to physical senses like sight and hearing. The SCHI Lab is developing a device that uses touch to communicate emotions in real time, using ultrasound mid-air haptic stimulation and temperature and vibration patterns. This understanding is crucial for creating effective HCI interfaces and immersive media experiences.
The SCHI Lab aims to develop guidelines for creating scent delivery technologies and classifying smell experiences, focusing on the interactions between smell and other senses. This knowledge-based advancement will be relevant in various contexts, including entertainment, rehabilitation, and virtual reality, enabling designers to fully exploit our sense of smell. SCHI Lab’s study suggests a design framework for taste experiences, focusing on temporality, affective reactions, and embodiment. This could be useful for improving LOLLio, a taste-based game device. The framework allows designers to adjust tastes to create different affective reactions and a sense of agency, highlighting the importance of understanding the interactions between taste and other senses in human-computer interaction.
Exploring the communication of emotions through a haptic system that uses tactile
stimulation in mid-air.6 The study focused on specific design implications based on the spatial, directional, and haptic parameters of the created haptic descriptions and illustrated the design potential for HCI.
In conclusion, multisensory design is revolutionizing human experiences, especially in Human-Computer Interaction. Moving forward, the key next steps involve intentional design, innovation, integration of senses, and addressing challenges in managing multisensory information. These actions will contribute to creating more immersive and meaningful interactions between humans and technology.
Resource:
- K. Aisling, T. Virginia., “Multisensory Experiences in HCI” Artful Media.
