Beyond Sight and Sound: How AI is Awakening Our Other Senses
At a recent TEDxMIT event, Paul Liang, an MIT professor and leader in the field of multisensory artificial intelligence, engaged in a compelling conversation about the future of AI and its potential to revolutionize how we perceive the world. The discussion explored the current limitations of AI, which primarily focuses on language and vision, and delved into the exciting possibilities of expanding AI's capabilities to encompass all human senses.
The Narrow Perception of Current AI
Paul Liang highlighted that today's AI systems, despite their advancements, possess a limited understanding of the world compared to humans. While AI excels at processing text and images, it lags far behind in replicating our ability to hear, touch, smell, and taste. This narrow focus restricts AI's capacity to genuinely interact with and comprehend the complexities of our environment.
The Power of Multisensory Intelligence
Liang's research group, Multisensory Intelligence, is dedicated to bridging this gap by developing AI that can perceive and process information from all senses. He emphasized the profound impact of smell, noting its ability to evoke memories, influence our perception of good and bad, and even provide insights into the past. Imagine, Liang mused, a future where we can share not just photos of our meals but also their aromas with loved ones.
Academic Foundations and the Rise of Multimodal Models
Liang's journey into multisensory AI began during his PhD studies in 2018, a period marked by the proliferation of deep learning in computer vision and the emergence of early language models. Initially torn between natural language processing, computer vision, and speech processing, Liang found his niche in understanding human communication. He recognized that human interaction is multifaceted, involving verbal cues, facial expressions, tone of voice, and gestures. This realization led him to develop multimodal models that combine these various sensory inputs to create a more comprehensive understanding of human behavior.
Ranking the Senses: The Potential of AI
When asked to rank the senses in terms of AI's potential, Liang acknowledged the significant progress made in language, vision, and audio, largely due to the abundance of available data. He identified touch as the next frontier, citing the Moravec paradox: what is easy for humans is hard for AI, and vice versa. While humans effortlessly grasp and manipulate objects, robots struggle with these tasks due to our inability to articulate the intuitive nature of touch.
Liang believes that advancements in tactile sensors, capable of digitizing force and spatial resolution, will soon enable AI to overcome this challenge. Smell and taste, however, remain the most distant goals, requiring new sensors, vast amounts of data, and innovative AI modeling paradigms.
Augmenting Reality: Haptics and Sensory Restoration
The discussion further explored the potential of AI to create artificial sensations and restore lost senses. Liang referenced the concept of crossmodal plasticity, where the brain compensates for the loss of one sense by enhancing others. His group is actively developing haptic intuition gloves that can allow individuals to feel sensations they would otherwise miss, offering hope for those with limited mobility or sensory impairments.
Beyond the Classic Five: Expanding Our Sensory Horizon
The conversation touched on the possibility of creating entirely new senses through technology. Liang suggested that memory could be considered a sense, with AI's vast storage capacity augmenting our ability to recall and utilize information from the past. He also highlighted the potential of AI to process physiological data, gathered from wearable sensors, to provide personalized insights into our health and well-being.
Brain-Computer Interfaces: A Distant but Promising Future
While brain-computer interfaces (BCIs) hold immense promise, Liang believes that their integration with AI is still 10 to 20 years away. He emphasized the need to address challenges related to robustness, privacy, and ensuring that AI augments rather than overrides human decision-making.
Building a Multisensory Team
Liang stressed the importance of interdisciplinary collaboration in advancing multisensory AI. His group seeks individuals with expertise in AI, biology, chemistry, neuroscience, and hardware development, fostering an environment where diverse perspectives converge to unlock new research opportunities.
A Glimpse into the Future: The Multisensory Experience
In closing, Liang offered a tantalizing glimpse into the future, describing an exhibition where individuals can input their favorite memories and experience them through a combination of sight, smell, and touch. This personalized multisensory experience exemplifies the transformative potential of AI to enhance our understanding of the world and enrich our lives.
Final Thoughts
Paul Liang's work at MIT is paving the way for a future where AI transcends its current limitations and interacts with the world in a truly multisensory way. By expanding AI's perceptual capabilities, we can unlock new possibilities for augmenting human abilities, restoring lost senses, and creating entirely new ways of experiencing the world around us. The journey is just beginning, but the potential is limitless.