Extended reality will unlock the next era of innovation
with Jody Medich, XR & Cognitive Ergonomics
Extended reality will unlock the next era of innovation
Show Notes
Jody Medich is a designer and researcher working at the intersection of AI, XR (extended reality), and robotics. She is responsible for the hand gesture system on Microsoft HoloLens that became the industry standard for spatial computing interaction, and has spent her career on a single animating question: how do we build technology that amplifies what it means to be human - rather than replacing humans or forcing them to adapt to machines?
Her framework is cognitive ergonomics: the discipline of designing interfaces that operate at the speed of human thought by working with the brain's innate processing architecture rather than against it. As AI becomes more capable, Jody argues, the limiting factor is not the AI - it is the interface between the AI and the human mind. Solving that interface problem is, in her view, the most important design challenge of the next decade.
The Skeuomorphic Trap: Why We're Thinking About AI Wrong
When the web first appeared, digital galleries looked like physical rooms with images attached to walls. It was a bad user experience, but it was the brain reaching for a familiar model to make sense of something new. That skeuomorphic instinct - mapping new technology onto old mental models - is happening again with AI. We are treating AI as a humanoid entity and interacting with it through chat interfaces, because chat is the familiar model we already have.
The problem with this, Jody argues, is not that it is wrong - it is that it is incomplete. It misses what AI actually is. The parallel she draws is to the early PC era: mainframe computers existed, but it took Douglas Engelbart at Stanford Research Institute, the team at Xerox PARC, and eventually Steve Jobs at Apple to create a fundamentally new mental model for what computing was - the graphical user interface, the mouse, the “if you can point, you can use a computer” paradigm. Before that paradigm shift, computers were terrifying industrial tools. After it, they were personal productivity devices. The technology had not changed. The interface had.
AI needs its own mother of all demos moment - a paradigm-shifting interface that makes it clear to everyone what this technology is actually for, and how it can be used by ordinary people to amplify their own capabilities. That demo has not happened yet. Jody believes XR (extended reality) is the interface layer that will provide it.
Cognitive Ergonomics: Designing Interfaces at the Speed of Thought
Cognitive ergonomics starts from a counterintuitive observation: you are not using 10% of your brain right now. That is a myth. What is true is that roughly 10% of your brain is focused on conscious attention. The other 90% is doing continuous background processing - reading the room, monitoring the peripheral environment, tracking spatial relationships, regulating the body. All of this happens without your conscious involvement.
The current model of human-computer interaction funnels everything through that 10% conscious attention layer. Every notification, every menu click, every search term - it all competes for the same narrow bandwidth of deliberate thought. This is why technology feels exhausting. It is not using the brain efficiently; it is using only the least efficient part of it.
Cognitive ergonomics asks: what if we designed AI interfaces to use the other 90%? Spatial memory is one example - humans have precise, effortless recall of physical locations (“my keys are by the door”) but almost no spatial recall of digital files. Building spatial systems for digital information would leverage an existing, highly efficient cognitive capability instead of adding cognitive load. Background processing is another: technology that monitors context and acts on ambient cues rather than requiring explicit input. Eye tracking, convergence-based interaction (focusing closer to activate a function), and micro-muscle cues are all channels that already exist in the human body and could interface with AI without demanding conscious attention.
XR Is to AI What the Monitor Is to the PC
Jody's core framing: extended reality (XR) - which encompasses both augmented reality (AR) and virtual reality (VR) - is the display layer for AI in the same way that the monitor was the display layer for the personal computer. Without the monitor, the PC was a box that most people could not use. With the monitor, it became the most transformative productivity tool in history. XR serves the same function for AI: it provides the interface that makes AI's capabilities accessible to human perception and cognition.
Jody is specifically an augmented reality advocate - she prefers staying in the present world with enhanced vision rather than being fully immersed in a virtual one. Her framing is Superman's x-ray vision: seeing the real world with additional information layered on top, rather than replacing the real world entirely. The device on your face becomes a sensor suite pointed at the world - a camera that can see what you see, track where your eyes go, understand your environment - rather than a display showing you virtual content.
This reframe from “display device” to “sensor suite” changes what you can do with it. You can look at physical objects and ask AI to connect them. You can walk through the world collecting inspiration for a book and have the AI help you compile the fragments when you sit down to write. You can use convergence (the natural focusing-closer eye movement that happens when looking at near objects) to pull digital content toward you without any deliberate input. The interface disappears into behavior you are already doing.
Snow World and the Walk Again Project: VR's Proven Neurological Effects
Two research programs demonstrate what immersive XR can do to the brain at a neurological level:
Snow World. Burn patients experiencing pain so severe that morphine only reduced it by 20% were placed in a VR environment - an icy snowscape where they could throw snowballs at snowmen, accompanied by Paul Simon music. Vision is the dominant sense: the brain uses it to triangulate all other sensory input, including pain. In the VR environment, these patients experienced a 60% reduction in pain - three times more effective than morphine, with no drugs. Over time, the patients trained their brains to visualize the snowy scene without the VR headset, achieving pain reduction on demand.
The Walk Again Project (Duke University). Paraplegic patients with spinal cord damage lost communication between the brain's M1 motor neuron (which controls all physical movement) and their legs. Researchers placed a sensor at the M1 site and had patients use the same neural impulse that controls normal leg movement to control virtual legs in VR. After mastering VR movement, they were fitted with exoskeletons. At six months, patients regrew the musculature and circulatory systems needed to support walking. At one year, all eight patients in the study experienced voluntary leg movement without any apparatus. The brain had built net-new neural pathways because VR is processed by the brain as real - not as an image on a screen requiring cognitive translation, but as direct sensory experience that reactivates neuroplasticity.
Jody's conclusion: if XR interfaces to AI are designed to be genuinely immersive and sensorial rather than screen-based, they can create net-new cognitive capabilities in users - capabilities we have not yet imagined as possible for humans to have.
Why Apple Vision Pro Failed (and What It Reveals)
Jody's diagnosis of Apple Vision Pro's failure is crisp: it tried to put the existing computer interface on your face. The same mental model that governs the desktop - a screen displaying information you hunt and peck through - was preserved and miniaturized into a headset. That is a fundamentally wrong approach to spatial computing. The failure was not technical; it was conceptual.
The deeper issue is that we still think of each device as a self-contained surface - a magic eight ball screen that things come to and we interact with individually. As devices migrate onto the body (glasses, earbuds, wrist devices, smart clothing), the correct mental model is not “each device is its own screen” but “all of these together are augmenting my senses.” Once you make that shift, the entire design philosophy changes: the question is not what should appear on the screen but how should this technology enhance what I am already perceiving and doing.
Data Sovereignty and the Privacy Imperative of Ambient Technology
When Jody was developing gesture systems for HoloLens, she ran an experiment to test false-trigger rates: she asked colleagues to wear GoPros on their heads for a week. On day one, people were careful and self-conscious. By day four, all of them had forgotten the cameras were running - they argued with spouses, used the bathroom, had private conversations. All of it was recorded.
The lesson: the more ambient and invisible technology becomes, the more completely it captures the totality of a person's life. The current model - where the Googles, Microsofts, and OpenAIs of the world scrape that data to train their models - is not sustainable as a social contract. Jody points to the work of Jaron Lanier and Glen Weil on data sovereignty: the idea that individuals should own their data and be compensated when it is used to train AI systems, rather than having it extracted without consent or compensation. As XR makes ambient capture ubiquitous, this question moves from abstract to urgent.
Tools & Resources Mentioned
- Microsoft HoloLens - Jody designed the gesture interaction system that became the AR industry standard; she considers herself an augmented reality specialist
- Notebook LM (Google) - Jody's favorite AI tool alongside Claude; she finds it nearly there but not yet fully useful for her workflow
- Claude (Anthropic) - her current primary AI tool; uses it for a wide range of tasks
- Snow World VR (University of Washington) - burn patient pain reduction experiment; 60% pain reduction in VR vs. 20% for morphine
- Walk Again Project (Duke University) - paraplegics using VR/EEG training to regrow neural pathways and achieve voluntary leg movement
- Douglas Engelbart / Xerox PARC / Stanford Research Institute - credited for creating the GUI + mouse paradigm that transformed computing from industrial to personal tool
- Jaron Lanier + Glen Weil - referenced for their data sovereignty framework; the argument that individuals should be paid for contributing data to AI training
- Be Computer / Field Lead - ambient wearable AI devices Jody has tried; not using daily due to privacy concerns
Frameworks
Cognitive Ergonomics
The discipline of designing human-AI interfaces that operate at the speed of human thought by using the brain's existing, largely unconscious processing architecture. Conscious attention is only about 10% of the brain's activity - everything else is background processing that runs continuously without deliberate engagement. Current technology interfaces compete for the 10%. Cognitive ergonomics asks how to build interfaces that use the other 90% - spatial memory, peripheral processing, ambient monitoring, and micro-motor cues - to create interactions that feel effortless rather than demanding.
XR as the Monitor of AI
Extended reality (XR) is to AI what the monitor was to the personal computer. Without the monitor, the PC was an industrial tool most people couldn't use. The monitor created the interface paradigm that made computing personal and accessible. XR provides the same function for AI: a spatial, sensorial interface that can display AI's capabilities in the context of the real world rather than on a flat screen. The critical shift is from thinking of XR as a display device to thinking of it as a sensor suite - a machine vision layer pointed at the world that enables entirely new relationships between humans, AI, and their environments.
The Device-as-Sensor-Suite Mental Model
The current mental model for smartphones and computing devices is a self-contained surface: a magic screen where information appears and you interact with it. As devices move onto the body - glasses, earbuds, wrist devices, smart clothing - the correct mental model is not 'each device is its own screen' but 'these devices together are augmenting my senses.' This reframe changes the design question from 'what should appear on screen?' to 'how should technology enhance what I am already perceiving and doing?' The practical implication: stop designing apps, start designing sensory augmentation.
Vision Dominance and Neuroplasticity
Vision is the dominant human sense - the brain uses it to triangulate all other sensory experience, including pain, temperature, and proprioception. This is why the Snow World experiment works: an immersive visual environment can override pain signals at a neurological level, not through distraction but through genuine sensory recalibration. It is also why VR-based rehabilitation works for paraplegics: the brain processes VR content as real sensory input, which reactivates neuroplasticity and allows for net-new neural pathway formation. The design implication: immersive, embodied interfaces can literally change what users are capable of - not just help them do existing things faster.
Data Sovereignty Imperative
As ambient technology makes continuous environmental capture unavoidable - always-on cameras, microphones, sensors on the body - the question of who owns that data and who benefits from it becomes existential. The current model (large AI companies extract data from user behavior to train models, users receive no compensation) was established when the scale of capture was relatively limited. As XR and ambient wearables make total-environment capture possible, Jody argues this model must shift toward data sovereignty: individuals own their data, consent to specific uses, and are compensated when their data trains commercial AI systems. Without this shift, ambient AI creates the most invasive surveillance infrastructure in human history by default.
FAQ
What is extended reality (XR) and how is it different from VR?
Extended reality (XR) is an umbrella term that includes virtual reality (VR), augmented reality (AR), and mixed reality. Virtual reality replaces your visual field entirely with digital content - you see nothing of the real world. Augmented reality overlays digital content on top of the real world - you see both simultaneously. Jody is specifically an AR advocate because she prefers staying present in the real world with enhanced capabilities (she uses the metaphor of Superman's x-ray vision) rather than being fully immersed in a virtual environment. Her critique of Apple Vision Pro is that it tried to replicate the VR/computer-on-your-face model when the more powerful approach is treating the device as a sensor suite that enhances how you perceive and interact with reality.
What is cognitive ergonomics and why does it matter for AI?
Cognitive ergonomics is the design discipline focused on creating interfaces that work with the brain's natural processing architecture rather than against it. The brain has roughly 21 senses (not just five) and runs continuous background processing that handles the majority of environmental awareness without conscious involvement. Current AI interfaces funnel all interaction through the narrow bandwidth of conscious attention - the 10% of the brain we deliberately use. Cognitive ergonomics asks how to design interfaces that leverage the other 90%: spatial memory, ambient processing, micro-motor cues, eye movement, and peripheral awareness. The goal is interfaces that feel effortless rather than demanding - AI interaction that happens at the speed of thought rather than the speed of typing.
What did the Snow World and Walk Again experiments prove about XR?
Snow World (University of Washington) demonstrated that immersive VR can reduce pain 60% in burn patients whose pain was so severe that morphine only reduced it 20%. The mechanism is vision dominance: the brain uses visual input to calibrate all other sensory experience, including pain signals. Over time, patients trained themselves to invoke the pain reduction effect without the VR headset. The Walk Again Project (Duke University) had paraplegics use VR to control virtual legs using the same neural impulse that would move real legs. After sustained practice with VR and then an exoskeleton, all eight patients regrew the physical musculature needed for walking and experienced voluntary leg movement without any assistance. The brain had built entirely new neural pathways. Jody's conclusion: the brain processes XR as real - not as a screen to be translated - which means XR can retrain the brain and create genuinely new human capabilities.
Why does Jody think we need a 'mother of all demos' moment for AI?
In 1968, Douglas Engelbart gave what became known as the mother of all demos - a 90-minute presentation that introduced the computer mouse, graphical user interface, hypertext, video conferencing, and collaborative real-time editing in a single session. It established an entirely new mental model for what computers were and how people could use them. Before that paradigm shift, computers were industrial tools most people found terrifying. After it, they were personal tools that anyone could use. Jody argues AI is at the same inflection point: all the enabling technology exists, but we lack the paradigm-shifting interface demonstration that shows ordinary people what AI is actually for and how it will work in their daily lives. That demo has not happened yet, and until it does, most people will continue to think of AI primarily as a chatbot.
What are Jody's concerns about ambient AI wearables?
When testing gesture systems for HoloLens, Jody had colleagues wear GoPros for a week. By day four, all of them had forgotten the cameras were running and were captured in fully private moments - arguments, bathroom use, intimate conversations. The experiment illustrated that ambient capture becomes invisible to the wearer extremely quickly. As XR and wearable AI become more common, the result is potentially the most comprehensive personal surveillance infrastructure ever created. Jody's concern is that the current model - where large AI companies extract behavioral data to train commercial models without compensating users - cannot be the right social contract for data collected at this intimacy and continuity. She points to Jaron Lanier and Glen Weil's data sovereignty framework as the direction this needs to go.
What would Jody study in college today, and what AI tools does she use?
Jody would study art. Her reasoning: as AI handles more of what has historically required technical skill, the ability that will differentiate people is creative intelligence - the capacity to imagine new things, make unexpected connections, and produce work that only a specific human perspective could generate. Being literate in coding will matter, but coding as a career skill may be largely automated within five years. Creative thinking is harder to automate and more foundational. For AI tools, she uses Claude as her primary tool and Notebook LM from Google as a close second - she finds it nearly but not yet fully useful for her workflow as a writer and researcher.