H. Branch Coslett, MD, was terrified as he stared at the ground below. He was shakily walking across a wooden plank hanging off the edge of the Empire State Building, and while he’d never thought of himself as afraid of heights, he couldn’t ignore the sinking feeling in the pit of his stomach. The figures he could make out on the sidewalk were at least a thousand feet down. As he neared the end of the precariously perched plank, he heard someone encouraging him forward: “Go ahead! Take that extra step.” No way.
In reality, Coslett’s feet were planted firmly and safely on the ground in a lab at the University of Nevada, Reno. Logically, he understood that, but as soon as he strapped a virtual reality (VR) headset on, he was overwhelmed by the power of the visual illusion. “I knew I was standing in a room, but the effect was striking in a visceral way. I was scared! The magnitude of the effects and what they induced was far more dramatic and compelling than anything I’d ever imagined,” he said. The former VR skeptic was hooked.
VR offers an immersive experience that simulates your physical presence in an artificial environment. Whether you’re on a video game battlefield, underwater, or floating through space, a VR experience that effectively mimics real world qualities like gravity, shadows, and depth can be very convincing for the brain. Coslett’s reaction to nearing the edge illustrates the impact of the brain being sufficiently tricked by the illusion. His brain perceived his simulated body’s placement and movement (proprioception) as dangerous, so cue the alarm bells and stress.
It’s these types of physical reactions to VR simulations that open some really interesting doors for neurologists, psychologists, and other researchers. While commercial headsets like the Oculus Rift and HTC Vive are often associated with gaming, VR has applications outside of entertainment, including in science and medicine. For example, studies are investigating whether veterans with post-traumatic stress disorder can be exposed to virtual triggers in order to work through their anxiety and stress.
At Penn, Coslett—vice chair of Neurology and co-director of the Laboratory for Cognition and Neural Simulation—has led the Neurology VR (NVR) Lab at the Perelman School of Medicine alongside computer scientist and Unity programmer Alex Miller since 2017.
The NVR Lab is a core resource of the Brain Science Center and is accessible to faculty, postdocs, and students across departments who are interested in integrating virtual, augmented, or mixed reality applications into their research. Using VR applications allows them to precisely control stimuli and modify subjects’ perception, all while collecting heaps of data. Since the lab’s founding, more than a dozen teams have reached out to collaborate and explore basic science questions, measure outcomes, develop diagnostic tools, and design therapies.
For example, one team investigated the psychological effect of architectural features by having subjects explore virtual rooms that Miller developed. Another team uses a game-like approach when studying patients with retinitis pigmentosa. In a new take on the classic Missile Command arcade game, subjects need to detect meteors emerging from their periphery, and a tracking system projects their hands into the game so they can stop the meteors by exploding them. (Who says rehab has to be boring?) Another project currently in development could help providers quickly diagnose vertigo in patients who come to the emergency department reporting dizziness. Providing a patient with a VR headset and tracking their eye movement could save them an unnecessary MRI scan.
Having a talented and passionate programmer onsite is a unique component of the NVR Lab. Because Miller developed the earliest applications from scratch, he is able to repurpose his own code to create new games and pursue new experiments. Whether researchers want to rent a headset for an hour or rent Miller’s creative coding skills for a few months, the NVR Lab’s staffing and setup ensure they don’t need to reinvent the wheel with each study.
“Alex is what makes the lab. It’s not about all this stuff,” Coslett said, waving to the shelves of headsets and tracking systems. “It’s about having somebody who knows what they’re doing.”
“We’re exploiting the interests of a bunch of gamer teenagers to do some really important and exciting work!” he continued with a laugh. “VR allows us to explore our hypotheses in a naturalistic way that wouldn’t be possible with traditional research methods.”
As VR technology continues to evolve, there are seemingly endless opportunities to explore both basic science and clinical questions, train medical students and health care professionals, and develop therapies and treatments. Here are just a few of the projects Coslett and Miller are currently working on:
Treating Phantom Limb Pain
The question: When an individual has a limb amputated, it’s theorized that their brain often subs another body part’s sensory input in place of the missing limb. While the brain is typically adaptable following a change in experience (plasticity), this change has negative consequences; the mismatched signals cause the brain to interpret the erroneous sensory input as debilitating pain. How can you treat phantom limb pain that’s resistant to medication and other interventions?
The traditional method: Mirror box therapy is often practiced to essentially trick the brain into not feeling pain. When a mirror is placed between an intact leg and an amputated leg, for example, the brain sees the healthy leg and its equally healthy reflection. If the intact foot wiggles its toes, both feet are seen doing it. The brain interprets that visual information as though the amputation has not occurred and resolves the phantom limb pain signals.
Penn’s VR application: Tracking sensors are placed above and below the knee to determine where the user is trying to move their intact and absent legs, and their VR avatar mimics these movements. One game involves feeding a puppy and kicking away menacing monsters, while another requires you to navigate a maze while pushing yourself around in an office chair and shoving blocks into pits. Miller also created a VR web browser in which users must use their feet to type out their searches.
“You’re controlling things like the cursor with your foot, even though there’s no foot there. You learn to control your responses using visual feedback,” Coslett said. “It's very cool, and it’s a powerful effect. We found that patients had lower ratings of pain, much lower ratings of depression and anxiety, and much lower ratings of failure to interact in the world because of fear of pain.”
Measuring Hemispatial Neglect Severity
The question: Following a stroke or traumatic brain injury, patients can exhibit hemispatial neglect. This can also occur in patients with neurodegenerative conditions, such as Alzheimer’s disease. The damage to the brain most often causes patients to behave as though the left side of space is nonexistent. How do you determine the severity of the neglect?
The traditional method: Hemispatial neglect is often diagnosed with a variety of paper tests. A patient who is asked to draw a clock may only include the numbers 12 to 6, or all of the numbers may be crammed into the right half of the clock. If they do draw the left side, it will likely be distorted or empty. Similarly, if a line is drawn on a piece of paper, and they are asked to designate the midpoint, they may cut the line off towards the right of the center.
Penn’s VR application: Miller developed a VR scenario in which a user examines three bushes covered in flowers, some symmetrical and some asymmetrical. They are tasked with picking all of the asymmetrical flowers, but if they are experiencing neglect, they may have a harder time noticing the asymmetries on the left side of the flower, the left side of the bushes, or the whole left side of the VR world. An eye tracking system detects what areas they are focused on, and the asymmetry becomes increasingly subtle in order to diagnose the earliest stages of neglect.
“If you have someone draw a flower and they omit the entire left side, that’s pretty overt. But what if the left side is just sparser or uneven?” Coslett said. “You’re not going to have a situation where there is a magic pill that brings someone from profound neglect to no neglect. You need a good measure of the severity in order to know if therapies are useful.”
Replicating Virtual Hand Illusion
The question: Some projects are focused less on clinical applications and more on understanding how the brain works. For example, we know that the brain integrates information from multiple sensory systems to understand where body parts are at any given time. But what happens when what you see (visual information) conflicts with what you feel (tactile information)?
The traditional method: The classic rubber hand illusion is induced when a participant places one of their hands out of sight and a lifelike rubber hand is placed in front of them. A researcher simultaneously brushes the fingers of the real and rubber hands at the same speed. Within minutes, seeing the rubber hand being stroked causes the participant’s brain to combine the visual and tactile information and conclude that it’s actually part of their body. Whip out a hammer to hit the rubber hand, and the subject—having taken ownership of it—will likely jump away in alarm!
Penn’s VR application: VR has allowed Miller and Coslett to take the illusion to another level. Users play a concentration card game, but while they’re focused on flipping over pairs, their simulated hand is slowly being rendered higher and higher from where it actually is. When asked to point to their real hand after a while, the false proprioceptive feedback they have received through the VR rendering causes users to point closer to the virtual hand. Just like a subject’s brain takes ownership of the rubber hand, the VR user accepts the rendered hand as their own.
“Since getting that similar rubber hand effect, we’ve started exploring other things, like implicit racial bias. How much does a hand have to look like your own for the illusion to work?” Miller said. “So far, we’ve found that white participants get the strongest effect with white hands, then purple hands—a totally alien thing with no attached biases—then black hands, and black participants are doing the inverse. This suggests that embodiment is a function of race, likely among other things.”
Are you interested in collaborating with the Neurology Virtual Reality Lab? Share your research ideas with Alex Miller at almill@pennmedicine.upenn.edu.
