News Story

Patients with paralysis can speak and move via a virtual avatar when using a new device created by scientists.

The new “brain computer interface” (BCI) is the first which enables speech and movement activities in one device.

Previous devices have helped paralysed patients speak or demonstrate movement, but not simultaneously.

The device allows brain activity be recorded in real time enabling the patient to control a personalised virtual avatar which enables gesture and communication.

The research team, led by experts from the University of California, San Francisco, said that natural communication often combines words and movement including facial expressions and manual gestures.

Yet previous work on BCIs, which work by detecting brain activity in the sensorimotor cortex of the brain, have looked at speech and movement in isolation.

Until now, it has been “unclear whether a single BCI implant can reliably decode speech and gestures when they are attempted together”, the authors said.

Researchers set out to see whether a BCI used to decode brain signals to make an avatar replicate face and arm movements in three patients with severe paralysis.

First they tested the device which translates brain activity and “decodes” it to create commands sent to external device.

They wanted to test the feasibility of real time speech and gesture “decoders” to see whether they could drive a personalised full-body avatar.

First they assessed the use of the BCI to decode movement and speech separately using a set of phrases and gestures, and then both together.

The team then assessed whether two patients could control a personalised virtual avatar with speech and movement simultaneously, including shrugging, fist pumps or nodding the head.

One of the patients 100% accuracy for both speech and gesture decoding in three of the assessments, the research team found.

The brain implant could benefit people paralysed by stroke or neurogenerative diseases such as motor neurone disease, which can impair both speech and movement, reducing a person’s ability to interact with others, they said.

But they said that testing in more participants and with larger sets of words and gestures will be needed before the technology can be used by patients.

This simultaneous, or “multi-effector”, speech and movement could help patients with speech and paralysis have “expressive interaction”, the authors said.

“These findings support the feasibility of simultaneous, multi-effector decoding from a single implant as a step toward more flexible, expressive communication for people with severe bodily and vocal-tract paralysis,” they wrote in the journal Nature Neuroscience.

Researcher Samantha Brosler, PhD candidate in bioengineering at UCSFUC Berkeley, told the Press Association: “We recorded brain activity from people with paralysis using electrodes implanted on the surface of the brain while they attempted to speak, make gestures, or do both at the same time.

“We then trained machine learning models to translate brain activity into attempted phrases or body movements.

“We also created a personalised, full-body avatar for each participant using the video game development platform Unreal Engine.

“In real time, we ran a speech decoder and a gesture decoder in parallel.

“If the participant attempted speech, the decoded phrase appeared as text on the screen; if they attempted a gesture, the avatar performed the corresponding movement. Because the two decoders operated independently, participants could use speech alone, a gesture alone, or both at the same time.”

She added: “The participants were excited to see the system working in real time.

“One participant even started laughing when he saw that his speech and gestures were being successfully decoded together and acted out through the avatar.

“The participants were also involved in designing how they wanted their personalised virtual avatars to look, so they were invested in the process from the beginning.

“I think that made it all the more meaningful for them to ultimately control those avatars themselves using their own brain activity.”

She went on: “To our knowledge, this is the first demonstration that speech and communicative gestures can be decoded at the same time from a single brain implant in people with paralysis, as well as the first demonstration of BCI-enabled control of a full-body virtual avatar.

“This is still an early proof of concept, with simultaneous speech and gesture decoding tested in two participants using a constrained set of phrases and gestures.

“But it shows the potential for a single brain implant to combine multiple forms of communication, including speech and body movements, to support more flexible and expressive interaction for people with paralysis.”

Commenting on the research, Professor Deb Lowe, medical director for the Stroke Association said: “Every day, around 240 people in the UK survive a stroke.

“For many, the resulting brain injury can have profound and lasting impacts, leaving stroke survivors with disabilities such as difficulty walking or talking.

“Specialist stroke rehabilitation will always be essential to help stroke survivors regain independence and some evidence-based technological innovations could complement existing support.

“So, we welcome such research which could provide stroke survivors with the opportunity to communicate more effectively.

“However, this research is in its early clinical stages, and further exploration is required to fully understand how it could truly benefit stroke survivors.”