A completely paralyzed man has been able to communicate entire sentences using a device that records his brain activity. The man was able to train his mind to use the device, which was implanted in his brain, to ask for massages, soup, and beer, and to watch films with his son.
It is the first time a completely locked-in person—someone who is conscious and cognitively able but completely paralyzed— has been able to communicate in this way, say the researchers behind the work.
Brain-computer interfaces record the electrical signals inside a person’s brain and convert them to commands that control a device. In recent years, BCIs have enabled partially paralyzed people to control prosthetic limbs or communicate a simple “yes” or “no” by thought alone. But this is the first time someone who is completely locked in, and unable even to control their eye movements, has used a BCI to communicate full sentences.
“It’s really remarkable to be able to reestablish communication with someone in a completely locked-in state,” says Jaimie Henderson, a neurosurgeon at Stanford University, who was not involved in the work. “To me, that’s a tremendous breakthrough and obviously quite meaningful for the research participant.”
To support MIT Technology Review’s journalism, please consider becoming a subscriber.
The man, who lives in Germany, was diagnosed with amyotrophic lateral sclerosis (ALS) in August 2015, when he was 30 years old. ALS is a rare, progressive neurological disease that affects neurons involved in movement. By the end of the year, he was unable to speak or walk. Since July 2016, he has relied on a ventilator to breathe.
He began using an eye-tracking device to communicate in August 2016. The device monitors eye movements, allowing users to select letters from a computer screen. But a year later, his condition had deteriorated and he was no longer able to keep his eyes fixed on a particular spot. The device was now useless. His family members started to use a paper-based approach, in which a family member would hold up a grid of letters against a background of four colors. Family members would point to each color section and row, and interpret any eye movement as a “yes.”
The man and his family worried that he would eventually lose the ability to move his eyes at all, so they sought out the help of Niels Birbaumer, then at the University of Tubingen, and Ujwal Chaudhary of ALS Voice gGmbH, a nonprofit that offers BCIs and other technologies to people that are otherwise unable to communicate.
On meeting the man in February 2018, Chaudhary tried to automate the communication system that the family was already using. The team connected an eye-tracking device to computer software that would read out colors and row numbers, allowing the man to select letters one at a time using his eye movements to spell out words.
But as the man increasingly lost control over his eye movements, he became less able to communicate using that device, too. “We proposed implanting [an electrode],” says Chaudhary. Tiny electrodes can be implanted in the brain to record the electrical activity of brain cells directly. The procedure—which tends to involve drilling a hole in the skull and cutting away the brain’s protective layers—comes with a small risk of infection and damage to the brain. So it was a last resort, says Birbaumer. “If the [non-invasive] BCIs and the eye-trackers don’t work anymore, there is no other choice,” he says.
The man consented to the procedure using eye movements, says Chaudhary. His wife and sister also gave their consent. By the time the procedure had been approved by an ethics committee and Germany’s Federal Institute for Drugs and Medical Devices at the end of 2019, the man had lost the ability to use the eye-tracking device. In March 2019, surgeons implanted two tiny electrode grids, each measuring 1.5 millimeters across, into the man’s motor cortex—a region at the top of the brain responsible for controlling movement.
Turning signals into commands
The day after the electrode was implanted, the team began trying to help the man communicate. At first, the man was asked to imagine making physical movements—this has helped other recipients control prosthetic limbs and exoskeletons, and is the approach that Elon Musk’s company Neuralink plans to take. The idea is to get a reliable signal from the brain, and translate this into some kind of command.
But the team couldn’t get it to work. After 12 weeks of trying, they scrapped the idea and decided to try an approach called neurofeedback instead. Neurofeedback works by showing a person their brain activity in real time so that they can learn how to control it. In this case, when the electrodes in the man’s brain recorded an increase in activity, a computer would play a rising audio tone. A fall in brain activity would play a descending tone.
“Within two days, he was able to increase and decrease the frequency of a sound tone,” says Chaudhary, who says he visited the man in his home every weekday during 2019, until coronavirus hit. “It was just incredible.” The man eventually learned to control his brain activity so that he could play a rising tone to signal “yes” and a descending tone to signal “no.”
The team then introduced software that mimicked the paper-based computer system the man had originally used to communicate with his family. The man would hear the word “yellow” or “blue” for example, to choose a block of letters from which to select. He would then be played individual letters and use a rising or descending tone to either select or dismiss each (see video).
Effortless
In this way, the man was soon able to communicate entire sentences. “[His family] were so excited to hear what he had to say,” says Chaudhary, who, along with his colleagues, published their findings in the journal Nature Communications on Tuesday. One of the first sentences the man spelled was translated as “boys, it works so effortlessly.”
Communication was still slow—it takes around a minute to select each letter. But researchers believe the device has significantly improved the man’s quality of life. He has asked for specific meals and soups, directed carers on how to move and massage his legs, and asked to watch films with his young son, for example. One sentence translated as “I love my cool son.”
“Many times, I was with him until midnight, or past midnight,” says Chaudhary. “The last word was always ‘beer.’”
One of the first sentences the man spelled was translated as “boys, it works so effortlessly.”
Chaudhary envisions developing a catalog of frequently used words that could eventually allow software to autocomplete the man’s words as he spells them, for example. “There are many ways in which we could make it faster,” he says.
No one knows how long the electrodes will last in the man’s brain, but other studies have found that similar electrodes are still functioning five years after being implanted in other people. But for a locked-in person, “a single day can make a difference,” says Kianoush Nazarpour at the University of Edinburgh, who was not involved in the work. “That is a fundamental opportunity for them to regain choice and control of their lives,” he says. “A day of high quality could be really important for that person.”
Nazarpour thinks that the technology could be routinely offered to similarly locked-in individuals within the next 10 to 15 years. “For a person that has absolutely no communication, even a “yes”/”no” is potentially life-changing,” he says.
Brian Dickie, director of research development at the Motor Neurone Disease Association in the UK, agrees that this timeline is realistic. But he wonders how many people with motor neuron disease—of which ALS is the most common type—stand to benefit from such BCIs.
Setbacks
The man who received the BCI has a form of ALS called progressive muscular atrophy (PMA). This form of the disease tends to target motor nerves that travel from the spine to muscles, leaving people unable to control their muscles. But around 95% of cases of ALS also involve degeneration of the motor cortex in the brain, too, says Dickie.
Even people with PMA can experience this degeneration, which might explain why, since the study was completed, the man appears to have lost some of his ability to communicate. For the past month or so, the man has only used the device to communicate “yes” or “no,” says Birbaumer.
“I have no clue why he is only saying yes and no and not formulating sentences, but that happens from time to time,” Birbaumer says. “There could be many reasons for this.”
It’s also possible, for example, that specialized cells in the brain have recognized the electrode as foreign and clumped around it, limiting its ability to function. “It could be psychological reasons, technical reasons, electrode reasons … but at least he is in good shape and has a good quality of life, as he communicates,” says Birbaumer.
If the electrode eventually fails, the man’s family members might request that it be removed and replaced with another, perhaps in a different brain region. But, for now, having the ability to say “yes” and “no” is enough, says Birbaumer. “[The family] told me … most information they needed was exchanged in the first one and a half years,” he says (the family members themselves have requested privacy at this time). “With ‘yes’ and ‘no’ you can say many things … if you ask the right questions.”