By Sarah Ward via SWNS
Hearing aids will be able to read lips even if people are wearing face masks, reducing ambient noise.
An international team of engineers and computing scientists developed the technology - which pairs radio-frequency sensing with Artificial Intelligence for the first time along with traditional hearing aid technology, to identify lip movements.
It could help tackle the ‘cocktail party effect,' a common shortcoming of traditional hearing aids as the broad spectrum of amplification can make it difficult for users to focus on specific sounds, like a conversation with a particular person.
New ‘smart’ hearing aids would combine conventional audio amplification with a second device to collect additional data for improved performance, unlike previous attempts using cameras to lip read which raised privacy issues and did not work with masks.
In a new paper published today (Sept 7, 2022) in the journal Nature Communications, the University of Glasgow-led team outline how they set out to harness cutting-edge sensing technology to read lips.
It preserves privacy by collecting only radio-frequency data, with no accompanying video footage.
To develop the system, the researchers asked male and female volunteers to repeat the five vowel sounds (A, E, I, O, and U) first while unmasked and then while wearing a surgical mask.
As the volunteers repeated the vowel sounds, their faces were scanned using radio-frequency signals from both a dedicated radar sensor and a wifi transmitter.
Their faces were also scanned while their lips remained still.
Then, the 3,600 samples of data collected during the scans were used to ‘teach’ machine learning and deep learning algorithms how to recognize the characteristic lip and mouth movements associated with each vowel sound.
Because the radio-frequency signals can easily pass through the volunteers’ masks, the algorithms could also learn to read masked users’ vowel formation.
The system proved to be capable of correctly reading the volunteers’ lips most of the time.
Wifi data was correctly interpreted by the learning algorithms up to 95 percent of the time for unmasked lips, and 80 percent for masked.
The radar data was interpreted correctly up to 91 percent without a mask, and 83 percent of the time with a mask.
Dr. Qammer Abbasi, of the University of Glasgow’s James Watt School of Engineering, is the paper’s lead author.
He said: “Around five percent of the world’s population - about 430 million people - have some kind of hearing impairment.
“Hearing aids have provided transformative benefits for many hearing-impaired people.
"A new generation of technology which collects a wide spectrum of data to augment and enhance the amplification of sound could be another major step in improving hearing-impaired people’s quality of life.
“With this research, we have shown that radio-frequency signals can be used to accurately read vowel sounds on people’s lips, even when their mouths are covered.
"While the results of lip-reading with radar signals are slightly more accurate, the Wi-Fi signals also demonstrated impressive accuracy.
“Given the ubiquity and affordability of Wi-Fi technologies, the results are highly encouraging which suggests that this technique has value both as a standalone technology and as a component in future multimodal hearing aids.”
Professor Muhammad Imran, head of the University of Glasgow’s Communications, Sensing and Imaging research group and a co-author of the paper, added: “This technology is an outcome of two research projects funded by the Engineering and Physical Sciences Research Council (EPSRC), called COG-MHEAR and QUEST.
“Both aim to find new methods of creating the next generation of healthcare devices, and this development will play a major role in supporting that goal.”
Researchers from the University of Glasgow, Edinburgh Napier University in the UK contributed to the paper, along with colleagues from the University of Engineering and Technology Lahore, Pakistan and Southeast University, Nanjing in China.
The team’s paper, titled ‘Pushing the Limits of Remote RF Sensing by Reading Lips Under the Face Mask’, is published here.
The research was supported by funding from the Engineering and Physical Sciences Research Council (EPSRC)
Stories and infographics by ‘Talker Research’ are available to download & ready to use. Stories and videos by ‘Talker News’ are managed by SWNS. To license content for editorial or commercial use and to see the full scope of SWNS content, please email firstname.lastname@example.org or submit an inquiry via our contact form.
Encuesta revela las mejores cosas de comprar en una pequeña empresa
La persona promedio comprar en una pequeña empresa 213 veces durante el año. Una encuesta reciente de 2,000 adultos reveló...
Study: Kids’ mental health and self-esteem can plummet starting at this age
The phenomenon applies to youngsters regardless of economic and family circumstances.
Tea with breakfast and wine with dinner could reduce risk of dementia
The number of dementia cases worldwide is expected to tripled to more than 150 million by 2050.
Study: COVID-19 vaccine provides substantial protection against reinfection
While many people have developed long-lasting natural immunity after contracting the virus, the new research shows that the COVID-19 vaccine...
Man born with half a body refuses to let it hold him back
"My independence is the thing I’m most grateful for."
- Shopping4 days ago
Why a lot of Americans plan to shop in person for Black Friday this year
- Holidays4 days ago
Healthy eating canceled for the rest of the year
- Small Biz5 days ago
Survey says these are the best things about shopping small
- Home6 days ago
Couple fixing house that’s been abandoned since the ’80s
- Sleep7 days ago
Average American loses this much sleep during the holidays
- Outer Space5 days ago
Check out NASA’s Artemis spacecraft’s stunning selfies
- Food & Drink6 days ago
Spicy food lovers more likely to think they’re hot
- Tech5 days ago
This $5.1 million submarine moves faster than a dolphin