Andreas Dahl has been hard of hearing since early childhood. He is not deaf, but he needs to look at the person he is talking to, to be able to lip-read and understand the conversation. (Feb. '05)
Hear My Voice, Read My Lips
Andreas Dahl has been hard of hearing since early childhood. He is not deaf, but he needs to look at the person he is talking to, to be able to lip-read and understand the conversation. A problem arises when he wants to talk to somebody on the phone. But now a new technological development could change his life.
At KTH, the Royal Institute of Technology in the heart of the Swedish capital Stockholm is the office of Jonas Beskow. He works in the Department of Speech, Music and Hearing and for the last ten years he has worked on computer-animated faces that offer a way of translating speech into visual information for hard hearing people.
He is now working on a project called SYNFACE, which aims to help people like Andreas Dahl to use the telephone. In fact, Andreas is one of the first hard-of-hearing people to trial the technology. The system consists of an animated representation of a human face that appears on the computer screen. As soon as the computer receives the speech signal, the artificial face starts talking too ? without any delay. It therefore seems that it is actually the computerised face talking, as it mimics the lip and mouth movements of a voice it detects.
?The idea is?, explains Jonas Beskow, ?to attach the computer to the telephone. When you make a phone call, the incoming voice is being translated into the lip movements of the artificial face?. The result: Andreas can actually read the lips of the artificial face while he talks to business partners, relatives or his girlfriend. But does it really work?
?It is not the same, as when you lip-read a real person, the lips seem a bit stiff?, Andreas admits, but it is something you get used to. ?I had to learn the way SYNFACE talks, but once you have the hang of it, it is really useful?. Andreas has already tried SYNFACE at home and at work, and it has made his life much easier, especially when he needs to talk to strangers who don?t know about his disability.
The first main challenges of SYNFACE were to analyse the speech into individual sounds. ?It was much harder than we had anticipated?, says Karl-Erik Spens, who initiated the project 5 years ago. The computer had to recognize the different sounds of different Swedish voices to be able to decode them for the computer. Then, the computer had to transform these sounds into the appropriate lip and face movements ? a second daunting task. ?Now we have a delay of 200 milliseconds, from the incoming voice to the lip movements of our artificial face. This is virtually real-time? says Jonas Beskow.
Karl-Erik Spens is convinced that SYNFACE is only the beginning. Research has shown that each of us can actually understand a voice much better when we see the face than just hear the voice. ?Future applications of SYNFACE could be used in noisy areas that rely on announcements, such as train stations or airports?, says Karl-Erik.
In the meantime, the Swedish speech experts have linked up with other European colleagues to teach the artificial face other languages beyond Swedish. With the help of the European Commission they were able to gain funds to make SYNFACE a truly multilingual device that could help millions of hear impaired people in Europe. Already, the artificial face understands English, Dutch and Swedish, and the research team are experimenting on other languages, such as Finnish, German and Italian.
Babel-Infovox, the industrial partner of the project, hopes that within the next two years this research will lead to a product that people like Andreas will be able to buy commercially. ?And the good thing?, concludes Karl-Erik Spens,?it will be very cheap, because it is a simple software application.?
WEBLINKS
> SYNFACE homepage
SYNFACE allows hard of hearing people to understand conversations also on the phone due to translation of speech into visual information.
> eInclusion & eAccessibility
European Commission web pages on eInclusion and eAccessibility. EInclusion aims to ensure that disadvantaged people are not left behind.
> IFHOH- International Federation of hard of hearing people
IFHOH- a nongovernmental organisation which supports cooperation between hard of hearing organisations worldwide and places emphasis on the inclusion of hard of hearing people in society.
> World Federation of the deaf
WFD is an international operating, non- governmental organisation founded in 1951. It represents millions of deaf people worldwide and concentrates on improving their status, education etc.
> Public Broadcasting Service
This web site is part of PBS. Among others it provides an insight to ?deaf culture?. It summarises information on e.g. living with deafness, voices of deafness etc.
> European Conference
Information on the 2004 EU conference about ?Access to the Information Society for deaf and hard of hearing people?.
CONTACTS
Dr Inger Karlsson
> Project Co-ordinator
Kungliga Tekniska H?olan (KTH)
Dept. of Speech, Music and Hearing
Stockholm
Sweden
inger@speech.kth.se
