Silent speech

Loud telephone calls are the number one irritation in trains. The first-ever Dutch lip reader might provide a solution: silent speech or mute miming.

If a talking mouth is all there is on the screen, the automated lip reader that Dr Alin Chitu has developed even beats experienced human lip readers. However, as soon as the image widens, humans win hands down. Context is perhaps the most important factor for humans in visual speech recognition. You needn’t be especially gifted to understand the request for a ‘Biertje?’ in a crowded, noisy bar.
For his PhD project at the faculty of Electrical Engineering, Mathematics and Computer Sciences, the Romanian-born Chitu compared three different methods for lip reading, setting up ‘corpus’, or database, of twelve hours of video recorded spoken Dutch that contains spoken digits, letters, texts read aloud and bits of spontaneous speech. As for which is the best method, that depends, Chitu says. At high speech rates, the method that continually fits a mouth shape over the camera image seems to perform best. At lower speech rates, the method that only captures the movement of the mouth is the best choice.

So far the system recognizes 92 percent of the spoken digits correctly; 60 to 70 percent of spelled letters; 50 percent of fenced-in talk (in a fixed menu) and about a third of free speech. Currently, the prototype needs about two to three times the utterance duration for processing, which is quite fast considering how many different options the system must weigh and discard.
Chitu’s PhD supervisor, Professor Leon Rothkrantz, thinks the first lip-reading application will be in mobile phones. “It’s a huge market,” he says. Recognition of silent speech would not only benefit rail commuters, but also crisis communication in noisy environments. Ever tried making a call from a helicopter? Yet another application could be in the very labour-intensive process of teaching hearing-impaired children to lip read.

Dr Alin Chitu, Towards Robust Visual Speech Recognition, 2 November 2010, PhD supervisor Professor Leon Rothkrantz.

Link to Delta site