๐Ÿ“–
A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition
Li Liu, Gang Feng, Denis Beautemps, Xiao-Ping Zhang

A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition

Science & Mathematics Published 2019 Verification Pending

Overview & Synopsis

Cued Speech (CS) is an augmented lip reading with the help of hand coding. Due to lips and hand movements are asynchronous and a direct fusion of these asynchronous features may reduce the efficiency of the recognition, the fusion of them in automatic CS recognition is a challenging problem. In our previous work, we built a hand preceding model for hand positions (vowels) by investigating the temporal organization of hand movements in French CS. In this work, we investigate a suitable value of the hand preceding time for consonants by analyzing the temporal movements of hand shapes in French CS. Then, based on these two results, we propose an efficient resynchronization procedure for the fusion of multi-stream features in CS. This procedure is applied to the continuous CS phoneme recognition based on the multi-stream CNN-HMMs architecture. The result shows that using this procedure brings an improvement of about 4.6% in the phoneme recognition correctness, compared with the state-of-the-art, which does not take into account the asynchrony of multi-modalities.

Indexed from repository: HAL Open Science (hal.science)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title A novel resynchronization procedure for hand-lips fusion applied to continuous French Cued Speech recognition
Author / Organization Li Liu, Gang Feng, Denis Beautemps, Xiao-Ping Zhang
Subject & Discipline Multidisciplinary Academic Open Research
Categories Science & Mathematics
Publication Year 2019
License / Rights Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website HAL Open Science (hal.science)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-204416
Original Source Link Visit Source Page on hal.science »