Skip to content

Publications

Detail of publication

Citation

Karpov Alexey and Ronzhin Andrey and Markov Konstantin and Železný Miloš : Viseme-Dependent Weight Optimization for CHMM-Based Audio-Visual Speech Recognition . Interspeech 2010, vol. 2010, p. 2678-2681, Curran Associates, Makuhari, Chiba, Japan, 2010.

Abstract

The aim of the present study is to investigate some key challenges of the audio-visual speech recognition technology, such as asynchrony modeling of multimodal speech, estimation of auditory and visual speech significance, as well as stream weight optimization. Our research shows that the usage of viseme-dependent stream weights improves the performance of state asynchronous CHMM-based speech recognizer. In addition, for a state synchronous MSHMM-based recognizer, fewer errors can be achieved using stationary time delays of visual data with respect to the corresponding audio signal. Evaluation experiments showed that individual audio-visual stream weights for each viseme-phoneme pair lead to relative reduction of WER by 20%.

Detail of publication

Title: Viseme-Dependent Weight Optimization for CHMM-Based Audio-Visual Speech Recognition
Author: Karpov Alexey ; Ronzhin Andrey ; Markov Konstantin ; Železný Miloš
Language: English
Year: 2010
Type of publication: Papers in proceedings of reviewed conferences
Title of journal or book: Interspeech 2010
Číslo vydání: 2010
Page: 2678 - 2681
ISBN: 978-1-61782-123-3
Publisher: Curran Associates
Address: Makuhari, Chiba, Japan
Date: 30 Sep 2010
2011-03-15 16:21:56 / 2011-03-15 16:21:56 / 1

Keywords

multimodal speech, audio-visual processing, phoneme, viseme, Hidden Markov Models, asynchrony, significance weights

BibTeX

@INPROCEEDINGS{KarpovAlexey_2010_Viseme-Dependent,
 author = {Karpov Alexey and Ronzhin Andrey and Markov Konstantin and \v{Z}elezn\'{y} Milo\v{s}},
 title = {Viseme-Dependent Weight Optimization for CHMM-Based Audio-Visual Speech Recognition},
 year = {2010},
 publisher = {Curran Associates},
 journal = {Interspeech 2010},
 address = {Makuhari, Chiba, Japan},
 volume = {2010},
 pages = {2678-2681},
 ISBN = {978-1-61782-123-3},
 url = {http://www.kky.zcu.cz/en/publications/KarpovAlexey_2010_Viseme-Dependent},
}