Audio-visual speaker verification using continuous fused HMMs

Abstract

This paper examines audio-visual speaker verication using a novel adaptation of fused hidden Markov mod-els, in comparison to output fusion of individual clas-siers in the audio and video modalities. A com-parison of both hidden Markov model (HMM) and Gaussian mixture model (GMM) classiers in both modalities under output fusion shows that the choice of audio classier is more important than video. Al-though temporal information allows a HMM to out-perform a GMM individually in video, this temporal information does not carry through to output fusion with an audio classier, where the dierence between the two video classiers is minor. An adaptation of fused hidden Markov models, designed to be more ro-bust to within-speaker variation, is used to show that the temporal relationship between video observations and audio states can be harnessed to reduce errors in audio-visual speaker verication when compared to output fusion.