Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild

Yibo He; Kah Phooi Seng; Li-Minn Ang

doi:10.3390/s23041834

Back

Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild

Journal article

Open access

Peer reviewed

Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild

Yibo He, Kah Phooi Seng and Li-Minn Ang

Sensors, Vol.23(4), pp.1-12

2023

DOI: https://doi.org/10.3390/s23041834

PMID: 36850432

Files and links (2)

pdf

Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild1.67 MBDownload View

Published VersionCC BY V4.0, Open Access

url

https://doi.org/10.3390/s23041834View

Published Version

Abstract

multimodal sensing

audio-visual speech recognition

deep learning

This paper investigates multimodal sensor architectures with deep learning for audio visual speech recognition, focusing on in the wild scenarios. The term "in the wild" is used to describe AVSR for unconstrained natural language audio streams and video stream modalities. Audio visual speech recognition (AVSR) is a speech recognition task that leverages both an audio input of a human voice and an aligned visual input of lip motions. However, since in the wild scenarios can include more noise, AVSR's performance is affected. Here, we propose new improvements for AVSR models by incorporating data augmentation techniques to generate more data samples for building the classification models. For the data augmentation techniques, we utilized a combination of conventional approaches (e.g., flips and rotations), as well as newer approaches, such as generative adversarial networks (GANs). To validate the approaches, we used augmented data from well known datasets (LRS2 Lip Reading Sentences 2 and LRS3) in the training process and testing was performed using the original data. The study and experimental results indicated that the proposed AVSR model and framework, combined with the augmentation approach, enhanced the performance of the AVSR framework in the wild for noisy datasets. Furthermore, in this study, we discuss the domains of automatic speech recognition (ASR) architectures and audio visual speech recognition (AVSR) architectures and give a concise summary of the AVSR models that have been proposed.

Details

Title: Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild
Authors: Yibo He (Author) - Xi’an Jiaotong-Liverpool University
Kah Phooi Seng (Corresponding Author) - Queensland University of Technology
Li-Minn Ang (Author) - University of the Sunshine Coast, Queensland, School of Science, Technology and Engineering
Publication details: Sensors, Vol.23(4), pp.1-12
Publisher: MDPI AG
Date published: 2023
DOI: 10.3390/s23041834
ISSN: 1424-8220
PMID: 36850432
Copyright note: © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/ 4.0/).
Organisation Unit: University of the Sunshine Coast, Queensland; School of Science, Technology and Engineering; Engage Research Lab
Language: English
Record Identifier: 99713297402621
Output Type: Journal article

Metrics

2 File views/ downloads

155 Record Views

6 Times Cited - Web of Science

See more details

InCites Highlights

These are selected metrics from InCites Benchmarking & Analytics tool, related to this output

Collaboration types: Domestic collaboration; International collaboration
Web Of Science research areas: Chemistry, Analytical; Engineering, Electrical & Electronic; Instruments & Instrumentation